where \(W_{bits}\) is memory bus width and \(R_{MT/s}\) is effective transfer rate.
LPDDR5 specs often publish 3200 MHz; for these platforms this corresponds to 6400 MT/s effective DDR data rate. NVIDIA’s own 128-bit @ 3200 MHz -> 102.4 GB/s confirms this convention [1]. For NVIDIA GeForce RTX 3080 Laptop GPU, use GDDR6 14 Gbps effective = 14000 MT/s; local nvidia-smi detects 16384 MiB VRAM. For NVIDIA GeForce RTX 5060, use GDDR7 28 Gbps effective = 28000 MT/s.
Price caveats: IQ-9075 price uses an available Lantronix SOM, not a Qualcomm direct MSRP. Jetson AGX price is a developer kit price. RTX 3080 Laptop GPU has no standalone card price. RTX 5060 street price is volatile; table keeps both NVIDIA MSRP and one retailer listing.
Calculations:
Platform
Formula
Peak BW
Qualcomm Dragonwing IQ-9075
\(\frac{96}{8}\cdot\frac{6400}{1000}\)
76.8 GB/s
NVIDIA Jetson AGX Orin 64GB
\(\frac{256}{8}\cdot\frac{6400}{1000}\)
204.8 GB/s
NVIDIA GeForce RTX 3080 Laptop GPU
\(\frac{256}{8}\cdot\frac{14000}{1000}\)
448.0 GB/s
NVIDIA GeForce RTX 5060
\(\frac{128}{8}\cdot\frac{28000}{1000}\)
448.0 GB/s
Caveat: Qualcomm’s public IQ-9075 brief gives memory capacity and clock, but not bus width or bandwidth. Public Linux notes describe QCS9075 as an IoT variant of SA8775P [4], so the 96-bit bus comes from public SA8775P-family platform docs [3].
Jetson family notes from NVIDIA official table [1]:
MLC INT4 benchmarks: Llama 3.1 8B (INT4): 14 / 19.14 tok/s; Llama 3.2 3B (INT4): 27.7 / 43.07 tok/s; Gemma 2 2B (INT4): 21.5 / 34.97 tok/s[5]. Nano has ~102 GB/s BW [6], so 8B INT4 theoretical is \(\frac{102.4}{8\cdot0.5}=25.6\) tok/s; published 19.14 tok/s is plausible after overhead.
Jetson AI Lab HF example
Llama-2-7B (FP16): 6.2 tok/s; Llama-2-7B GPTQ (dtype not published in source): 10.3 tok/s; Llama-2-7B AWQ (dtype not published in source): 11.2 tok/s[7]. Transformers stack example; much slower than optimized MLC/TensorRT-style paths.
I ran a local benchmark on my laptop; results roughly track the theoretical max TPS at BF16, but dip down below 50% of theoretical max at lower quant levels.
llama.cpp provides direct GGUF dtype/quant sweeps. Local build: llama.cpp commit d749821, CUDA backend, llama-bench, GPU NVIDIA GeForce RTX 3080 Laptop GPU 16GB. Build used repo-local .deps/llama.cpp plus CUDA compiler/libs from .venv (nvidia/cu13); build/runtime env is in bench_env.sh and build_llama_cpp.sh.
Models are unsloth/Qwen3.5-{2B,4B,9B}-GGUF. Run config: prompt processing = 512 tokens, generation eval = 256 tokens, repetitions = 3, batch/ubatch defaults (2048/512), all layers GPU (-ngl -1), KV cache f16, flash-attn auto. llama.cpp reports prompt processing tok/s (n_prompt=512) separately from decode tok/s (n_gen=256). GPU memory is sampled with nvidia-smi; delta subtracts desktop baseline. 9B BF16 skipped: GGUF is 17.92 GB before KV/overhead, over local 16GB VRAM.
Run the next hidden cell from this notebook to replace llama_cpp_tps_results.jsonl next to index.ipynb, then render the table. The script deletes that result file before writing fresh rows.
Prompt processing is prompt-ingest throughput, not generation throughput; compare decode to decode.