Gemma 4 26B-A4B · vLLM · NVFP4 · conc 16
status: done
Configuration
| Model | google/gemma-4-26B-A4B-it |
|---|---|
| Company | |
| Family | Gemma |
| Parameters | 26B / 4B (MoE) |
| Engine | vLLM |
| Quant / precision | NVFP4 |
| Why this quant | NVIDIA's own NVFP4 build (TensorRT-Model-Optimizer / modelopt) — Blackwell-native 4-bit. The autoregressive reference line (no drafter) for the decode-vs-concurrency crossover figure. |
| Download | nvidia/Gemma-4-26B-A4B-NVFP4 |
| Context window | 65536 |
| Input modalities | text, image (served text-only here) |
Measured results
| Prefill tok/s | 312.65 |
|---|---|
| Decode tok/s | 273.61 |
| Peak memory (GB) | 109.3 (system MemAvailable delta (10s sampling) — vLLM static KV reservation (util 0.85)) |
| Completed | 2026-07-03 15:15 +0800 |
Full run command
VLLM_IMAGE=vllm/vllm-openai:nightly-aarch64 scripts/bench-vllm-serving.sh nvidia/Gemma-4-26B-A4B-NVFP4 65536 16 1000 700 256
# 824/1000 prompts (hit the time cap), 0 errors, 711.5s. ready after 293s. TTFT median 186.1 ms, TPOT median 56.9 ms, req thr 1.158/s.
Notes
Decode 273.61 tok/s aggregate at concurrency 16. Autoregressive NVFP4 reference point, conc 16, for the decode-vs-concurrency crossover figure.
- Image (pinned):
vllm/vllm-openai:nightly-aarch64@sha256:e414712fdc04…— the SINGLE image for all 24 cells of this figure. Ready after 293 s. - Workload: ShareGPT V3, concurrency 16. 824/1000 completed, 0 errors before the 700 s time cap.
- Throughput: decode 273.61 tok/s aggregate, prefill 312.65 tok/s. TTFT median 186.1 ms, TPOT median 56.9 ms, req throughput 1.158/s.
- Memory: 109.30 GB = vLLM
--gpu-memory-utilization 0.85reservation, not the model footprint.