Gemma 4 26B-A4B · vLLM · NVFP4 + MTP · conc 16
status: done
Configuration
| Model | google/gemma-4-26B-A4B-it |
|---|---|
| Company | |
| Family | Gemma |
| Parameters | 26B / 4B (MoE) |
| Engine | vLLM + MTP (Google assistant drafter) (speculative decoding) |
| Quant / precision | NVFP4 |
| Why this quant | NVIDIA NVFP4 base (modelopt) + Google's official MTP assistant drafter (google/gemma-4-26B-A4B-it-assistant) via vLLM's native gemma-4 MTP path — the near-free-drafter line of the crossover figure. |
| Download | nvidia/Gemma-4-26B-A4B-NVFP4 |
| Context window | 65536 |
| Input modalities | text, image (served text-only here) |
Measured results
| Prefill tok/s | 498.8 |
|---|---|
| Decode tok/s | 436.71 |
| Peak memory (GB) | 108.39 (system MemAvailable delta (10s sampling) — vLLM static KV reservation (util 0.85) + Gemma4 MTP head) |
| Completed | 2026-07-03 16:32 +0800 |
Full run command
VLLM_IMAGE=vllm/vllm-openai:nightly-aarch64 scripts/bench-vllm-serving.sh nvidia/Gemma-4-26B-A4B-NVFP4 65536 16 1000 700 256 \
--speculative-config '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":3}'
# 1000/1000 prompts (clean full run), 0 errors, 540.3s. ready after 328s. TTFT median 276.6 ms, TPOT median 33.9 ms, req thr 1.851/s.
# SpecDecoding: mean acceptance length 2.64 (2.30-2.89) · avg draft acceptance 55% (43-63%) · per-position ~0.74/0.53/0.37.
Notes
Decode 436.71 tok/s aggregate at concurrency 16. NVFP4 + Google MTP assistant drafter, conc 16 — the near-free-drafter line of the crossover figure.
- Image (pinned):
vllm/vllm-openai:nightly-aarch64@sha256:e414712fdc04…— the SINGLE image for all 24 cells of this figure. Ready after 328 s. - Workload: ShareGPT V3, concurrency 16. 1000/1000, 0 errors in 540.3 s (clean full run).
- Throughput: decode 436.71 tok/s aggregate, prefill 498.8 tok/s. TTFT median 276.6 ms, TPOT median 33.9 ms, req throughput 1.851/s.
- Spec-decode acceptance: mean acceptance length 2.64 (2.30-2.89) · avg draft acceptance 55% (43-63%) · per-position ~0.74/0.53/0.37 (num_speculative_tokens=3).
- Memory: 108.39 GB = vLLM
--gpu-memory-utilization 0.85reservation, not the model footprint.