DiffusionGemma-26B-A4B · vLLM · NVFP4 · conc 8
status: done
Configuration
| Model | nvidia/diffusiongemma-26B-A4B-it |
|---|---|
| Company | NVIDIA |
| Family | Gemma |
| Parameters | 25.2B / 3.8B (MoE, diffusion) |
| Engine | vLLM |
| Quant / precision | NVFP4 |
| Why this quant | NVIDIA's official NVFP4 (nvidia/diffusiongemma-26B-A4B-it-NVFP4, ModelOpt) discrete-diffusion Gemma-4 MoE — decodes in parallel 256-token denoising blocks (no drafter, no verify). The diffusion line of the crossover figure — fastest at low concurrency, saturates ~199 tok/s from conc 8 onward. |
| Download | nvidia/diffusiongemma-26B-A4B-it-NVFP4 |
| Context window | 65536 |
| Input modalities | text, image, video (served text-only here) |
Measured results
| Prefill tok/s | 292.29 |
|---|---|
| Decode tok/s | 199.18 |
| Peak memory (GB) | 108.8 (system MemAvailable delta (10s sampling) — NVFP4 MoE + diffusion bidirectional-attention KV (util 0.70)) |
| Completed | 2026-07-04 00:33 +0800 |
Full run command
VLLM_IMAGE=vllm/vllm-openai:nightly-aarch64 scripts/bench-vllm-serving.sh nvidia/diffusiongemma-26B-A4B-it-NVFP4 65536 8 500 300 256 \
--trust-remote-code --attention-backend TRITON_ATTN --reasoning-parser gemma4 --tool-call-parser gemma4 --enable-auto-tool-choice --gpu-memory-utilization 0.70
# 265/500 prompts (hit the time cap), 0 errors, 305.6s. ready after 222s. TTFT median 8942.8 ms, TPOT median 0.0 ms, req thr 0.867/s.
Notes
Decode 199.18 tok/s aggregate at concurrency 8. DiffusionGemma NVFP4 (parallel 256-token denoising blocks), conc 8, for the crossover figure.
- Image (pinned):
vllm/vllm-openai:nightly-aarch64@sha256:e414712fdc04…— the SINGLE image for all 24 cells of this figure. Ready after 222 s. - Workload: ShareGPT V3, concurrency 8. 265/500 completed, 0 errors before the 300 s time cap.
- Throughput: decode 199.18 tok/s aggregate, prefill 292.29 tok/s. TTFT median 8942.8 ms, TPOT median 0.0 ms, req throughput 0.867/s.
- TPOT is meaningless here — decode happens in 256-token parallel denoising blocks, so the client’s per-token latency reads near-zero. Trust the aggregate decode tok/s only.
- Reduced load (util 0.70, not 0.85). At
--gpu-memory-utilization 0.85this cell hung the box — the ~103 GB KV reservation plus diffusion’s runtime denoising buffers exceeded the 121 GB unified ceiling. Re-run at 0.70 (reservation ~84 GB) it peaked at 108.80 GB (~12 GB under the ceiling), 0 errors. Shorter 300 s window — decode tok/s is a steady-state rate, unaffected. - Memory: 108.80 GB peak (MemAvailable delta) = ~84 GB vLLM KV reservation (util 0.70) plus diffusion’s denoising/activation buffers (~25 GB) — not just the reservation. That buffer overhead on top of a 0.85 reservation is exactly what breached the ceiling at higher concurrency.