DiffusionGemma-26B-A4B · vLLM · NVFP4 · conc 4
status: done
Configuration
| Model | nvidia/diffusiongemma-26B-A4B-it |
|---|---|
| Company | NVIDIA |
| Family | Gemma |
| Parameters | 25.2B / 3.8B (MoE, diffusion) |
| Engine | vLLM |
| Quant / precision | NVFP4 |
| Why this quant | NVIDIA's official NVFP4 (nvidia/diffusiongemma-26B-A4B-it-NVFP4, ModelOpt) discrete-diffusion Gemma-4 MoE — decodes in parallel 256-token denoising blocks (no drafter, no verify). The diffusion line of the crossover figure — fastest at low concurrency, saturates ~199 tok/s from conc 8 onward. |
| Download | nvidia/diffusiongemma-26B-A4B-it-NVFP4 |
| Context window | 65536 |
| Input modalities | text, image, video (served text-only here) |
Measured results
| Prefill tok/s | 222.82 |
|---|---|
| Decode tok/s | 184.07 |
| Peak memory (GB) | 113.37 (system MemAvailable delta (10s sampling) — NVFP4 MoE + diffusion bidirectional-attention KV (util 0.85)) |
| Completed | 2026-07-04 00:33 +0800 |
Full run command
VLLM_IMAGE=vllm/vllm-openai:nightly-aarch64 scripts/bench-vllm-serving.sh nvidia/diffusiongemma-26B-A4B-it-NVFP4 65536 4 500 500 256 \
--trust-remote-code --attention-backend TRITON_ATTN --reasoning-parser gemma4 --tool-call-parser gemma4 --enable-auto-tool-choice
# 405/500 prompts (hit the time cap), 0 errors, 502.1s. ready after 248s. TTFT median 4794.2 ms, TPOT median 0.0 ms, req thr 0.807/s.
Notes
Decode 184.07 tok/s aggregate at concurrency 4. DiffusionGemma NVFP4 (parallel 256-token denoising blocks), conc 4, for the crossover figure.
- Image (pinned):
vllm/vllm-openai:nightly-aarch64@sha256:e414712fdc04…— the SINGLE image for all 24 cells of this figure. Ready after 248 s. - Workload: ShareGPT V3, concurrency 4. 405/500 completed, 0 errors before the 500 s time cap.
- Throughput: decode 184.07 tok/s aggregate, prefill 222.82 tok/s. TTFT median 4794.2 ms, TPOT median 0.0 ms, req throughput 0.807/s.
- TPOT is meaningless here — decode happens in 256-token parallel denoising blocks, so the client’s per-token latency reads near-zero. Trust the aggregate decode tok/s only.
- Memory: 113.37 GB peak (MemAvailable delta) = ~102 GB vLLM KV reservation (util 0.85) plus diffusion’s denoising/activation buffers (~11 GB) — not just the reservation. That buffer overhead on top of a 0.85 reservation is exactly what breached the ceiling at higher concurrency.