Gemma 4 26B-A4B · vLLM · NVFP4 + EAGLE3 · conc 64
status: done
Configuration
| Model | google/gemma-4-26B-A4B-it |
|---|---|
| Company | |
| Family | Gemma |
| Parameters | 26B / 4B (MoE) |
| Engine | vLLM + EAGLE3 (speculative decoding) |
| Quant / precision | NVFP4 |
| Why this quant | NVIDIA NVFP4 base (modelopt) + RedHatAI's official EAGLE3 speculator (RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3) — the EAGLE3 line of the crossover figure. c64 point of the beyond-c32 ceiling-finding extension. |
| Download | nvidia/Gemma-4-26B-A4B-NVFP4 |
| Context window | 65536 |
| Input modalities | text, image (served text-only here) |
Measured results
| Prefill tok/s | 1017.85 |
|---|---|
| Decode tok/s | 888.98 |
| Peak memory (GB) | 110.79 (system MemAvailable delta (10s sampling) — vLLM static KV reservation (util 0.85) + EAGLE3 head; ~7.9 GB free at peak) |
| Completed | 2026-07-04 13:11 +0800 |
Full run command
# Run under scripts/mem-watchdog.sh (floor 3.5 GB) — GB10 unified memory; see notes/BENCHMARKING.md.
VLLM_IMAGE=vllm/vllm-openai:nightly-aarch64 scripts/bench-vllm-serving.sh nvidia/Gemma-4-26B-A4B-NVFP4 65536 64 1000 900 256 \
--speculative-config '{"model":"RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3","method":"eagle3","num_speculative_tokens":3}'
# 1000/1000 prompts (clean full run), 0 errors, 264.8s. ready after 323s. TTFT median 394.5 ms, TPOT median 65.1 ms, req thr 3.777/s. Watchdog no trip (7.9 GB free).
# SpecDecoding: mean acceptance length 2.06 · avg draft acceptance 35% · per-position ~0.57/0.32/0.17.
Notes
Decode 888.98 tok/s aggregate at concurrency 64. NVFP4 + RedHatAI EAGLE3 speculator, first point of the beyond-c32 ceiling-finding sweep (context held at 65536; splices onto the c1–c32 curve).
- Still climbing: decode 888.98 tok/s vs the c32 point (596.32) = 1.49× per doubling — between base (1.62×) and MTP (1.47×), consistent with its middling acceptance. Sits below base and MTP in absolute terms (lower acceptance → fewer free tokens/step) but scales the same way.
- Memory ~7.9 GB free at peak (110.79 GB delta) — like MTP, the EAGLE3 draft head + per-seq draft buffers make it tighter than base (~9–10 GB free); this matters at c256 (see the collapse note there). Watchdog (floor 3.5 GB) armed, did not trip.
- Acceptance flat and modest: mean acceptance length 2.06, avg draft acceptance 35%, per-position 0.57/0.32/0.17 — identical to c1–c32; EAGLE3’s weaker draft (vs MTP’s ~57%) is a fixed property of this speculator on ShareGPT, not concurrency-driven.
- Image (pinned):
vllm/vllm-openai:nightly-aarch64@sha256:e414712fdc04…. Ready after 323 s. - Workload: ShareGPT V3, concurrency 64. 1000/1000, 0 errors in 264.8 s (no time cap). TTFT median 394.5 ms, TPOT median 65.1 ms, req throughput 3.777/s.
</content> </invoke>