gpt-oss-20b · vLLM · MXFP4 + EAGLE3 · conc 2
status: done
Configuration
| Model | openai/gpt-oss-20b |
|---|---|
| Company | OpenAI |
| Family | gpt-oss |
| Parameters | 21B / 3.6B (MoE) |
| Engine | vLLM + EAGLE3 (speculative decoding) |
| Quant / precision | MXFP4 |
| Why this quant | conc-2 point of the EAGLE3 fine-grained sweep (EXPERIMENTS.md |
| Download | openai/gpt-oss-20b |
| Context window | 65536 |
| Input modalities | text |
Measured results
| Prefill tok/s | 67.51 |
|---|---|
| Decode tok/s | 58.98 |
| Peak memory (GB) | 108.43 (system MemAvailable delta (10s sampling) — vLLM static KV reservation (util 0.85) + EAGLE3 head) |
| Completed | 2026-07-01 16:02 +0800 |
Full run command
# conc-2 EAGLE3, cu130-nightly + RedHatAI speculator. Harmony vocab via VOCAB_DIR override.
VLLM_IMAGE=vllm/vllm-openai:cu130-nightly VOCAB_DIR=$HOME/tiktoken_encodings \
scripts/bench-vllm-serving.sh openai/gpt-oss-20b 65536 2 1000 600 256 \
--speculative-config '{"model":"RedHatAI/gpt-oss-20b-speculator.eagle3","method":"eagle3","num_speculative_tokens":3}'
# 141/1000 prompts (hit 600 s cap), 11 harmony errors. ready after 195 s.
Notes
conc-2 EAGLE3 — spec-decode is a NET LOSS here, confirming the “+28%@c32 is an artifact” thesis. gpt-oss-20b MXFP4 + RedHatAI EAGLE3 speculator on vLLM, conc 2.
- Result (conc 2): prefill 67.51 / decode 58.98 tok/s aggregate; 141/1000 prompts (hit the 600 s cap), 11 harmony errors; peak mem 108.4 GB.
- EAGLE3 vs base at conc-2: −29.3% (58.98 vs base 83.37). The draft doesn’t just fail to help — it actively hurts at low batch, because the wasted draft/verify work isn’t offset by any acceptance. This is the decisive evidence for #15: if the conc-32 “+28%” came from acceptance, spec would help here too; instead it’s strongly negative, so the c32 win is a scheduling/prefill effect at that one batch size, not a real draft win.
- Acceptance is dismal (as at c1/c8): mean accept-len ~1.1–1.45 (≈1 extra token per 3 drafted), avg draft acceptance ~3–15% (centered ~6%), per-position collapsing to near-zero by slot 3. Far below EAGLE3’s ~3.0 / ~70% expectation — ShareGPT general chat + gpt-oss’s harmony reasoning channel is off-distribution for this draft (see notes/INCOMPATIBILITIES.md).
- TTFT/TPOT are buffered-reasoning artifacts — aggregate decode tok/s is the valid metric.
- Sweep:
-c1·-c4·-c8·-c16·c32(main). Base:-c2.