Qwen3.8-27B · vLLM · NVFP4 + MTP · conc-16
status: done
Configuration
| Model | Qwen/Qwen3.8-27B |
|---|---|
| Company | Alibaba |
| Family | Qwen |
| Parameters | 27B (dense, hybrid linear-attn) |
| Engine | vLLM + MTP (speculative decoding) |
| Quant / precision | NVFP4 |
| Why this quant | Same unsloth NVFP4 checkpoint plus its in-repo BF16 MTP head, at conc-16 — the midpoint of the decaying half of the speculative-speedup-vs-batch curve. |
| Download | unsloth/Qwen3.8-27B-NVFP4 |
| Context window | 65536 |
| Input modalities | text, image, video (served text-only here) |
Measured results
| Prefill tok/s | 245.47 |
|---|---|
| Decode tok/s | 194.99 |
| Peak memory (GB) | 105.69 (system MemAvailable delta (10s sampling) — vLLM static KV reservation (util 0.85) + MTP head) |
| Completed | 2026-08-21 02:33 +0800 |
Full run command
# conc-16 point with the in-repo MTP head. Same recipe as conc-32 except --max-num-seqs.
scripts/bench-vllm-serving.sh unsloth/Qwen3.8-27B-NVFP4 65536 16 1000 900 256 \
--trust-remote-code --dtype bfloat16 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
# 744/1000 prompts in the 900 s window (vs 499 for the non-spec baseline), 0 errors. ready after 383 s.
# TTFT median 616.7 ms, TPOT median 76.8 ms.
Notes
+52% decode at conc-16 — still worth it, but the batch is now doing half the work.
- Result (conc 16): decode 194.99 tok/s vs the conc-16 base’s 128.60 — +51.6%. TPOT median 118.7 → 76.8 ms (1.55× faster per token). Prefill 245.5 vs 149.0 (1.65×). 744/1000 completed vs 499 in the same window. 0 errors, load 383 s.
- Fits the decay curve exactly. +91% (c1) → +94% (c2) → +83% (c4) → +69% (c8) → +52% (c16) → +31% (c32). Once past the low-batch plateau the speedup falls roughly linearly in log₂(batch), losing ~14 points per doubling. Speculation and batching are two ways to spend the same idle compute; the more of it the batch consumes, the less is left for drafted tokens.
- MTP still beats a batch doubling on latency. c16+MTP does 195.0 tok/s at 76.8 ms TPOT; the non-speculative c32 does a comparable 210.0 tok/s but at 144.1 ms TPOT. For the same rough throughput, MTP at half the batch gives users ~1.9× better per-token latency — the clearest argument on this sweep for spending memory on a draft head instead of on more concurrent slots.
- Acceptance ~57% (last-20 samples 51.2–63.8%, median 56.8%), mean acceptance length ~2.72 of a
possible 4.0, per-position 0.76 / 0.55 / 0.41. Six MTP runs on this model now span ~52–58%
with no trend in concurrency — acceptance is workload-driven, per
notes/BENCHMARKING.md. - TTFT cost: 374.1 → 616.7 ms (+65%), the largest relative hit so far apart from c32’s +88%. The constant per-request draft overhead is now landing on top of a queue deep enough to amplify it.
- Memory 105.7 GB, mid-band for the sweep.