Qwen3.6-27B · vLLM · FP8 + MTP
status: done
Configuration
| Model | Qwen/Qwen3.6-27B |
|---|---|
| Company | Alibaba |
| Family | Qwen |
| Parameters | 27B (dense) |
| Engine | vLLM + MTP (speculative decoding) |
| Quant / precision | FP8 |
| Why this quant | Qwen3.6-27B FP8 + the model's own native MTP module (mtp.safetensors ships in the base repo) — built-in multi-token-prediction speculative decoding, no separate draft. |
| Download | Qwen/Qwen3.6-27B-FP8 |
| Context window | 65536 |
| Input modalities | text, image (served text-only here) |
Measured results
| Prefill tok/s | 241.68 |
|---|---|
| Decode tok/s | 240.92 |
| Peak memory (GB) | 107.09 (system MemAvailable delta (10s sampling) — vLLM static KV reservation (util 0.85) + MTP head) |
| Completed | 2026-06-22 18:46 +08 |
Full run command
# base Qwen3.6-27B-FP8 + native MTP (mtp.safetensors ships in-repo) via vLLM --speculative-config.
docker run -d --gpus all --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface --env HF_TOKEN=*** \
vllm/vllm-openai:cu130-nightly Qwen/Qwen3.6-27B-FP8 \
--host 0.0.0.0 --port 8000 --max-model-len 65536 --gpu-memory-utilization 0.85 --max-num-seqs 32 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
python3 scripts/bench-serving.py --base-url http://localhost:8000 --model Qwen/Qwen3.6-27B-FP8 \
--dataset benchmark_data/ShareGPT_V3_unfiltered_cleaned_split.json \
--num-prompts 1000 --max-seconds 900 --concurrency 32 --max-tokens 256
Notes
Native MTP is a clean +56% win on Qwen3.6-27B — and the first config with a captured acceptance rate, which explains exactly why. Qwen3.6-27B FP8 + the model’s built-in MTP head, on vLLM.
- Workload: ShareGPT V3, concurrency 32. 867/1000, 0 errors — clean, hit the cap. Loaded + CUDA-graph captured in 504 s (MTP adds compile time).
- Throughput (aggregate, conc 32): prefill 241.7 tok/s, decode 240.9 tok/s vs the base Qwen3.6-27B’s 154.7 → +56% decode from MTP at conc 32.
- Acceptance rate (captured from vLLM’s
SpecDecoding metrics): mean acceptance length ≈ 3.0, avg draft acceptance ≈ 67%, per-position 0.84 / 0.67 / 0.51 (for the 3 draft positions). Withnum_speculative_tokens=3, ~3 tokens are emitted per target step on average (the always-accepted token + ~2 accepted drafts). High acceptance on ShareGPT’s fairly predictable chat continuations is what converts into the +56%. - Why this won where gpt-oss-120b’s EAGLE3 lost — same conc 32, opposite size. A 27B model still has GB10 compute headroom at batch 32, so the MTP draft+verify rides on otherwise-idle FLOPs and the ~3× tokens/step turns into real throughput. The 120B gpt-oss was already compute-saturated, so the same idea went −45%. Acceptance ~67% here is healthy; even with good acceptance, a saturated large model wouldn’t benefit. Headroom × acceptance is the predictor — and this 27B has both.
- Memory: 107.1 GB is the vLLM
--gpu-memory-utilization 0.85reservation (+ the small MTP head), not the footprint (FP8 weights ≈ 27 GB).