Qwen3.6-27B · vLLM · NVFP4 + MTP · conc 8
status: done
Configuration
| Model | Qwen/Qwen3.6-27B |
|---|---|
| Company | Alibaba |
| Family | Qwen |
| Parameters | 27B (dense) |
| Engine | vLLM + MTP (speculative decoding) |
| Quant / precision | NVFP4 |
| Why this quant | Concurrency-8 point of the Qwen3.6-27B NVFP4 + native-MTP sweep — same stack as the conc-32 run (unsloth NVFP4 base + in-repo MTP), lower batch for the latency-characterization point. Acceptance should hold ~constant vs conc-32 (workload-driven); what changes is whether the MTP speedup materializes. |
| Download | unsloth/Qwen3.6-27B-NVFP4 |
| Context window | 65536 |
| Input modalities | text, image, video (served text-only here) |
Measured results
| Prefill tok/s | 120.27 |
|---|---|
| Decode tok/s | 109.05 |
| Peak memory (GB) | 106.65 (system MemAvailable delta (10s sampling) — vLLM static KV reservation (util 0.85) + MTP head) |
| Completed | 2026-06-23 10:33 +08 |
Full run command
# conc-8 latency point (500 prompts / 300 s cap, matching the FP8-MTP -c8 convention).
scripts/bench-vllm-serving.sh unsloth/Qwen3.6-27B-NVFP4 65536 8 500 300 256 \
--trust-remote-code --dtype bfloat16 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
# 134/500 prompts (hit 300 s cap), 0 errors. TTFT median 579.5 ms, TPOT median 60.8 ms.
Notes
conc-8 point of the Qwen3.6-27B NVFP4 + MTP sweep. Lower-batch latency characterization (cap 500 prompts / 300 s — reached 134 at the time cap, 0 errors).
- Result (conc 8): prefill 120.3 / decode 109.05 tok/s aggregate; per-stream TTFT median 579.5 ms, TPOT median 60.8 ms (≈16 tok/s/stream). Peak mem 106.7 GB.
- Acceptance ~71% / mean accept-len 3.1 (per-position 0.87 / 0.69 / 0.55) — holds vs the conc-32 run’s 67% (even a touch higher), confirming acceptance is workload-driven, not concurrency-driven. Aggregate tok/s is lower than conc-32 (274) purely because 8 streams push fewer total tokens — the meaningful low-conc metric is the per-stream TPOT above.