Qwen3.6-27B · vLLM · NVFP4 + MTP · conc 1
status: done
Configuration
| Model | Qwen/Qwen3.6-27B |
|---|---|
| Company | Alibaba |
| Family | Qwen |
| Parameters | 27B (dense) |
| Engine | vLLM + MTP (speculative decoding) |
| Quant / precision | NVFP4 |
| Why this quant | Single-stream (conc-1) point of the Qwen3.6-27B NVFP4 + native-MTP sweep — the best-case MTP latency win (no batch contention), same stack as the conc-32/conc-8 runs. Single-stream decode tok/s + TPOT is where spec-decode helps most. |
| Download | unsloth/Qwen3.6-27B-NVFP4 |
| Context window | 65536 |
| Input modalities | text, image, video (served text-only here) |
Measured results
| Prefill tok/s | 3.35 |
|---|---|
| Decode tok/s | 16.85 |
| Peak memory (GB) | 107.59 (system MemAvailable delta (10s sampling) — vLLM static KV reservation (util 0.85) + MTP head) |
| Completed | 2026-06-23 10:48 +08 |
Full run command
# conc-1 single-stream latency point (200 prompts / 300 s cap).
scripts/bench-vllm-serving.sh unsloth/Qwen3.6-27B-NVFP4 65536 1 200 300 256 \
--trust-remote-code --dtype bfloat16 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
# only 20/200 prompts in the 300 s window (single stream + 515 s cold load). TTFT 336.6 ms, TPOT 51.5 ms.
Notes
conc-1 (single-stream) point of the Qwen3.6-27B NVFP4 + MTP sweep. Best-case MTP latency — no batch contention.
- Result (conc 1): decode 16.85 tok/s single-stream; TTFT median 336.6 ms, TPOT median 51.5 ms (≈19 tok/s/stream — bandwidth-bound for a 27B at ~4-bit on GB10). Peak mem 107.6 GB.
- Small-sample caveat: only 20 prompts completed in the 300 s window (single stream + a 515 s cold load left little time), so acceptance is noisy — windows ranged 53–69%, mean accept-len ~2.6–3.1. Treat as directional; the conc-8/conc-32 runs (134/986 prompts) are the reliable acceptance numbers (~67–71%). The single-stream TPOT above is the solid metric here.