Qwen3.6-27B · vLLM · NVFP4
status: done
Configuration
| Model | Qwen/Qwen3.6-27B |
|---|---|
| Company | Alibaba |
| Family | Qwen |
| Parameters | 27B (dense) |
| Engine | vLLM |
| Quant / precision | NVFP4 |
| Why this quant | Unsloth's NVFP4 (W4A4) quant of Qwen3.6-27B (unsloth/Qwen3.6-27B-NVFP4), calibrated on HF UltraChat @16K. NVFP4 is a genuine GB10 fast-path — beat the official FP8 base by ~21% decode here. Base (non-speculative) config; the repo's MTP module is exercised in the -mtp sibling. |
| Download | unsloth/Qwen3.6-27B-NVFP4 |
| Context window | 65536 |
| Input modalities | text, image, video (served text-only here) |
Measured results
| Prefill tok/s | 181.8 |
|---|---|
| Decode tok/s | 187.74 |
| Peak memory (GB) | 108.18 (system MemAvailable delta (10s sampling) — vLLM static KV reservation (util 0.85)) |
| Completed | 2026-06-23 09:54 +08 |
Full run command
# NVFP4 (modelopt_fp4 auto-detected from the unsloth checkpoint) on vLLM nightly-aarch64. Base, conc-32.
scripts/bench-vllm-serving.sh unsloth/Qwen3.6-27B-NVFP4 65536 32 1000 900 256 \
--trust-remote-code --dtype bfloat16
# 683/1000 prompts, 0 errors, hit the 900 s time cap (the cold load took 648 s — download + compile —
# leaving < 5 min of the window for prompts). TTFT median 796.9 ms, TPOT median 155.1 ms.
Notes
The NVFP4 fast-path pays off on the 27B too — ~21% faster decode than the official FP8 base. Unsloth NVFP4 (W4A4) of Qwen3.6-27B, served text-only on vLLM (the card’s recommended path), at conc-32.
- Result (conc 32): decode 187.7 tok/s vs the [FP8 base]’s 154.7 (+21%); prefill 181.8 vs 168.9. Peak mem 108.2 GB (vLLM static KV reservation at util 0.85), ~same as FP8 (107.5) — NVFP4 saves on weights but the KV reservation dominates the headline. 0 errors.
- Time cap: the run hit the 900 s cap at 683/1000 prompts — the cold start took 648 s (first-time ~15 GB download + torch.compile), leaving little of the window. The tok/s is steady-state and comparable, but the entry count is capped; the MTP/SGLang siblings reuse the cached weights and run the full count. (Flagged per the run-cap policy.)
- Repo choice — unsloth (no NVIDIA option). Policy prefers an official
nvidia/NVFP4 when one exists, but NVIDIA publishes none for the 27B (only for the 35B-A3B sibling); the 27B has only community NVFP4 quants (unsloth, mmangkad, sakamakismile, …). So this uses unsloth (a trusted, well-known quantizer per the repo policy). - Pair: base (this) +
qwen3-6-27b-nvfp4-vllm-mtp(with MTP). SGLang siblings under*-sglang*.