Qwen3.6-35B-A3B · vLLM-ultimate (AEON) · NVFP4 + DFlash · conc 2
status: done
Configuration
| Model | nvidia/Qwen3.6-35B-A3B-NVFP4 |
|---|---|
| Company | Alibaba |
| Family | Qwen |
| Parameters | 35B / 3B (MoE, hybrid GDN+full-attn) + DFlash external drafter |
| Engine | vLLM (aeon-vllm-ultimate custom container, v0.23.0+aeon.sm121a.dflash) + DFlash (z-lab/Qwen3.6-35B-A3B-DFlash @31977fbe small-page rev, num_speculative_tokens 11) (speculative decoding) |
| Quant / precision | NVFP4 (modelopt_mixed — W4A16_NVFP4 experts + FP8 GDN gates) |
| Why this quant | conc-2 fine-grained point of the DFlash money-chart line. REVISED 2026-07-02 — re-run at ctx 65536 (was 40960) as part of the full ctx-matched six-point re-sweep, removing the context confound vs the matched MTP series. Same one-boot protocol (official checkpoint, small-page drafter, max-num-seqs 64). SAFETY — untrusted third-party image; run with NO credentials, official weights + drafter READ-ONLY, port loopback-only. |
| Download | nvidia/Qwen3.6-35B-A3B-NVFP4 |
| Context window | 65536 |
| Input modalities | text, image, video (served text-only here) |
Measured results
| Prefill tok/s | 168.05 |
|---|---|
| Decode tok/s | 151.3 |
| Peak memory (GB) | 111.3 (system MemAvailable delta (idle baseline 118 GiB → 6.7 GiB available at load) over the one-boot conc-1/2/4/8/16/32 sweep — vLLM static KV (util 0.85) + DFlash drafter) |
| Completed | 2026-07-02 07:21 +0800 |
Full run command
# UNTRUSTED image — NO creds; official weights + drafter READ-ONLY; loopback port. ONE boot (ctx 65536,
# max-num-seqs 64, small-page drafter @31977fbe) sweeping client conc 1/2/4/8/16/32 — see the main
# (c1) page for the full docker run command. Only the client --concurrency differs.
python3 scripts/bench-serving.py --base-url http://127.0.0.1:8000 --model official \
--dataset benchmark_data/ShareGPT_V3_unfiltered_cleaned_split.json --concurrency 2 --num-prompts 1000 --max-seconds 600 --max-tokens 256
# 356/1000 prompts (hit 600 s cap), 0 errors.
Notes
conc-2 point of the Qwen3.6-35B-A3B NVFP4 + DFlash line — REVISED 2026-07-02, now ctx-matched to
MTP/base (65536). Official nvidia/Qwen3.6-35B-A3B-NVFP4 on the AEON image, DFlash n=11 via the
small-page drafter, one-boot sweep. 0 errors.
- Result (conc 2): prefill 168.05 / decode 151.3 tok/s aggregate; 356/1000 prompts (hit the 600 s cap), 0 errors. vs MTP conc-2 (161.21): −6.1%.
- The crossover is already behind by conc-2, ctx-matched. DFlash’s conc-1 point is now a wash (+0.7%
vs matched-cap MTP — see
c1(main)), so the sign flip effectively happens at conc-1→2 rather than DFlash holding a real lead first. The old ctx-40960 measurement (147.61 decode, same absolute number as this run within noise) read −8.4% against the same MTP baseline — the ctx fix barely moved this point (−8.4% → −6.1%), confirming most of the c2 loss was real, not a context artifact. - Why — wasted draft compute. DFlash drafts 11 tokens at only ~27% acceptance / accept-len ~3.94 → ~7 wasted drafter forward passes per step. Even at conc-2 that extra compute already outweighs the bandwidth saving; MTP’s 3-token / ~67%-accept / accept-len-3.0 draft has almost no waste. Acceptance is workload-driven (flat vs conc — matches c1’s 26.8% almost exactly).
- One server lifetime for the whole six-point sweep → mem is the single ~111.3 GB reservation
(max-num-seqs 64). TPOT 0.0 is the
qwen3reasoning-parser client artifact — decode tok/s is the real number. - Series:
c1(main) ·c4·c8·c16·c32. Matched MTP:-mtp-c2.