Qwen3.6-35B-A3B · vLLM-ultimate (AEON) · NVFP4 + DFlash · conc 4
status: done
Configuration
| Model | nvidia/Qwen3.6-35B-A3B-NVFP4 |
|---|---|
| Company | Alibaba |
| Family | Qwen |
| Parameters | 35B / 3B (MoE, hybrid GDN+full-attn) + DFlash external drafter |
| Engine | vLLM (aeon-vllm-ultimate custom container, v0.23.0+aeon.sm121a.dflash) + DFlash (z-lab/Qwen3.6-35B-A3B-DFlash @31977fbe small-page rev, num_speculative_tokens 11) (speculative decoding) |
| Quant / precision | NVFP4 (modelopt_mixed — W4A16_NVFP4 experts + FP8 GDN gates) |
| Why this quant | conc-4 fine-grained point of the DFlash money-chart line, tracing the shallow-loss plateau between the 1→2 crossover and the conc-16/32 collapse. REVISED 2026-07-02 — re-run at ctx 65536 (was 40960) as part of the full ctx-matched six-point re-sweep. Same one-boot protocol (official checkpoint, small-page drafter, max-num-seqs 64). SAFETY — untrusted third-party image; NO credentials, weights + drafter READ-ONLY, port loopback-only. |
| Download | nvidia/Qwen3.6-35B-A3B-NVFP4 |
| Context window | 65536 |
| Input modalities | text, image, video (served text-only here) |
Measured results
| Prefill tok/s | 193.44 |
|---|---|
| Decode tok/s | 210.73 |
| Peak memory (GB) | 111.3 (system MemAvailable delta (idle baseline 118 GiB → 6.7 GiB available at load) over the one-boot conc-1/2/4/8/16/32 sweep — vLLM static KV (util 0.85) + DFlash drafter) |
| Completed | 2026-07-02 07:21 +0800 |
Full run command
# UNTRUSTED image — NO creds; weights + drafter READ-ONLY; loopback port. ONE boot (ctx 65536,
# max-num-seqs 64, small-page drafter @31977fbe) sweeping client conc 1/2/4/8/16/32 — see the main
# (c1) page for the full docker run command. Only the client --concurrency differs.
python3 scripts/bench-serving.py --base-url http://127.0.0.1:8000 --model official \
--dataset benchmark_data/ShareGPT_V3_unfiltered_cleaned_split.json --concurrency 4 --num-prompts 1000 --max-seconds 600 --max-tokens 256
# 499/1000 prompts (hit 600 s cap), 0 errors.
Notes
conc-4 point of the Qwen3.6-35B-A3B NVFP4 + DFlash line — REVISED 2026-07-02, now ctx-matched to
MTP/base (65536). Official nvidia/Qwen3.6-35B-A3B-NVFP4 on the AEON image, DFlash n=11 via the
small-page drafter, one-boot sweep. 0 errors.
- Result (conc 4): prefill 193.44 / decode 210.73 tok/s aggregate; 499/1000 prompts (hit the 600 s cap), 0 errors. vs MTP conc-4 (232.43): −9.3%.
- The shallow-loss plateau, slightly steeper once ctx-matched. Across the ctx-matched sweep DFlash sits ~−6 to −12% behind MTP through conc 2/4/8 (−6.1% / −9.3% / −12.0%) — a modest but now monotonically widening loss, not the flatter ~−7-8% plateau the old ctx-40960 numbers showed — before the batch-fill collapse at conc-16 (−21.5%) and conc-32 (−24.8%). So the DFlash penalty is closer to smooth-then-cliff than flat-then-cliff once context is controlled.
- Why — wasted draft compute. DFlash drafts 11 at ~26.5% acceptance / accept-len ~3.92 (~7 wasted forward passes/step); MTP drafts 3 at ~67% / accept-len 3.0. On this compute-bound MoE the extra draft work already dominates by conc-2 and worsens as streams divide the GPU. Acceptance flat vs conc (workload-driven) — matches c1 (26.8%) and c2 (26.7%) within noise.
- One server lifetime for the whole six-point sweep → mem is the single ~111.3 GB reservation. TPOT 0.0 =
qwen3reasoning-parser client artifact. - Series:
c1(main) ·c2·c8·c16·c32. Matched MTP:-mtp-c4.