Nemotron-3 Puzzle 75B-A9B · vLLM · NVFP4 · conc 8
status: done
Configuration
| Model | nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 |
|---|---|
| Company | NVIDIA |
| Family | Nemotron |
| Parameters | 75.3B / 9.3B (MoE, hybrid Mamba-Transformer) |
| Engine | vLLM |
| Quant / precision | NVFP4 |
| Why this quant | Blackwell-native FP4 (W4A4) — GB10 gets native FP4 compute; FP8 targets Hopper, BF16 (~150 GB) won't fit 121 GB. NVIDIA's own quant, ~44.5 GB. Same Nemotron-3-Super family/quant as the 120B sibling (served on cu130-nightly; older images reject its quant_algo whitelist). |
| Download | nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 |
| Context window | 65536 |
| Input modalities | text |
Measured results
| Prefill tok/s | 112.82 |
|---|---|
| Decode tok/s | 94.04 |
| Peak memory (GB) | 108.86 (system MemAvailable delta (10s sampling) — vLLM 0.85 reservation; real NVFP4 weights ~53.5 GB) |
| Completed | 2026-07-11 16:54 +08 |
Full run command
# vllm/vllm-openai:nightly-aarch64 — loads this NVFP4 checkpoint cleanly. ENTRYPOINT ["vllm","serve"].
scripts/bench-vllm-serving.sh nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 \
65536 8 1000 900 256 --trust-remote-code --reasoning-parser nemotron_v3
Notes
Nemotron-3 Puzzle 75B-A9B (NVFP4) — conc 8. Compressed variant of Nemotron-3-Super (75.3B total / 9.3B active, hybrid Mamba-Transformer MoE: 40 Mamba + 40 MoE + 8 attention blocks). Benchmarked as the DGX-Spark-suitable variant (Blackwell-native FP4).
- Load: 535 s on
nightly-aarch64. NVFP4 weights ~53.5 GB. - Workload: ShareGPT V3, concurrency 8. Hit the 15-min cap at 340/1000 prompts, 0 errors (912.6 s).
- Throughput (aggregate, conc 8): prefill 112.82 tok/s, decode 94.04 tok/s; median TTFT 21.1 s. ~4.7× the conc-1 decode (19.84 → 94.04) — near-linear batch scaling on the 9.3B active params. TTFT rises (12.8 s → 21.1 s) as 8 prompts queue their prefills; still 0 errors.