gpt-oss-20b · llama.cpp · MXFP4
status: blocked
Configuration
| Model | openai/gpt-oss-20b |
|---|---|
| Company | OpenAI |
| Family | gpt-oss |
| Parameters | 21B / 3.6B (MoE) |
| Engine | llama.cpp |
| Quant / precision | MXFP4 |
| Why this quant | gpt-oss's native FP4 format; the ggml-org GGUF is a single 12 GB MXFP4 file. |
| Download | ggml-org/gpt-oss-20b-GGUF |
| Context window | 131072 |
| Input modalities | text |
Measured results
| Prefill tok/s | — |
|---|---|
| Decode tok/s | — |
| Peak memory (GB) | — |
| Completed | 2026-06-21 |
Full run command
# Model loads & generates via raw /completion, but the OpenAI /v1/chat/completions
# endpoint cannot be benchmarked — see below.
Notes
BLOCKED — llama.cpp can’t serve gpt-oss’s harmony chat format on this build.
The GGUF downloads and loads fine, and raw text completion works
(/completion "The capital of France is" → “Paris…”). But every request to the
OpenAI chat endpoint (/v1/chat/completions, which the ShareGPT harness and every
other engine use) fails mid-stream with:
500 — The model produced output that does not match the expected peg-native format
This is llama.cpp’s harmony response-format parser rejecting the model’s channel
output. It persists with --jinja and --reasoning-format none (build b9744-063d9c156).
So no comparable OpenAI-chat throughput number can be produced on this engine right now.
This is consistent with the Spark guidance: gpt-oss’s recommended engines are SGLang
(claimed SOTA, ~70 tok/s decode on 20b) and vLLM, not llama.cpp. The benchmark for
gpt-oss-20b will be produced on vLLM / SGLang (see the gpt-oss-20b · vLLM · MXFP4
config) once that engine is stood up. Revisit llama.cpp if a later build fixes harmony chat
parsing (raw /completion is a fallback but isn’t OpenAI-chat-comparable and omits usage tokens).