The test bed
One harness (bench.py) talks to each model through its server's OpenAI-compatible API, sends the same prompts, grades the answers automatically, and saves every prompt, reasoning trace, answer, verdict, token count and latency. Everything here is reproducible from the repository; the scripts that ran are in runs/.
Models and how they were served
| Model | Weights | Server | Speculative decoding | Reasoning setting |
|---|---|---|---|---|
| Muse-Glimmer-30B | unsloth Q4 GGUF (4-bit dynamic); also NVIDIA NVFP4 for the quantization check | llama.cpp (vLLM for NVFP4) | DFlash drafter (Meta's), llama.cpp only | system prompt "Reasoning strength: high", top_k 64 |
| Qwen3.6-27B | unsloth Q4 GGUF | llama.cpp | none (the GGUF has no MTP head) | default thinking mode, top_k 20 |
| Qwen3.8-27B | unsloth NVFP4, the exact checkpoint the Spark serves | vLLM 0.29 (the Spark: 0.26.1) | MTP head, 3 tokens (the Spark's recipe); also run without | default thinking mode, top_k 20 |
Sampling for everyone: temperature 1.0, top_p 0.95, both vendors' recommendation; seed 1234. Reasoning is returned separately by the servers and is never graded, only counted. Hardware: two RTX A6000 (48 GB), one model per GPU; plus the team's DGX Spark over the network for one run.
Tasks
| Task | What it tests | Items | Grading |
|---|---|---|---|
| GSM8K | grade-school math word problems | 200 (fixed sample of the 1,319-item test set) | last "Final answer: N" equals the reference |
| MMLU-Pro | 10-option multiple choice, 14 subjects | 200 (fixed sample of ~12k) | last "Answer: X" equals the reference letter |
| HumanEval | 164 Python functions | all 164 | the problem's unit tests are executed; all must pass |
| GPQA Diamond | graduate-level physics, chemistry, biology, 4 options | all 198 | last "Answer: X" or \boxed{X} |
| LiveCodeBench v6 | competitive programming, contests Jan–Apr 2025 (after the models' training data) | all 175 | the program is run on every public and private test (~40 per problem); all must pass |
Zero-shot, one chat turn per item, our own prompt wording. Dataset revisions are pinned in bench.py (Hugging Face commit SHAs), so a re-run sees the same items.
Datasets, papers, code
| Benchmark | Dataset (the one we run) | Paper | Code |
|---|---|---|---|
| GSM8K | openai/gsm8k | Training Verifiers to Solve Math Word Problems (OpenAI, 2021) | openai/grade-school-math |
| MMLU-Pro | TIGER-Lab/MMLU-Pro | MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024) | TIGER-AI-Lab/MMLU-Pro |
| HumanEval | openai/openai_humaneval | Evaluating Large Language Models Trained on Code (OpenAI, 2021; the Codex paper) | openai/human-eval |
| GPQA Diamond | hendrydong/gpqa_diamond_mc (ungated mirror of the 198-item Diamond split) | GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023) | idavidrein/gpqa |
| LiveCodeBench | livecodebench/code_generation_lite, release v6 (test6.jsonl) | LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (2024) | LiveCodeBench/LiveCodeBench |
Why these: GSM8K and HumanEval have known expected scores and validate the harness; MMLU-Pro, GPQA Diamond and LiveCodeBench have headroom for models of this class and cover knowledge, science and code. AIME was considered and dropped: vendors report 94–95% for these models. None is an agentic workflow. All three grading methods are automatic and objective (exact match, executed tests); no human or model judge.
The token budget
These are reasoning models: they generate thinking tokens before the answer, as many as they want. Every evaluation caps that, and so do the vendors (Qwen reports LiveCodeBench with up to 80k tokens). We used 12,288 tokens per answer for the first round and 32,768 for the second. An answer that hits the cap without a final answer counts as wrong, and is reported separately as "not finished". The cap turned out to be the variable that decides the ranking, so both budgets are shown wherever they exist.
Speed
- Decode: one short prompt, 512 tokens, streaming, 3 sequential requests; tokens ÷ (time from first to last token); median.
- Prefill: prompts of ~1k, ~4k and ~12k tokens with max_tokens=1; prompt tokens ÷ time to first token; 3 repetitions, each with a unique prefix so a prefix cache cannot serve a repeat; median.
- Prefix cache: the same prompt sent twice; the second request's time and the server's hit counter tell whether the cache served it. Five usage patterns; the server checked idle before each call.
- Token counts always come from the server's usage field, never estimated.
Draft-vocabulary coverage (Spark, run 10)
The Spark's Flash-Next image runs MTP speculative decoding with a drafter restricted to 65,536 token ids. To estimate what that restriction costs, tools/draft-vocab-coverage.py loads the id list (tools/draft_vocab_65536.npy), tokenizes every saved reasoning trace and final answer of every run with the Flash-Next tokenizer, and reports the share of tokens inside the set, per run and task, plus the most frequently missed ids. Because the target model verifies every drafted token against its full output head, a token outside the set can only lower acceptance, never change the answer, so coverage bounds the possible loss. The Spark run itself was taken with tools/spark-idle-run.sh, which refuses to start unless the server is idle, samples its request count every 15 s, and snapshots /metrics before and after so the acceptance figures belong to that run alone (tools/metrics-delta.py).
Corrections made along the way
- HumanEval scorer: the first pass did not keep the problem's helper functions in scope; 4 Muse solutions failed with NameError. Fixed, and every saved answer re-graded. All later runs used the fixed scorer.
- LiveCodeBench scorer: an injected
from math import *shadowed the built-inpow(a, b, mod); programs now run exactly as written. And the judge environment: 23 Muse solutions used numba, which the real AtCoder judge provides; scores are reported both with the standard library only (strict) and with numpy/scipy/numba available (judge env, the headline). Saved answers were re-graded under both, nothing regenerated. - Every wrong answer of every run was classified (wrong / not finished / parser miss / harness error). Parser misses: zero in all runs.
What is and is not reproducible
- Pinned: container images (by digest), Python packages, dataset revisions, model snapshots with file hashes, item samples (seed).
- Not deterministic: sampling at temperature 1.0 and GPU kernels. The same Qwen3.8 checkpoint scored 74.5 / 77.0 / 76.0 on MMLU-Pro in three runs; read every difference against that ±1.5.
- Repository:
README.md(reproduction),results.md(full report),MODELS.md,docs/spark-measurements.md(the command behind every Spark number).