MTP speculation, width-laddered
K is chosen per concurrency (1:3,2:1,4:2,8:2,16:1) rather than
fixed, and self-disables above 32 concurrent sequences where verify cost exceeds the win.
Slide 1 of 0
Verification steps
Everything the front page claims about concurrency, batch sizing and vLLM, restated as commands you can run on your own box. 8 rungs, 1.012× to 1.333×.
atlas ed118b7199 · 2026-09-21
How to read this
Diligence on an inference engine is not a code review. It answers three things, and the third is the one that decides the round.
The claim
A performance claim without every axis named is not yet a claim. This is the whole of ours.
Atlas 1.0.0-beta-preview (build
bce152b18) sustains higher mean decode throughput than vLLM 0.27.1 onunsloth/Qwen3.8-27B-NVFP4, served on one NVIDIA GB10 Grace Blackwell, 121.7 GB unified, at every concurrency C = 1, 2, 4, 8, 16, 32, 64, 128 — ISL 128, OSL 1024, temperature 0, seed 42, mean tok/s over 3 timed reps (1 warmup discarded). Margins run 1.012× to 1.333×.
Every noun in that sentence is a knob someone could have turned to flatter us. The next act hands you each of them.
Scope
The fastest way to waste your week is to test something we never said.
Why the desktop is the right instrument
Measuring on a DGX Spark is not a smaller claim than measuring in a datacenter. It is the rung NVIDIA built for exactly this, and the path off it is theirs, not our extrapolation.
“AI has transformed every layer of the computing stack. It stands to reason a new class of computers would emerge — designed for AI-native developers and to run AI-native applications. With these new DGX personal AI computers, AI can span from cloud services to desktop and edge applications.”
NVIDIA's own framing in the same release: the full-stack platform lets DGX Spark users “seamlessly move their models from their desktops to DGX Cloud or any accelerated cloud or data center infrastructure — with virtually no code changes.”
Reference
Six lines that decide whether anything after them is comparable. If your box differs on any of them, you are measuring something else — fine, but say so.
Reference
The commonest way to manufacture a speedup is to leave one of these unmatched. Ten axes, ten pins — driven by one script, not two.
Reference
vLLM 0.27.1 registers Qwen3_5MTP and this checkpoint ships mtp.* weights, so vLLM can speculate here. Running it without would have been the easy 2×, and a fabricated one.
The earlier reference in this campaign ran speculative decoding off. It understated vLLM badly, so it was replaced and the old column kept in view rather than deleted.
The published table therefore carries two baselines: the matched MTP configuration we claim against, and the unmatched no-speculation leg, labelled as such. At C=128 the unmatched configuration is actually faster than the matched one (vLLM's speculation costs it throughput at high concurrency), so it is plotted, not scored. The headline ratio stays against the matched fingerprint.
Step 1
Five checks, in this order. Every one of them has been the reason a run was thrown away in this campaign, so none of them is ceremony.
nvidia-smi # GB10, driver 580+free -g # ~121 GB unified, not nvidia-smiexport PATH=/usr/local/cuda/bin:$PATHnvcc --version # must report CUDA 13.0docker run --rm --gpus all \ nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smidf -h ~/.cache/huggingface # weights land here, tens of GB
Two GB10 particulars, both of which have cost this campaign time. nvidia-smi reports memory as `Not Supported` — the 121 GB is a unified LPDDR5X pool, so `free` is the instrument. And CUDA ships outside PATH: without that export, `nvcc --version` says command-not-found and the cargo build in Step 2 dies in cudarc's build script rather than anywhere informative. The docker line is the one people skip: it proves the NVIDIA Container Toolkit is wired up, not just installed.
Step 2
The container serves models; the binary measures them. You need both, and the binary has to come from the same tree as the commit you are testing.
git clone https://github.com/Avarok-Cybersecurity/atlas.gitcd atlas && git checkout 4012c9b7e1docker build -f docker/gb10/Dockerfile -t atlas-gb10 .sudo apt-get install -y build-essential pkg-config \ cmake clang libclang-devcargo build --release -p spark-server --bin spark
Both builds run from the repository root, with CUDA still on PATH from Step 1. The multi-target image compiles PTX for every supported model; the first cargo build takes 15–30 minutes for the same reason and leaves 3–5 GB under target/. 4012c9b7e1 is the certified sha rather than bce152b18, the tree the numbers were measured on: that one was a local merge and was never pushed, so it does not exist in your clone. The two differ only in doc comments and gate machinery — no executable change.
./target/release/spark --version./target/release/spark benchmark list./target/release/spark benchmark list concurrency-sweep
The last line prints every parameter of the sweep with its default — the schema the next steps override. If it prints, the toolchain is sound and the rest of this deck will run.
The gate's self-start also reads a cached recipe index at ~/.avarok/atlas-recipes/index.json. Open the TUI library once to
populate it, or Step 6 stops with exactly that message.
Step 3
Pinned by digest, not by tag — “latest” is not a version.
docker run --rm --gpus all --network host \ vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967 \ --model unsloth/Qwen3.8-27B-NVFP4 \ --max-model-len 2048 --max-num-seqs 128 \ --gpu-memory-utilization 0.85 \ --kv-cache-dtype fp8 --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
num_speculative_tokens 3 is K=4 — the same draft width Atlas runs. Context 2048 and batch cap 128 are the pinned pair; changing either invalidates the comparison in both directions.
Step 4
Same box, same checkpoint, same client. Every flag, not the interesting ones — this is the whole certified configuration, rendered from the record the harness wrote.
ATLAS_PREFILL_CODISPATCH=1 ATLAS_FP8_ROWWISE=1 \ATLAS_MTP_DCUT_RATIO=1.0 ATLAS_MTP_K_LADDER=1:3,2:1,4:2,8:2,16:1 \spark serve unsloth/Qwen3.8-27B-NVFP4 --host 0.0.0.0 --port 8888 \ --model-name unsloth/Qwen3.8-27B-NVFP4 --max-seq-len 2048 \ --max-batch-size 128 --gpu-memory-utilization 0.85 \ --kv-cache-dtype fp8 --enable-prefix-caching true \ --ssm-cache-slots 8 --ssm-checkpoint-interval 32 --speculative \ --num-drafts 3 --mtp-quantization bf16 --scheduling-policy fifo \ --tool-call-parser qwen3_coder --disable-tool-grammar true \ --disable-thinking --request-timeout 0 --ssm-h-dtype f16-pool \ --gdn-fused-norm --ssm-batched-recurrent --ssm-tail-midchunk false \ --mtp-gate force --prefill-varlen-batch --no-tui
Do not trim this. Six of these are kernel and scheduling knobs whose defaults are the OPPOSITE of the certified values — ssm-h-dtype, gdn-fused-norm, ssm-batched-recurrent, ssm-tail-midchunk, mtp-gate and prefill-varlen-batch — and serving without them measures a different engine. An abridged version of this command, run on 2026-08-26, landed 4.7% under the published ladder at C=1 and 14% under at C=4, the gap widening with concurrency exactly as those knobs predict. With the full command the same box reproduced every rung to within 2.2%.
Step 5
The subcommand drives an endpoint that is already serving — it neither loads a model nor touches the GPU. So the binary that gates our own pull requests is the binary that measures vLLM. This measures the GATE's workload, not the chart's; Step 5b is the chart.
spark benchmark run concurrency-sweep \ --url http://127.0.0.1:8000 --model unsloth/Qwen3.8-27B-NVFP4 \ --param concurrencies=1,4,8,16 --param isls=512 --param osl=320 \ --skip-coherence-probe --format json > vllm.jsonspark benchmark run concurrency-sweep \ --url http://127.0.0.1:8888 --model unsloth/Qwen3.8-27B-NVFP4 \ --param concurrencies=1,4,8,16 --param isls=512 --param osl=320 \ --format json > atlas.json
http:// only. stdout carries the record and stderr the progress, so the redirect gives a clean file. Those --param values are the ones the gate pins; an unknown key is an error, never a silent no-op.
Step 5b
Step 5 measures what CI gates. This measures what this page publishes — the same driver, at the same shape, that produced every number in the ladder overleaf.
python3 -m venv .venv && .venv/bin/pip install aiohttp.venv/bin/python bench/ladder38/harness_w55_conc_ladder.py \ --url http://127.0.0.1:8888 --model unsloth/Qwen3.8-27B-NVFP4 \ --label atlas --out atlas_ladder.json \ --concs 1,2,4,8,16,32,64,128 \ --reps 3 --isl 128 --osl 1024 --warmup 1
Every knob is a required argument — the driver defaults nothing silently, so the command IS the methodology. Swap --url and --label for the vLLM leg and run them back to back. Budget about an hour per leg on a GB10: C=128 alone streams 131072 tokens per rep, and there are 3 timed reps plus 1 warmup at every rung.
Step 6
The same subcommand in its other mode. This one starts a server: it serves the benchmark's own recipe on a free port, waits up to 900 s for a cold NVFP4 load, and tears it down.
for g in ttft-cold-gate ttft-warm-gate vision-fidelity \ ssm-state-poisoning-gate decode-floor concurrency-sweep \ video-fidelity bfcl-subset bfcl-subset-echolp \ kat-equality-gate agentic-webserver; do timeout 21600 ./target/release/spark benchmark run "$g" \ --pull-request-gate --yesdonespark benchmark --pull-request-gate-check # what CI then runs
One gate per process, which is why it is a loop. Each run writes .benchmarks/<id>/<date>-<sha>.json carrying the metrics, the verdict, the hardware fingerprint, the exact command and the commit sha.
--url and --pull-request-gate are mutually
exclusive by design: a run pointed at someone else's server can be measured and argued about,
but it can never become a record. The CLI draws the line between an experiment and evidence,
so a reviewer does not have to.
Result
8 of 8 rungs, 1.012× to 1.333×. Log2 X, because the rungs double and linear spacing would crush the band where single-stream latency lives.
Batch sizing
The trap that caught the person who wrote the note warning about it — recorded as a correction rather than quietly fixed.
| C=32, same box, same day | cap 32 | cap 128 | effect |
|---|---|---|---|
| Atlas | 277.31 | 278.93 | flat |
| vLLM + MTP | 284.54 | 277.12 | +2.7% at cap 32 |
Lowering the cap to match the rung looks neutral. It is not: vLLM sizes its KV blocks and
scheduler budget from max_num_seqs, so a smaller cap changes
allocation, preemption and prefix reuse — and materially favours it.
A matched cap-32 pair, adopted in good faith to dodge a hardware hazard, produced 0.975× and inverted the true ordering. Measured again at the certified cap-128 configuration on the same box the same day: 1.007×, with non-overlapping distributions — Atlas's worst rep beat vLLM's best.
The certified table pins cap 128 on both engines at every rung, independently of the concurrency being driven. That pin is load-bearing.
If you re-cap to survive a hazard on your own box: say so beside the number, and re-pin both engines to the same value. Do not fold it into the certified column.
Mechanism
Ask this before the numbers. A speedup with no stated mechanism and no known ceiling is a configuration artefact waiting to be found.
K is chosen per concurrency (1:3,2:1,4:2,8:2,16:1) rather than
fixed, and self-disables above 32 concurrent sequences where verify cost exceeds the win.
Prefill overlaps decode rather than stalling it; row-wise fp8 moves fewer bytes on the bandwidth-bound path. Decode at this scale is a memory-traffic problem, so this is where a real win has to come from.
It is not that Atlas gets faster — it is that vLLM's C=128 falls below its own C=64 when speculation stays on at high concurrency. Atlas's ladder has already switched it off. The margin is a scheduling decision, and it is reproducible for that reason.
Self-audit
Heiser's benchmarking-crimes taxonomy, run against our own campaign. Three rows stay open, because a checklist with nothing open is marketing.
Artifacts
A reproduction that needs a conversation with us is not one. Everything below is already in the repository, at the commit the numbers were taken on.
Method
RESULTS.md runs past 1,500 lines and reads as a lab notebook: dated rounds, hypotheses, and the ones that died.
decode_tps rejected as a gate.Durability
A margin that is only ever measured by hand decays silently. These are the mechanisms that make a regression loud.
Ten gates are required for a pull request to land, and thresholds are not a percentage
band: they are absolute per-metric floors committed in kernels/gb10/<model>/BENCH.toml, each with an explicit noise slack. CI refuses a slack above 5% of its own bound — larger
than that is a threshold change wearing a measurement-noise costume.
A record is voided by content, not ancestry. Eight paths invalidate one: crates/, kernels/, Cargo.toml, Cargo.lock, vendor/, 3rdparty_patches/, rust-toolchain.toml — and jinja-templates/,
which is runtime input rather than build input: the server loads one over the checkpoint's own
chat template, so editing it changes the bytes every prompt renders to. A dirty tree fails, and
an entry declaring no thresholds fails rather than passes.
| concurrency-sweep floor | min tok/s | noise |
|---|---|---|
| c1_aggregate_tok_s | 17.0 | 0.8 |
| c4_aggregate_tok_s | 35.0 | 1.5 |
| c8_aggregate_tok_s | 52.0 | 1.5 |
| c16_aggregate_tok_s | 73.5 | 1.5 |
| peak_aggregate_tok_s | 73.5 | 1.5 |
| vacuous_cells | max 0 | — |
Calibrated at mean minus max(3σ, ~5%) from three fresh reps on the same instrument, never from a best rep — the derivation is a comment beside each floor.
IP and licensing
The first question an investor's counsel asks about a serving engine is the licence, so here it is with the machinery that keeps it honest.
Engineering
House rules only count if a machine says no. These do.
file-size-cap.ymlSECURITY.mdOver to you
The strongest thing we can hand a technical analyst is not a chart. It is the box, the commands, and a file that already records what we got wrong.
spark benchmark list concurrency-sweepspark benchmark run concurrency-sweep --url YOUR_VLLM_URL --model CHECKPOINT \ --param concurrencies=1,4,8,16 --param isls=512 --param osl=320spark benchmark run concurrency-sweep --pull-request-gate --yes
The first line prints every parameter and its default. The second measures whatever you already have serving. The third is the one CI runs. Expect ~2 hours on a single GB10 including the model download.
Every number in this deck is read from the same generated records the front page renders. If the ladder is re-run and a rung is lost, these slides say so on the next build.