Everything that lied to us

Every finding on this site came from a machine where a benchmark can be wrong in at least six ways without anything failing. No exception was thrown, no HTTP error returned, no log line printed. In each case the number looked entirely reasonable. That is the hazard: a broken measurement does not announce itself, it just quietly reads a bit high or a bit low, and you publish it.

Twelve ways to measure the wrong thing

Each of these cost real time here. They are ordered by how much damage they did.

Know your noise floor — it is three different numbers

“Noise” here is not one quantity, and conflating the three leads to both false positives and false confidence.

The instrument can be the error, not the reading

Every trap above is a way of measuring the right quantity badly. This one is worse: measuring a quantity that was never the one in question, carefully, and publishing it.

We wanted to know whether a quantized KV cache changes what the model believes at depth. What we measured for a week was free-running greedy divergence: generate from the same prompt under bf16 and under fp8, count characters until the two outputs differ. It is a natural thing to measure and it is almost useless, for two reasons.

• It compounds. Every generated token feeds the next. One near-tie flipped by rounding at character 105 sends the two continuations down permanently different paths. The number records where the first coin-flip landed, not how wrong the cache is — and it will land early whatever the cache does, because there is always a near-tie early.
• It cannot reach the depths that matter. It is defined only where the bf16 reference exists. The fp8 and KVarN profiles exist for 150K and 200K precisely because bf16 cannot go there, so the instrument is blind at exactly the depths the profiles are for.

Above the reference ceiling the trick is to stop looking for ground truth and compare independent quantizations against each other. Two schemes that fail differently will disagree; two that are both faithful will agree. fp8 and KVarN agree to 0.005 nats on the model’s own text at 150K, which is the strongest statement available about a depth where nothing can be checked directly.

What a 27B model does when you ask it for 200,000 tokens

The fidelity study needed documents the model could not have memorised, so we had it write three of them — a novel, an engineering design document and an essay sequence, at 200K tokens each, by chaining continuation requests. That turned into a study of its own, because a model asked for that much text does not fail by producing nonsense. It fails by winding down into one of four modes, each of which reads fine for a paragraph.

All four are self-seeding: leave one in the document and the next continuation, which is handed that tail as context, does more of it. So detection alone is not enough — the trailing run has to be trimmed off before the text is kept, or the mode propagates. The detector that resulted rejects a chunk on internal n-gram repetition, distinct-word ratio, mean sentence length, one-line paragraph fraction, and overlap with earlier text, and separately trims chant and anaphora off the tail.

The instrument

These lessons were consolidated into a benchmark framework, so that testing the next model is a config file rather than a re-derivation of method. Six axes, three tiers, one YAML per model.

Acceptance sits on the speed axis deliberately: it is the causal variable behind most of the tok/s spread. The gap between 144 tok/s on coding and 105 on prose was an acceptance difference, not a hardware one. And correctness is the one axis a speculator change cannot move — speculative decoding is mathematically exact — which makes an unchanged score across speculators a useful self-test of the harness.

The fairness contract

Eleven conditions are hashed into a fingerprint attached to every result. The comparison tool refuses to place two cards side by side when their fingerprints differ, naming every field that diverged.

conditions.power_cap_w        250
conditions.single_tenant      true
conditions.thinking_disabled  true
conditions.prefix_caching     0
conditions.temperature        0
conditions.max_tokens         256
conditions.corpus_sha         eb13204dad3d3188
conditions.edit_format        diff
conditions.concurrency_levels [1, 4, 16, 32, 64]
conditions.endpoint_mode      pinned-replica
harness.framework             1.0.0
                              -> fingerprint cdf5261ea5eaca65

The gate itself needed testing. An early version printed “untrustworthy cards”, then printed “comparable — identical fingerprint”, then exited zero — because the fingerprint covers conditions, not validity. A card measured against an endpoint returning empty content passed the strict gate cleanly. Testing a defence against the actual failure it defends against is not optional.

Silent failures, encoded as preflight checks

The six hazards above are executable checks rather than prose. preflight.sh fails on: reasoning detected active (probing for empty content or a stray reasoning field), client_max_body_size unset, a power cap above expectation, or missing measurement scripts. It warns on least_conn load balancing, which biases traffic toward the faster replica, and on foreign processes holding a GPU. A blocked run cannot produce a card unless forced, and a forced card is stamped untrustworthy.

Why none of this is comparable to a leaderboard

The figures on this site measure one deployment, not a model's capability. Four things differ from any published number, each moving results independently:

• Quantization — a 4-bit checkpoint with a further-quantized lm_head, not the released weights
• Harness version — a specific commit, whose prompts and scoring differ from other commits
• Reasoning disabled — deliberately, for comparability with our own earlier runs
• Serving stack — a community fork with performance patches upstream does not have

The card died twice, and the second one broke our explanation

GPU 0 fell off the PCIe bus twice, seven days apart, with an identical signature both times:

Aug 22 16:24:03  NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
Aug 29 21:49:55  NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
                 NVRM: Xid 154, GPU recovery action ... 0x1 (GPU Reset Required)

There were no AER errors in either boot — no correctable or uncorrectable PCIe errors, nothing preceding the fault. It fails clean, without warning. And Xid 79 with no AER is consistent with either a power transient or a link fault, so these logs cannot separate the two, and we are not going to pretend otherwise.

What can be said precisely: a power cap bounds average board draw and does nothing about microsecond transients. And the workload present at the second fault, tensor parallelism, is uniquely bad in that respect — synchronised allreduce barriers make both cards peak simultaneously and repeatedly, where every other workload in this study has one card working while the other idles. So the cap being in place rules out sustained over-draw; it does not exonerate power.

The topology is the more useful finding

The two cards are wired completely differently. GPU 1 sits behind the chipset, so all cross-card traffic crosses the DMI link — and with P2P unavailable, NCCL was staging through host memory. Every allreduce ran GPU0 → RAM → DMI → PCH → GPU1 and back, over x8 and x4 links.

That independently re-explains the tensor-parallel performance result. The −29% single-stream and −62% throughput penalty is not a generic “PCIe without NVLink” finding, as we first assumed — it is an x4-behind-the-chipset finding, specific to this board.

A fix that worked, on a checkpoint not worth deploying

A community checkpoint (cyankiwi/Qwen3.8-27B-AWQ-INT4, group-32 and asymmetric) had failed to load, and we spent a while believing the group size was a vLLM limitation. It was not: vLLM’s Marlin kernel advertises MARLIN_SUPPORTED_GROUP_SIZES = [-1, 32, 64, 128]. The failure was our own preparation scripts, three of which hard-coded GROUP = 128 while vLLM sized the expected scale tensor from what the checkpoint declared — size of tensor a (160) must match tensor b (40), which is 5120/32 against 5120/128.

The scripts now derive the group size from the checkpoint’s own config.json, and the fix is verified end to end: all three prep stages derived group 32, passed their round-trip gates, and the checkpoint served through the fast stack at 139.5 ±0.2 tok/s with 72.8% draft acceptance and coherent output. A second checkpoint assumption fell out on the way — the draft-vocabulary builder expected a model_extra_tensors.safetensors shard that only our own checkpoint ships, and now creates it when absent.

A benchmark that looped for ten hours, and a score that flattered the winner

Testing the Swift fine-tune against the base model with thinking on produced two traps worth writing down, one in the harness and one in the result.

The harness. Aider’s benchmark does not stream, and its request timeout is a module constant: 600 seconds. With thinking on, the base model writes 17,000–28,000 tokens for one reply on a hard exercise. At llama.cpp’s ~25 tok/s per slot that is twelve to nineteen minutes. Every such reply timed out client-side after the server had finished generating it, and Aider re-sent the identical prompt with exponential backoff: 0.2 s, 0.5 s … 4,096 s. Seventeen hours in, nine exercises were done, two of them had consumed ten hours each, and the server had generated three million tokens that nobody read. Nothing errored. The signature is in the usage log, not the harness output: the same prompt_tokens value sixty times in a row. The fix is one line (timeout: 7200 under extra_params), and the general rule is older than this project: when a slow engine meets a client default, the default wins silently.

The result. Swift scored 71.7% on LiveCodeBench against the base model’s 63.3%, and it would have been easy to report that as a better model. It is not. Of 44 answers across all arms that hit the 32,768-token cap, none passed; on the 40 problems where neither model was cut off the score is 37 to 37. The fine-tune changes how often the model finishes, not how well it solves. That is still worth having — an unfinished answer is worth nothing — but it is a different claim, and it predicts something the headline does not: raise the cap far enough and the accuracy gap should close while the token gap stays. Any benchmark of a thinking model under an output cap is partly a benchmark of the cap. Report truncations next to every pass rate.

Provenance

Conditions changed across five days as we learned what mattered. Two tables in the log can look directly comparable without being so; this is the key.