Every number we measured

Swift against the base model, thinking on 13 SEP

ukisai/Swift-Qwen3.8-27B is a fine-tune of Qwen3.8-27B trained to penalise the reasoning-marker tokens its author associates with overthinking. The claim, made on bf16 weights over five seeds: LiveCodeBench v6 76.8% → 81.6% with 46% fewer median thinking tokens, and under one point lost across ten benchmarks. This section tests that claim on this hardware, under the author’s own conditions.

LiveCodeBench v6 — 60 fixed problems from the v6 increment (Jan–Apr 2025)

Paired bootstrap on the pass@1 difference: bartowski arm [+0.0, +16.7] points, author’s quant [+3.3, +23.3]. Discordant pairs: 1 problem only the base solved, 6 (9) only Swift solved — every one of them a problem the base ran out of tokens on.

Aider polyglot — the fixed 36-exercise subset, two attempts, diff format

Reasoning and answer tokens are estimated by splitting each request’s completion_tokens by the character share of reasoning_content, recorded by a logging proxy in front of the server. The answers are the same length in both arms; all of the saving is thinking.

The ladder

Same model, same card, same prompt. The only variable is the serving software. Every rung is the same 4-bit checkpoint — nothing was quantized further between the bottom and the top.

Canonical single-GPU figures

Re-measured strictly single-tenant, one configuration at a time, mean of five runs with standard deviations. 512-token greedy completions, reasoning disabled, 250 W power cap. Where an earlier section of this study disagrees, these win.

Spreads are 0.1–2.0%, far tighter than the ~5% noise floor quoted elsewhere in this study. Both facts are true and they are not the same fact: repeated measurements against one running server agree closely, but restarting the server shifts the whole set — the same card and configuration produced 165.4 then 159.6 tok/s across a restart. Treat the restart spread as the floor for comparing two configurations.

A fourth engine was added later and is not in this table because it was measured on the other card under a different comparator: ExLlamaV3 1.4.5 at EXL3 3.00 bpw with MTP reaches 94.8 ±0.5 tok/s, against 43.6 with speculation off. That is 63% of the patched stack’s speed under the matching idle-neighbour condition, and it buys 128K–200K of context on one card. Full figures, and what it costs in fidelity, are in Long-context fidelity.

Prompt length changes speed only through acceptance

A control settles what looked like a measurement error. With speculation off, stock vLLM measures 32.6 tok/s at 55 tokens and 32.8 at 1K — a 0.6% difference, i.e. none. Turn speculation on and the same engine gains 29%; the patched stack with DFlash2 gains 46%, its acceptance climbing 0.420 → 0.661. A longer prefix of coherent context is simply easier to draft against.

The consequence: any speed comparison across different prompt lengths is meaningless unless acceptance is reported alongside it. This study made exactly that mistake once, “discovering” a 20% contention error that did not exist.

Throughput under concurrency

Two replicas win up to about 16 concurrent requests while keeping every individual request fast; past that, the batch profile’s INT8 tensor-core GEMMs take over. An earlier run measured the single-GPU C=64 figure at 1,023 tok/s — that measurement overlapped other GPU work, but the card was already saturated at 100% utilisation and 249 W, so saturated-throughput figures are largely contention-insensitive and the two agree within 6%.

The 64K ceiling was wrong

vLLM states the constraint precisely when you exceed it — “7.56 GiB KV cache is needed, which is larger than the available KV cache memory (6.31 GiB)” — so the right behaviour is to surface that message, not to pre-emptively refuse. KV scales at roughly 13,700 tokens per GiB on this model, which is unusually generous because only 16 of its 64 layers hold attention KV; the other 48 use fixed-size GDN recurrent state.

The real single-card bf16 ceiling is ~90,000. Reading the middle rows of that table, we concluded more context had to be bought by cutting concurrency — max_num_seqs falling 8 → 4 → 2.

85,000 is now applied and live on both replicas, verified by serving rather than merely loading: a 78,652-token prompt returned a correct completion in 86 s, and /v1/models reports 85,000 on the load balancer and both replicas. The speed cost is nil — 148.5 ±0.1 tok/s at the new setting against a 149.6 ±0.2 control at the old one, a 0.7% difference well inside noise. The panel’s derived client guidance moved with it: max output at an 8K prompt reservation rose from 57,344 to 76,808.

Context scaling to 64K

Time to first token is the honest cost of long context, and it is brutally linear: +1.14 seconds per 1,000 tokens, correlation 1.00. At 61K a request waits 69 seconds before its first token — roughly six times the entire decode time of a 500-token reply. Speculative decoding accelerates generation, not prefill, so the speculator is nearly irrelevant here and prefix caching is the only lever that matters.

Beyond 64K

The 64K ceiling above is the deployed profile’s, not the model’s — Qwen3.8 is 262K native. Reaching further means changing the KV cache, and each option pays differently.

Two caveats on the KVarN row. Its documentation describes a 245,760-token ceiling, but the launcher resolved max-model-len to 200,000 on this host, so 192.5K is the deepest point measured rather than a hard limit. llama.cpp fails to allocate past roughly 160K on a single card — -c 200000 OOMs at load.

Correctness

Speed is half the question. Speculative decoding cannot answer the other half — it is mathematically exact, so a speculator can never change what the model would have said. Any quality difference comes from the quantization or the KV dtype.

The two runs above are the same suite measured twice, and they disagree by 2.2 points on pass_1 and by 13 points on Rust (26.7% versus 40.0%, n=30). Both used greedy decoding at temperature zero, which is nominally deterministic — but at four concurrent threads, batch composition varies and reduction order flips near-tie tokens. Aggregate scores resolve to roughly ±2 points; per-language scores only to about ±10. Read the per-language bars as indicative.

Three of the 225 cases (1.3%) failed for infrastructure reasons rather than model capability: one context exhaustion, one test timeout, and one HTTP 413 caused by an unset client_max_body_size on the reverse proxy. The remaining 24 malformed responses across 21 cases are genuine diff-format failures, concentrated in C++.

Agentic behaviour

Everything above is single-turn against a clean context. To reach the multi-turn regime, a coding agent built Tetris from scratch across 68 assistant turns, context growing 5.4K → 61.5K.

A natural experiment confirms it: the agent compacted its context at turn 52, cutting 61.5K → 30.4K. The twelve turns before averaged 70.8% acceptance and 113.1 tok/s; after, 67.4% and 112.2. Freeing context bought nothing — exactly what you would predict if acceptance was never context-limited.

Why acceptance varies so much

Draft acceptance is the causal variable behind almost all of the tok/s spread on this page. It is determined by the task, far more than by the configuration.

Repo edits accept highest — emitting a diff means reproducing code already in the context, and acceptance there rises with more surrounding code. Prose summarization is the floor. Computing π was predicted to be the memorized ceiling and turned out to be near the bottom of the coding classes, because dense numeric code is not stereotyped boilerplate.

Each speculated token is conditioned on the ones before it, so acceptance falls with draft depth — and the rate of that fall is task-dependent, not a property of the hardware. On a repo edit, DFlash2 still lands 55% of its seventh draft token, so deep drafting pays. On prose the same speculator collapses to 12%, meaning positions 5–7 are verify compute spent on tokens that will be discarded. This is the mechanism behind every optimal-draft-depth result in this study — and behind the earlier finding that n=4 beat a model card’s recommended n=7, which was measured on prose.

Which numbers to trust

Which speculator to serve: MTP vs DFlash2, both cards

Single-GPU DFlash2 is the fastest configuration we measured (212.1 tok/s). The obvious next question — run it on both replicas and get that speed plus concurrency — went unanswered for most of this study, and shipped in the control panel as an untested profile. This settles it.

Aggregate tok/s through the load balancer, ~1K-token prompts, 512-token completions. C=1 is the mean of five with standard deviation; the rest are single runs.

The mechanism is visible in the replicas' own counters. Under this load DFlash2 ran at 46.9% draft acceptance against MTP's ~76%. It proposes seven tokens per step and discards most of them, and that wasted verification work is precisely what a saturated GPU cannot spare. The same property that makes it excellent for one user — drafting deep and often being right — becomes a liability once the verify batches are already full.

It also answers the other half of the question: pairing DFlash2 does not preserve its single-GPU advantage. At C=1 both configurations land near 125 tok/s through the load balancer, nowhere near the 212 a single dedicated GPU delivers. One request goes to one replica; the second card contributes nothing but its presence.

Tensor parallel: the full native window, at a price

Two cards had only ever been used as two independent replicas. Splitting one model across both — tensor parallelism — was untested until now, and it changes the context picture entirely.

That 4.2× exceeds the model’s own 262,144 native window, so bf16 under TP reaches the architectural ceiling — and the fp8 and KVarN profiles become pointless there, since they exist only to stretch a single card. A 204,058-token prompt was verified served, not merely loaded, completing in 311 s at roughly 656 tok/s of prefill.

KV-cache quantization is not free

Every quality number in this study before now was measured on bf16 KV. The weights have always been 4-bit; the cache dtype was the untested variable, and it is the one that governs how much context fits on a single card. Measured on a fixed, deterministic 72-exercise subset.

These scores are comparable to each other only. They are a 72-exercise subset and must not be read against the 225-case reference of 12.5 / 48.7 / 90.2.

Both quantized caches change what the model says

Each arm is internally deterministic. Compared against the bf16 reference, however, both diverge within the first ~100–200 characters at every length tested — early, not merely at depth.

Long-context fidelity: the measurement we said could not be made

The scope limit above stood for three days: the fp8 and KVarN profiles exist for 150K and 200K, and every fidelity number we had was taken at ≤64K, because the bf16 reference cannot go further. The way out was not a bigger reference. It was a different instrument.

The replacement is teacher-forced excess negative log-likelihood. Feed a long document as the prompt, ask the server for the log-probability of every prompt token (prompt_logprobs), and report mean NLL per 8K-token bucket of position. The ground truth is the document itself, so no reference model is needed and the metric is defined at 200K exactly as at 8K. Teacher forcing pins the context, so nothing compounds: a rounding error at position p affects the prediction at p+1 and nothing else. And because prefill is chunked, almost everything a token at position 150K attends to is read back through the quantized cache — which is the thing under test.

Memorised text would defeat it, so the document set is five frozen documents of ≥200K tokens each: three the model wrote itself (202,296 / 214,368 / 212,578 tokens — a novel, an engineering design document, an essay sequence), and two built from 22 arXiv papers apiece submitted between 20 July and 1 September 2026, after the model’s assumed training cut (207,877 and 206,061 tokens). Repeat runs of the harness are bit-identical, so the determinism floor is exactly zero and any excess NLL is signal.

Excess NLL against bf16, 0–76K

Mean and worst 8K bucket, in nats per token, against the patched stack at bf16 KV. The threshold, written down before the run, was 0.02 nats.

gen1–gen3 are the model’s own text (mean NLL 0.2–0.5); nat1–nat2 are recent arXiv prose (mean NLL ~4.7). Negative numbers are not evidence that a quantized cache is better than bf16 — they are the two attention paths (FlashAttention bf16 against FlashInfer fp8) differing by kernel numerics.

Past the reference: cross-profile agreement at 80–150K

Above 76K there is no bf16 to compare against, so the profiles are compared to each other, with fp8 as the anchor because it matched bf16 everywhere it could be checked. Figures are “other minus fp8”, mean / worst bucket, then minimum top-1 agreement.

fp8 shows no inflection with depth on any of the five documents. Its 80–150K bucket means track the difficulty of the text, and on the generated documents they are flat. KVarN drifts. Its gap to fp8 on nat1 roughly triples between the sub-76K band (+0.025 mean) and the 80–150K band (+0.072 mean, +0.129 worst); on generated text it stays inside 0.01. So it degrades with depth, mildly, and only where the text is hard. Its 150–200K band could be cross-checked against nothing but EXL3 at q8, which is itself worse. EXL3’s gap at depth is the same size as its gap to bf16 in the overlap band — a weight-quantization cost, flat with depth, not a cache effect.

Known-answer probes at depth

NLL says whether the distribution moved. It does not say whether a user gets a wrong answer. Three probes with known answers — eight-needle key/value retrieval, verbatim copy of a marked paragraph, and a three-step cross-reference chain — run at depth against filler drawn from the frozen documents.

Every probe instance succeeded at every depth for every profile — KVarN at 196K and EXL3 3.0 bpw at 123K included. So the fidelity cost measured above does not surface as a retrieval, copying or multi-hop failure at these depths. It is a distribution shift, not a capability loss — which is exactly why it needed a distributional instrument to see at all.

ExLlamaV3: the context is real, the speed is not

The other route to long context on one card is a smaller checkpoint on a different engine. EXL3 at 3.0 bpw fits far more cache than the 4-bit compressed-tensors weights do, and — contrary to what we assumed when we closed the sub-4-bit line — ExLlamaV3 does support MTP speculative decoding on this model. Measured on GPU 1 with the GPU 0 replica idle, the same condition as the 149.6 comparator.

MTP is worth 2.2× here, and acceptance matches vLLM’s 70–76%, so EXL3 avoids the wall llama.cpp hit. What it does not do is close the gap: 94.8 against the patched stack’s 149.6 under the same conditions is 63% of the speed. What it buys is context on one card — 128K at fp16 KV, 200K at q8 — and decode is flat across cache sizes. The fp16 ceiling is sharp: with no draft model resident, 131,072 tokens loads with 0.3 GiB of headroom and 135,168 already fails autosplit.

The decision: fp8 KV at 150K

Unsloth: resolved, and the answer is no

Verified from the repository manifest in the local cache rather than from documentation: Unsloth ships GGUF only for Qwen3.8-27B — 30 variants, no AWQ, GPTQ or AutoRound exports, and their NVFP4 build is Blackwell-only and unusable on Ampere.

The decisive constraint is vLLM’s own documentation, which describes its GGUF support as “highly experimental and under-optimized”, notes it has been moved out-of-tree to a plugin, and states that “the only confirmed use case is memory footprint reduction”. So Unsloth’s genuine quality-per-byte advantage stays locked inside llama.cpp, at 27.6–42.6 tok/s against the patched stack’s 167.8 — roughly 4× the speed to buy it.

Sub-4-bit on the fast stack is closed

Weights and KV cache compete for the same 24 GB, and KV scales at ~13,700 tokens per GiB, so every GiB shaved off the weights buys roughly 13,700 tokens of context at full bf16 fidelity and full speed. Our weights are ~17.5 GiB. That arithmetic makes a smaller checkpoint the most attractive remaining route to more context — and it is why we looked at quantizing the model ourselves to ~3.5 bits in compressed-tensors format, the only format the fast stack accepts.

That leaves EXL3 (3.0 bpw ≈ 12.89 GiB, which would imply roughly 150K of context on one card) as the only remaining route to a smaller checkpoint — and it requires leaving the fast stack for ExLlamaV3. Whether that stack supports MTP speculative decoding on this GDN hybrid is unknown, and worth settling before committing to a 15 GB download: without speculation it would hit precisely the wall llama.cpp did.