The measurement log

1. Weight Quantizations

All formats available on HuggingFace for Qwen3.8-27B, ranked by accuracy. VRAM is weights-only; add 2–8 GB for KV cache depending on context length and cache dtype.

2. GGUF Variants (llama.cpp / Ollama)

Quality measured by KL divergence and top-1 token agreement against BF16 reference. Lower KLD is better; higher agreement is better. Highlighted row fits a single 3090 with good KV cache headroom. Unsloth “UD” prefix = Dynamic v3.0 quantization (per-layer quant type selection + 1.5M-token imatrix calibration); independently validated to have ~2× lower KLD than standard Q4_K_M at similar size. UD-Q4_K_XL and above include MTP heads in the GGUF; below Q2_K_XL, MTP is stripped (separate 1.37 GB MTP Q4_0 file available). Unsloth’s only RTX 3090-usable format is GGUF—their NVFP4 is Blackwell-only, and they don’t produce AWQ/GPTQ/AutoRound checkpoints.

3. KV Cache Quantization

KV cache dtype affects context capacity and accuracy. On Ampere, FP8 KV is storage-only (dequants to BF16 for matmul) but still halves KV memory.

4. MTP Speculative Decoding

Qwen3.8-27B ships with Multi-Token Prediction heads. Speculative decoding is mathematically exact—rejected drafts are discarded, so output quality is identical to non-speculative. The question is purely speed vs VRAM overhead.

MTP Depth Sweep (llama.cpp, single RTX 3090)

Alternative Speculators

5. Serving Frameworks

Evaluated for Qwen3.8-27B on Ampere RTX 3090s specifically. “Day-one” means confirmed support at Qwen3.8 launch (Aug 14, 2026).

6. Dual-3090 Deployment Strategies

Four viable patterns for 2× RTX 3090 without NVLink. “C=1” is single-stream latency; “Aggregate” is total throughput under concurrent load.

7. Published Performance Numbers

Numbers from community benchmarks. Sources marked with framework and hardware. ^3.6 = measured on Qwen3.6-27B (same architecture, directly comparable).

Single RTX 3090

Dual RTX 3090

8. Accuracy Deep-Dive

Qwen3.8-27B baseline scores and measured degradation under quantization. Focus on coding benchmarks since the target use case is agentic coding.

Qwen3.8-27B Baseline (BF16)

AutoRound MixedInt4 Recovery (Pilcothink)

dbirks AutoRound W4A16 (Coding Benchmarks)

syv-ai Full Optimization Stack Quality

9. Recommended Benchmark Configurations

Ranked by expected performance × quality on our dual RTX 3090s. All configs should be benchmarked with the same prompt shape and concurrency levels for fair comparison.

10. Measured Results THIS MACHINE

Everything below was run on this host, August 22–24, 2026: 2× RTX 3090, driver 580.173, CUDA 13.0, 250 W power cap unless marked 350 W. Single-stream numbers are 512-token greedy chat completions on a ~46-token prose prompt, mean of 3. Sections 1–9 are the research; this is what the research predicted versus what happened.

Single-Stream Leaderboard (C=1)

Concurrency Scaling

DFlash2 Depth Sweep (llama.cpp, GPU 1, 250 W)

The same DFlash2 drafter is 2.5× faster inside vLLM (153.9) than inside llama.cpp (60.7). The multiplier is the framework’s verify path—split-KV verify attention and a CUDA-graphed spec step—not the drafter. Accepted length on this prose/greedy prompt (3.5–4.3) is well under the card’s 5.3, which is GSM8K with sampling; expect closer to that on code and math.

llama.cpp KV-Cache Dtype (llama-bench, 350 W)

Quantised KV is a free +24% on prompt processing and neutral on generation, confirming section 3. A second GPU buys +35% prefill and only +10% decode—PCIe layer split is not where the wins are.

Quality Gate (patched vLLM, GPU 0)

No non-speculative greedy reference was available from this stack (the launcher has no SPEC=none path), so exactness is established by self-consistency rather than against a plain decode.

Incidents & Open Questions

• Xid 79 “GPU has fallen off the bus” on GPU 0 during stock vLLM’s Triton GDN autotuner (rapid idle↔350 W transients). Not recoverable without a reboot. Diagnosed as a 12 V-rail brownout; both cards now run at nvidia-smi -pl 250 -pm 1 and nothing has faulted since, including 32 s at 249.8 W pinned under C=64. The cap costs ~14% on unassisted llama.cpp decode and is invisible on the vLLM winner.
• GPU 0 vs GPU 1 gap in MTP mode: 135 vs 150 tok/s, same image, model, prompt, and power cap, reproduced twice. Unexplained—candidates are PCIe lane width on slot 1 vs slot 2, or silicon lottery. DFlash2 on GPU 0 (153.9) was not re-run on GPU 1; it may go higher.
• DFlash2 in llama.cpp needs an unmerged PR (#27342, z-lab:dflash2); the stock --spec-type draft-dflash only loads v1 drafters. Built separately at a separate build directory.
• Not yet run: two batch replicas behind the LB; DFlash2 on GPU 1; a SGLang DP=2 retry at 250 W; DSpark (research #4); any TP=2 config; long-context (>4K) throughput.

11. Speed vs Context Length THIS MACHINE

Section 10’s leaderboard used a 46-token prompt. That flatters every speculative config, because a short open-ended prompt is the easiest possible draft target. These curves re-run the top five configs across the full context window on one fixed task — “write a detailed technical summary of the documents above, then list ten concrete recommendations” over a growing prefix of real, non-repeating technical prose (llama.cpp and vLLM documentation, ~1.9 tokens/word). 256-token greedy completions, mean of 2 after a warm run, prompt caching disabled on both frameworks so time-to-first-token is comparable. Prompt length is calibrated per request against each server’s own tokenizer.

Beyond 64K — what each profile costs

The 64K ceiling in the charts above is the deployed profile’s, not the model’s: Qwen3.8 is 262K native. Reaching further means changing the KV cache, and each option pays differently.

Two caveats on the KVarN row. The repo documents CTX=huge at 245,760 tokens, but the launcher resolved max-model-len to 200,000 on this host, so 192.5K is the deepest point measured rather than a hard ceiling. And 240K was never reached: the harness’s length calibrator refused to build a prompt past the resolved limit.

12. Coding Workloads, Measured Clean THIS MACHINE

Everything above is summarization. That matters because speculative decoding’s speed is a function of draft acceptance — how predictable the next tokens are — and prose summarization is a hard draft target. This section runs an 11-cell suite of coding tasks built from real files on this machine, against three configs, single-tenant with the other GPU idle. 256-token greedy completions, prompt caching off, 250 W cap, acceptance read from each server’s own counters.

Decode tok/s by task class

Acceptance percentages are only comparable within an engine: llama.cpp reads 88–100% but at n=1 its ceiling is 2.0 tokens per step, so the honest cross-engine metric is tokens/step — vLLM 2.8–5.9 against llama.cpp’s 1.8–2.0. That gap, not the acceptance rate, is why llama.cpp runs at a third of vLLM’s speed here.

Why k=4 beats k=7 — and when it doesn’t

Each speculated token is conditioned on the ones before it, so acceptance falls with draft depth. The rate of that fall is what decides the optimal draft count — and it is task-dependent, not a property of the hardware. On a repo edit, DFlash2 still lands 55% of its seventh draft token, so deep drafting pays. On prose the same speculator collapses to 12%, meaning positions 5–7 are verify compute spent on tokens that will be thrown away. This is the mechanism behind every k-sweep in this document, including the earlier finding that n=4 beat the model card’s recommended n=7 — that sweep was run on prose.

13. Multi-Turn Agentic Profile THIS MACHINE

Every measurement above is single-turn against a clean context. Real agentic coding is not: context accumulates across turns of tool calls, file reads and edits. To reach that regime we pointed the Pi coding agent (v0.84.3, MIT) at a single pinned replica and had Qwen3.8-27B build Tetris from scratch — core loop, then hold-piece, SRS wall kicks, scoring — each feature requiring it to re-read and extend its own earlier code. 68 assistant turns, context growing 5.4K → 61.5K against a 65,536 ceiling. Decode rate and TTFT come from stamping pi’s streaming JSON deltas on arrival; acceptance from the replica’s own spec-decode counters over the same windows.

Compaction bought nothing. Pi compacted at turn 52, cutting context 61.5K → 30.4K. The twelve turns before averaged 70.8% acceptance and 113.1 tok/s; after, 67.4% and 112.2. Freeing context did not restore speed — exactly what you would predict if acceptance was never context-limited to begin with, and a useful independent check on the headline result.

The build worked. 618 lines of Python: curses rendering, SRS wall kicks, hold-piece, a 7-bag queue. It parses and imports cleanly. The ladder reached 4 of 11 planned steps before a turn hit the 900 s timeout — the agent proved far more iterative than designed for, spending 30 assistant messages on step one alone. That is a limitation of the ladder, not of the model: context still reached 61.5K, past the 58K target.

14. Independent Cross-Check THIS MACHINE

Section 13’s flat-acceptance result contradicted our own prediction, which is exactly when a finding deserves a second instrument. We repeated the Tetris ladder with Qwen Code v0.22.1 — a different codebase, a different prompt construction, and the Qwen-native XML tool-call format rather than Pi’s. The /metrics acceptance join is harness-agnostic and carried over untouched.

Two corrections were needed before the comparison meant anything. The raw aggregate read 93.7% acceptance against Pi’s 70% — because templated tool-call messages, which stream as a single delta and are trivially predictable, were pooled with real generation. Separated: real generation 74.5%, tool calls 94.0%. Worse, the rows carrying context and the rows carrying streaming deltas were disjoint — the join silently never happened, so the first fit was computed entirely on tool-call messages. Merging by adjacency yields the 33 properly paired rows above. Neither fault raised an error; both produced plausible-looking numbers.

15. Correctness — Aider Polyglot THIS MACHINE

Every section before this measures speed. Speed is only half the question, and speculative decoding cannot answer the other half — it is mathematically exact, so a speculator can never change what the model would have said. Any quality difference must come from the quantization or the KV dtype. This is the first correctness measurement on this stack: 224 of 225 Exercism exercises across six languages, two attempts with test feedback after a failure, diff edit format, four threads across both replicas, ~2 h wall.

Rust is the clear weak spot at 26.7%, less than half of Python’s rate, and it is weak cold as well (3.3% first attempt). C++ shows the opposite profile — near-worst cold at 3.8% but second-best with feedback at 57.7% — and it is also where most malformed edit blocks concentrated.

Edit format was settled by measurement, not assumption

On a 10-exercise smoke test, diff won both pass rates despite whole being perfectly well-formed — and whole bypasses the malformed-edit check that the benchmark partly exists to apply. The pass_2 gap is a single case at n=10; the pass_1 gap is the stronger signal. diff also matches Aider’s own shipped setting for every Qwen coder model.

16. A Comparison That Could Not Be Run THIS MACHINE

§15 measured correctness for one quantization. The obvious follow-up — the same 225 exercises against a second 4-bit checkpoint, isolating quantization method at equal bit width — looked free: cyankiwi/Qwen3.8-27B-AWQ-INT4 was already cached, no download required. It could not be run, and why it could not is more useful than the number would have been.

It is not one variable, it is five

The checkpoint is not classic AWQ at all — it is compressed-tensors / pack-quantized. Against the deployed checkpoint:

Three of those five were closed by running the serving stack’s own prep on a copy (quant_lm_head.py rel. error 0.0064, quant_embed.py 0.0056, quant_mtp.py ~0.007), which would have left body quantization as the sole correctness-affecting variable. The clean experiment was within reach.

A useful by-product: the noise floor

The same card and configuration spans 165.4 → 159.6 tok/s across restarts. Single-stream differences below roughly 5% on this box are not meaningful. That threshold applies retroactively: the GPU 0 / GPU 1 gap reported earlier in this document reproduced here at ~4%, which sits at the edge of noise — treat it as suggestive rather than established.

17. Choosing a Serving Layer for Comparison THIS MACHINE

§16 established that the patched stack is checkpoint-specific and cannot serve a group-32 checkpoint. That is disqualifying for an instrument meant to evaluate arbitrary new models, so we retested the obvious alternative — stock vllm/vllm-openai, the same image that previously crashed this machine.

The distinction is categorical, not marginal: where stock vLLM fails on the group-32 checkpoint it fails with OOM, a resource problem, never a shape error. It accepts the format; the patched stack structurally cannot.

The first actual quantization comparison

Matched conditions — eager, no speculation, 4096 context. Indistinguishable: 0.6% apart, far inside the ~5% noise floor established in §16. AWQ costs +1.46 GiB because its lm_head and MTP head remain BF16 where AutoRound-fast’s are int4. This is speed only; a correctness comparison still needs the two-hour Aider run.

What going stock costs

Two non-obvious requirements

gpu_memory_utilization does not reserve for CUDA graph capture. At 0.93, vLLM consumed 22.99 GiB against a 22.1 GiB budget and then OOM’d needing 784 MiB for graphs. Pin the KV pool explicitly with --kv-cache-memory instead of trusting the utilization fraction. And max_num_seqs is bounded by Mamba cache blocks — hybrid GDN models need one block per decode sequence, so pinning KV to 1.5 GiB left 31 blocks against a default of 256 and raised a hard ValueError. Both belong in a preflight check, not in a troubleshooting log.

18. The Framework’s First Report Card THIS MACHINE

Everything in §§10–17 was measured by hand-driven scripts. Those results are only reusable if the same measurements can be re-run on a new model without re-deriving the method each time, so they were consolidated into a benchmark framework: six axes, a declarative model config, and a machine-checked fairness contract. This is its first execution against live hardware — and the point of the exercise is not the numbers, which we already knew, but whether the instrument reproduces them.

Framework output vs the hand-driven reference

The TTFT slope reproducing to three significant figures across two independently written harnesses is the strongest single signal that the instrument measures what it claims to. Prefill cost is the least noisy thing on this box, so it makes a good calibration target.

Full card — deployed stack, core tier

Throughput saturates at C≈32 (404 tok/s) and does not improve at C=64 — consistent with MAX_SEQS=8 per replica across two replicas, so beyond ~16 concurrent requests the excess queues rather than adding throughput.

Peak power reached 276.8 W against a 250 W cap. The cap governs sustained draw rather than instantaneous excursions, so this is expected — but it is worth recording given that uncapped transients were what took a GPU off the bus earlier in this project. No Xid events occurred during the run.

19. Reading These Numbers

This document accumulated over five days, and its measurement conditions changed as we learned what mattered. Two tables in it can look directly comparable and not be. This section is the provenance key — the result of auditing every campaign’s wall-clock window against every other, using server logs and sampler files to establish what was running when.

20. Canonical Numbers THIS MACHINE

The definitive single-GPU figures, re-measured strictly single-tenant, one configuration at a time, mean of five with spreads. 512-token greedy completions, thinking disabled, 250 W cap. Where earlier sections and this one disagree, this section wins.

Spreads are 0.1–2.0% — far tighter than the ~5% noise floor quoted elsewhere. That floor is a restart phenomenon, not run-to-run variance: repeated measurements against one live server agree closely, while a server restart shifts the whole set. Both facts matter, and they are not the same fact.

Throughput

One open discrepancy, stated rather than smoothed: DFlash2 at ~1K measures 212.1 here against §12’s 181.5, a 17% gap. Both runs were single-tenant and this one has a ±0.6% spread across five repeats. The cause is unidentified. The patched-versus-stock advantage is 1.96× at both prompt lengths, revising the 2.3× quoted in §17.

21. The 64K Ceiling Was a Default — 30 August

Sections 11 and 17 treat ~64K as the practical bf16 limit on a single card. It is not a limit. start_qwen.sh line 74 reads MAX_LEN=${MAX_LEN:-65536}, an overridable default that was mistaken for a hardware constraint — while the running server was reporting GPU KV cache size: 89,437 tokens at those same settings.

vLLM names the constraint exactly when exceeded: “7.56 GiB KV cache is needed, which is larger than the available KV cache memory (6.31 GiB)”. KV scales at roughly 13,700 tokens per GiB here — generous because only 16 of 64 layers hold attention KV. The real ceiling is ~90,000, and we had been serving 37% less than the card held. The cost of more context is concurrency: max_num_seqs falls 8 → 4 → 2.

22. Tensor and Pipeline Parallelism — 29 August

Both cards had only ever been used as independent replicas. Six configurations were planned; four ran, one was blocked by a model-implementation gap, one was abandoned as redundant.

The 4.2× capacity exceeds the model’s own 262,144 native window, so bf16 under TP reaches the architectural ceiling and the fp8/KVarN profiles become pointless there. PP=2 is unreachable: NotImplementedError: Pipeline parallelism is not supported for this model — Qwen3_5ForConditionalGeneration does not implement vLLM’s SupportsPP. The fp8-at-native configuration was abandoned after ten minutes of FlashInfer JIT compilation, being redundant once bf16 reached 262,144.

23. Two GPU Faults, One Broken Explanation — 22 and 29 August

Aug 22 16:24:03  NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
Aug 29 21:49:55  NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
                 NVRM: Xid 154, GPU recovery action ... 0x1 (GPU Reset Required)

Same card, identical signature, seven days apart. No AER errors in either boot — no correctable or uncorrectable PCIe errors, nothing preceding the fault.

Verdict: do not retry TP=2 on this machine. The topology makes it a poor fit on the merits; it is the only workload twice associated with a dead GPU; and its payoff is weakly motivated now that ~90K is reachable on one card. NCCL_P2P_DISABLE buys nothing, since P2P was already unavailable. The real remedy is hardware — moving GPU 1 to a CPU-attached slot.

24. KV Quantization Is Not Free — 30 August

Every quality figure in this document before this section was measured on bf16 KV. The weights were always 4-bit; the cache dtype was the untested variable. Fixed deterministic 72-exercise subset; arm-to-arm comparable only, not against the 225-case 12.5 / 48.7 / 90.2 reference.

fp8 is 22% slower than bf16, not faster. Its only benefit is context. The −5.6-point pass_2 gap sits inside the resolution limit of a 72-case run and is not a proven regression, but the direction is unhelpful and the speed cost is not in doubt.

Each arm is internally deterministic, yet both quantized caches diverge from the bf16 reference within the first ~100–200 characters at every length. KV quantization measurably changes what the model says, early rather than only at depth.

Two harness traps found en route. The divergence harness initially called the load balancer, so two “identical” calls hit different replicas and bf16 falsely appeared non-deterministic. And a probe run concurrently with a benchmark had batch-shape variation flip near-ties, producing another false non-determinism result. Both produced plausible-looking wrong answers; both were fixed by pinning to one replica and running serially.