Field notes · 2× RTX 3090 · August–September 2026

The AI Frontier: No Datacenter Required

Running Qwen3.8-27B optimally on an RTX 3090

212 tokens a second on a graphics card you can buy secondhand, and a fine-tune that halves how long it thinks without costing it an answer. Here is what that gets you, what it cost to get there, and whether it is worth your weekend.

Update · 18 September 2026

The best way to serve it, and the best way to use it

Three weeks of measurement come down to two decisions. Serve it through patched vLLM: that is the 212 tokens a second below, 7.7× the out-of-the-box install. And when you turn thinking on, run the Swift fine-tune instead of the base weights: it reaches the same answers on about half the reasoning, which on real coding work is the difference between a nine-hour job and a five-hour one.

212tok/s

base model, patched vLLM, thinking off

−50%

median output tokens with Swift, LiveCodeBench

−57%

median reasoning per request with Swift, Aider

37 = 37

problems solved where neither model ran out of room

9.6→5.5h

same 36 coding tasks, time spent

The second decision is new. Qwen3.8-27B with thinking on is a better programmer than with it off, and it is also exhausting: given a coding exercise, the base model wrote a median of 17,900 tokens to get through it, on the hard ones 20,000 to 28,000 for a single reply, and one request in thirteen ran into the 32,768-token ceiling and returned nothing at all. Swift-Qwen3.8-27B is a fine-tune by UkisAI that penalises the tokens associated with that spiralling. Its author claims 46–58% fewer thinking tokens for under a point of accuracy. We gave both models the same problems under the author’s own conditions, with the quantization held identical, to see whether that survives contact with somebody else’s hardware. It does.

How much each model writes before it answers · lower is better

Same problems, thinking on	Base Qwen3.8-27B	Swift
LiveCodeBench v660 problems, one attempt each	63.3%	71.7%
output tokens, median	14,190	7,035−50%
answers cut off at the 32K ceiling	19 of 60	14 of 60
Aider polyglot36 fixed exercises, two attempts	69.4%	66.7%
reasoning tokens per request, median	4,678	1,994−57%
time spent on the whole setsummed per exercise, two at a time	9.6 h	5.5 h

Read the accuracy row carefully, because it flatters Swift for the wrong reason. Not one answer that hit the 32K ceiling passed, from either model. On the 40 LiveCodeBench problems where neither ran out of room, the score is identical: 37 and 37. Swift is not a better programmer. It is the same programmer who finishes the thought, and on hard problems finishing is most of the battle. The Aider result, where the ceiling bites less, shows the honest picture: parity, inside the noise of a 36-problem sample.

What to run. Thinking off, or latency above all: the base model on patched vLLM, 212 tok/s. Thinking on: Swift, which today means its GGUF through llama.cpp at about 55 tok/s — a third of the decode speed, and still 43% less time on agentic work because it writes half as much. On this box both now sit behind one endpoint and the request’s model field picks the card. Nobody has published a 4-bit vLLM checkpoint of Swift; when one exists the two wins stack, and that combination is unmeasured. Sample sizes are 60 and 36 with one seed, so treat the pass rates as ±12 points and the token reductions as solid. Full method and numbers →

A little under two years ago, running a 27-billion-parameter language model meant renting time on somebody else's datacenter and hoping the bill stayed reasonable. Today it means a used RTX 3090 — about the price of a decent secondhand motorbike — sitting under your desk, answering faster than you can read, with nothing you type ever leaving the building.

That last part is the bit worth pausing on. Every prompt you send to a hosted model is a prompt you no longer control. Your half-finished code, the client's schema, the thing you were embarrassed to ask a colleague — all of it lands on somebody else's disk under somebody else's retention policy. Running the model yourself makes that question go away entirely.

So: what do you actually get?

The short version

What you get	On one RTX 3090	On two
Speed, one person typing	212 tok/smeasured with one replica running	212 tok/sbut ~11% less once both are resident
Speed, whole team hammering it	960 tok/sat 64 concurrent	905 tok/sat 16 concurrent, low latency
How much it can read at once	85K tokensnow the default; 150K with fp8 KV	262Ksplit across both, at ⅔ speed
Real coding tasks solved	48.7%Aider polyglot, 225 exercises	same
Power drawn	250 W	500 W

212 tokens a second is roughly ten times faster than you can read. In practice it means the model finishes a function before you have finished reading its first line — the latency stops being something you notice. 85,000 tokens of context is about 6,500 lines of code, or several long documents, held in mind at once.

And 48.7%? That one needs unpacking, because it is the number people get wrong.

What it can genuinely do — and what it can't

We ran the Aider polyglot benchmark against it: 225 real programming exercises across Python, JavaScript, Java, Go, C++ and Rust. Each one gets two attempts, with the failing test output shown after the first.

It solved 12.5% on the first attempt. Show it the failing tests, and that jumps to 48.7%.

That gap is the whole story of how to use this thing. Twenty-eight problems solved cold; eighty-one more solved once it could see what broke. The model is mediocre at guessing right first time and genuinely good at fixing what it can see is wrong. Point it at a codebase with a test suite and it will iterate its way to working code. Ask it to write something correct in one shot, with no feedback loop, and you will be disappointed roughly nine times in ten.

So the honest framing is this: a very fast, very private pair-programmer that needs a feedback loop to be useful. Not a replacement for a frontier model on hard novel problems. Excellent at the enormous middle ground of software work — refactors, boilerplate, test-writing, "why is this failing", translating between languages, explaining unfamiliar code.

It is also uneven by language. Python and C++ land around 58%, Rust around 27–40%. If you write Rust all day, temper expectations.

Field check, 29 August: pointed the Pi coding agent at this box and it one-shotted a fully fledged, playable Tetris — 7-bag randomizer, ghost piece, sound and all — in about four minutes; the caveat is that its first attempt died silently by spending the entire output budget on reasoning before writing a single byte, so mind your max_tokens. The full account is on the Running on Pi tab.

Concretely, what can you point it at?

A coding assistant in your editor. At 212 tok/s a single developer never waits. This is the use case the hardware is best suited to.
A small team. Two cards behind a load balancer served 610–905 tok/s spread across 16 simultaneous requests — comfortably a handful of developers with a coding assistant each, all of it staying on your own network.
Bulk work overnight. One card in batch mode hit 960 tok/s at 64 concurrent requests. Documentation passes, test generation, migrating a few hundred files — the sort of job you start before bed.
Anything you cannot send to a hosted API. Client code under NDA, medical or legal text, an air-gapped network. This is where local models stop being a hobby and start being the only option.

One caveat that will bite you at long context. The model reads a 61,000-token prompt in about 69 seconds before it produces a single token. Generation is fast; reading is not, and it scales linearly with how much you give it. For chat and agent work, turn on prefix caching — it makes the second turn over the same document almost instant instead of another minute of waiting.

A quantized KV cache buys context, never speed. fp8 measured 22% slower than plain bf16, not faster — quantizing the cache saves memory and adds dequantization work to every attention operation. Reach for it when the context will not otherwise fit, and for nothing else.

If you do need the context, fp8 is the one to use — and we had to build a new instrument to say so. We originally reported that both quantized caches “changed what the model says”, on the evidence of how quickly their output diverged from bf16. That measurement compounds one rounding difference into two different paragraphs and tells you almost nothing. Measured properly — per-token likelihoods over 200,000-token documents the model has never seen — fp8 is indistinguishable from bf16 all the way to 150K, and scores 100% on retrieval, verbatim-copy and multi-hop probes at 147K. The 4-bit cache that reaches 200K does cost something real, and a smaller 3-bit checkpoint on a different engine costs more. The Results tab has the numbers.

How we got from 27 to 212

Here is the part that surprised us. When we first installed this model the honest, out-of-the-box, follow-the-README answer was 27.6 tokens a second. Perfectly usable. Also about an eighth of what the same card can do.

Same model, same card, same prompt — only the software changes

Nothing on that chart involves a different model, a smaller quantization, or an overclock. Every bar is the same 4-bit checkpoint answering the same question. The entire 7.7× is serving software.

The single biggest lever: speculative decoding

If you take one technical idea away, make it this one. A small, fast "drafter" guesses the next few tokens; the big model checks them all in a single pass. Guesses it accepts are effectively free.

The clever part is that this is mathematically exact — rejected guesses are thrown away, so the model produces precisely the text it would have produced anyway. There is no quality trade to weigh up. It is simply faster. On our hardware it took the same engine from 32.8 to 85.6 tokens a second.

Cool, right? It gets better.

Then: whose code runs the verification

We assumed the drafter was the interesting choice. It wasn't. What mattered far more was which implementation ran the verify step.

A community fork of the exact same vLLM version — patched with a smarter attention kernel for the verification pass, a 4-bit output head, and a trimmed draft vocabulary — reached 167.8 tok/s where the stock release reached 85.6. Nearly double, from patches that never touch the model weights. Swapping in an external block drafter took it to 212.1.

We ran the identical drafter under two engines and got 60.7 tok/s in one and 212.1 in the other. If you are chasing speed, the engine deserves more of your attention than the drafter does.

One caveat we only nailed down at the end: the drafter that wins for one person loses for a team. DFlash2 is the fastest thing here at C=1, but run it on both cards and serve eight or sixteen people and it falls 12–34% behind plain MTP — it drafts seven tokens a step, accepts under half of them, and that wasted checking is exactly what a busy GPU cannot afford. Fast for you, slower for everyone. The Results tab has the full comparison.

And one thing to do before any of it

Cap your GPUs to 250 W first. Our first serious benchmark run ended with a card falling off the PCIe bus — Xid 79, unrecoverable, reboot required. Capping to 250 W and re-running the exact trigger eight times produced no recurrence, and it costs about 14% on the slow path and nothing measurable on the fast one. Do it.

But we called that solved too early. The same card died again a week later — with both GPUs verified at 250 W. Xid 79 with no PCIe error logged is consistent with either a power transient or a link fault, and our logs cannot tell them apart. A cap bounds average draw, not microsecond spikes. The second failure happened under tensor parallelism, the one workload that makes both cards peak simultaneously rather than taking turns. The Lessons tab has the full forensics, including the discovery that our two cards are wired to completely different buses.

sudo nvidia-smi -pl 250    # per-GPU power limit, watts
sudo nvidia-smi -pm 1      # persistence mode

Both reset on reboot, so re-apply them before you start serving. The Setup tab walks through the whole thing, one rung at a time, with the commands that actually worked.

Is it worth your weekend?

Depends what you want from it.

If you want a fast private coding assistant and you already own a 3090, then yes — and you can stop at rung two. Tuning llama.cpp with three extra flags gets you 42.6 tok/s in about twenty minutes with no Docker and no forks. That is a genuinely pleasant local assistant.

If you want the 212, you are signing up for a third-party fork of vLLM with real caveats: it hard-codes assumptions about how the model was quantized and refuses checkpoints that don't match. Wonderful when it fits your model, an afternoon of debugging when it doesn't.

And if you want to serve a team, two cards and a load balancer is the shape that works — not splitting one model across both, which cost us 29% of single-stream speed and 62% of throughput on this machine. Splitting is worth it for exactly one thing: it reaches the model’s full 262,000-token window, which nothing on a single card can.

What we would not tell you is that this replaces a frontier model. It doesn't. What it does is sit close enough for most day-to-day software work, at zero marginal cost, on hardware you own, with none of your code leaving the room.

Where the rest of it lives

Five days of measurement went into that chart, and a fair amount of it was us being wrong in interesting ways — a benchmark that scored a model whose output it could not read, a day of results invalidated by two GPUs quietly interfering with each other, a correction we published that turned out to be a worse error than the thing it corrected.

Results — every number and graph, with provenance attached to each
Lessons — the six ways this machine reported wrong numbers, and how to not repeat them
Setup — the commands, rung by rung
Running on Pi — a coding agent on the other end of the wire, and what it built
Log — the unedited record, corrections and dead ends included

Honest limits. One model, one machine, one architecture family, mostly at 4-bit. The fastest configuration is a third-party fork whose optimizations are specific to how this checkpoint was quantized. None of these figures are comparable to published leaderboard numbers — different quantization, harness versions, reasoning settings and serving stacks all move them independently. They are honest measurements of one deployment, not claims about what the model can do in general.

Predecessor write-up: Qwen3.6 on RTX 3090. Corrections welcome — several findings here began as confident mistakes.