Getting this running

You need one RTX 3090 (24 GB) for everything up to rung 4. Rung 5 uses two. Docker with the NVIDIA container toolkit is assumed from rung 3 onward. Weights are 4-bit throughout, so the model itself is ~17–19 GB and the rest of the card is KV cache.

Before anything else: cap the power

sudo nvidia-smi -pl 250     # per-GPU power limit, watts
sudo nvidia-smi -pm 1       # persistence mode

Both reset on reboot. Re-apply them every time, before you start a server. The cap cuts peak 12 V current by roughly 29%. It costs about 14% on unassisted llama.cpp decode and nothing measurable on the vLLM path — a trade worth making, because the failure it prevents is a hard lockup mid-run.

Rapid idle → full → idle cycling is what provokes it, which is exactly what kernel autotuners and benchmark loops do. If you only ever run steady inference you may never see it; that is not a reason to skip the cap.

Rung 1 — llama.cpp, as it comes

27.6 tok/s   16.9 GB VRAM   ~20 minutes, no Docker

Build llama.cpp with CUDA, pull a 4-bit GGUF, point it at the card. This is where most people start and it is a perfectly reasonable place to stop if you only need occasional single-user generation.

# build (CUDA, Ampere = compute capability 8.6)
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --config Release -j

# fetch a 4-bit GGUF (~17 GB)
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q4_K_XL.gguf

# serve — every layer on the GPU, nothing else tuned
./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -ngl 99 --host 127.0.0.1 --port 8080

Rung 2 — the same llama.cpp, with flags

42.6 tok/s   +54%   16.9 GB VRAM   no new software

Flash attention, a quantized KV cache, and speculative decoding using the model's own multi-token-prediction head. Same binary, same weights — this is the cheapest large win available.

./build/bin/llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -ngl 99 \
  -fa on \                        # flash attention
  -ctk q8_0 -ctv q8_0 \           # 8-bit KV cache: ~+24% prefill, no quality cost
  --spec-type draft-mtp \         # use the model's built-in MTP head
  --spec-draft-n-max 1 \
  -c 65536 \
  --host 127.0.0.1 --port 8080

The KV quantization is close to free: it bought about 24% on prompt processing here with no measurable effect on generation. It also halves the KV memory, which is what makes a 64K context comfortable on a 24 GB card.

Draft depth is workload-dependent. With the separate DFlash2 drafter, depth 4 beat the model card's recommended 7 on this hardware — the larger verify batch cost more than the extra accepted tokens returned. Sweep it on your own traffic rather than trusting a default.

Rung 3 — stock vLLM

85.6 tok/s single   ~960 tok/s at 64 concurrent   20.3 GB VRAM

Move here when you need concurrency. llama.cpp has no continuous batching; vLLM does, and that is the entire reason to take on the extra complexity. Single-stream speed roughly doubles too, from speculative decoding done well.

docker run -d --name qwen38 --gpus '"device=0"' --ipc=host \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model dbirks/Qwen3.8-27B-W4A16-AutoRound \
  --gpu-memory-utilization 0.95 \
  --kv-cache-memory=1500000000 \
  --max-model-len 4096 \
  --max-num-seqs 8 \
  --enforce-eager \
  --trust-remote-code

--enforce-eager skips CUDA graph capture entirely. It is the safe starting point; dropping it is worth about +20% once you have the memory budget settled.

Rung 4 — patched vLLM

167.8 tok/s (MTP)   212.1 tok/s (DFlash2)   22.4–22.7 GB VRAM   single replica

Those two speeds were measured with the second replica stopped. Running two replicas behind a load balancer — rung 5 — costs about 11% single-stream just for the neighbour being resident: the same MTP config measures ~149.6 there. Budget for it if you are serving a team.

The largest remaining gain comes from syv-ai/qwen38-27b-rtx3090, a third-party fork of vLLM 0.27.1 carrying fourteen patches. They target the verify step, which is where speculative decoding actually spends its time: split-KV verify attention, an int4-requantized lm_head and drafter, and a 40k-token draft vocabulary covering what the model actually emits.

git clone https://github.com/syv-ai/qwen38-27b-rtx3090
cd qwen38-27b-rtx3090
echo "VLLM_API_KEY=$(openssl rand -hex 24)" > .env

docker compose --profile single up -d     # one or a few users
# or:  docker compose --profile batch up -d   # many concurrent requests

The build downloads the checkpoint (~19.5 GB) and runs its own requantization steps. Expect 10–20 minutes for the image, ~5 for prep, and up to 15 minutes on a first-ever cold start while torch.compile, CUDA graph capture and FlashInfer JIT populate the cache. Warm restarts are about 60 seconds — preserve that cache volume, it is the difference.

Choosing a speculator and a context profile

MAX_LEN=85000 GPU_UTIL=0.96 ./up.sh --wait   # MAX_SEQS stays at 8

Rung 5 — two cards

905.6 tok/s at 16 concurrent   151 tok/s single-stream

Run two independent replicas behind a load balancer — do not split one model across both cards. A tensor/layer split gained only about 10% on decode here, because the PCIe hops it adds very nearly cancel the extra memory bandwidth. It did buy ~35% on prefill, so it is not useless; it is just the wrong trade for most workloads. Two replicas double your throughput instead.

cp env.example .env            # set VLLM_API_KEY, REPO_DIR
./preflight.sh                 # power cap, image, ports, disk
./up.sh --wait                 # both replicas + LB, block until healthy
# traffic goes to :18000 — the replica ports are for debugging and telemetry

Round-robin, not least_conn, and that choice is deliberate: least_conn sends more traffic to whichever replica is currently faster, so if you are ever comparing two configurations the faster arm also gets a larger and differently-distributed sample. Equal weights keep the comparison honest.

Thinking on? Serve Swift beside it

With thinking enabled the base model spends most of its output reasoning, and Swift-Qwen3.8-27B reaches the same answers on about half of it. There is no 4-bit vLLM checkpoint of Swift yet, so it runs as a GGUF through llama.cpp. Q4_K_M is the size that leaves a 24 GB card room for two 40K slots; Q6_K (22.8 GB) does not fit with usable context.

hf download ukisai/Swift-Qwen3.8-27B-GGUF Swift-Qwen3.8-27B-Q4_K_M.gguf --local-dir models/gguf

docker run -d --name swift --gpus '"device=1"' --ipc=host -p 18021:18021 \
  -v $PWD/models/gguf:/models:ro ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/Swift-Qwen3.8-27B-Q4_K_M.gguf --alias swift-qwen3.8-27b \
  --host 0.0.0.0 --port 18021 --api-key $KEY \
  -ngl 99 -fa on -ctk q8_0 -ctv q8_0 \
  -c 81920 -np 2 \                       # two slots of 40K: a 32K thinking answer plus its prompt
  --spec-type draft-mtp --spec-draft-n-max 1 \
  --reasoning-format deepseek --jinja     # thinking arrives in reasoning_content, not content

About 55 tok/s for one user, 20.5 GB of VRAM. To put both models behind one endpoint, route on the request’s model field. nginx cannot read a JSON body from plain config, and a js_set variable used in proxy_pass is evaluated before the body has been read — every request falls through to the default upstream and the wrong model answers with a 404. Read the body in a content handler and redirect internally instead:

# router.js
function route(r) {
  var m = ""; try { m = JSON.parse(r.requestText).model || ""; } catch (e) {}
  var up = m.toLowerCase().indexOf("swift") !== -1 ? "swift" : "base";
  r.internalRedirect("/_" + up + r.uri);
}
export default { route };

# nginx.conf (http block; image nginx:1.27-alpine ships the njs module)
load_module modules/ngx_http_js_module.so;
js_import router.js;
upstream base  { server replica-gpu0:18020; }
upstream swift { server replica-gpu1:18021; }
server {
  listen 18000;
  client_max_body_size 32m; client_body_buffer_size 32m;   # body must be in memory for requestText
  location / { js_content router.route; }
  location ^~ /_base/  { internal; rewrite ^/_base(/.*)$  $1 break; proxy_pass http://base;
                         proxy_buffering off; proxy_read_timeout 1800s; }
  location ^~ /_swift/ { internal; rewrite ^/_swift(/.*)$ $1 break; proxy_pass http://swift;
                         proxy_buffering off; proxy_read_timeout 7200s; }
}

Set client timeouts generously on the Swift side. A 32K-token answer at 25–55 tok/s is ten to twenty minutes, and most OpenAI clients give up at ten.

What fits in 24 GB

A 4-bit checkpoint whose lm_head and drafter stay in BF16 costs about 1.5 GB more than one where they are also quantized — enough that speculation stopped fitting on a single card in one of our comparisons. If you are tight on memory, that is the first thing to look at.

Two 4-bit quantizations of the same model measured 32.7 and 32.9 tok/s under identical conditions — indistinguishable. Do not agonize over which 4-bit checkpoint to use; do check whether its lm_head is quantized.

Three things that will waste your afternoon

{"model": "...", "messages": [...],
 "chat_template_kwargs": {"enable_thinking": false}}

client_max_body_size 32m;

Verify it works

curl -s http://127.0.0.1:18000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{"model":"qwen3.8-27b",
       "messages":[{"role":"user","content":"Write a Python function that reverses a linked list."}],
       "max_tokens":200, "temperature":0,
       "chat_template_kwargs":{"enable_thinking":false}}' | jq -r '.choices[0].message.content'

If that returns an empty string, you have hit gotcha 1 above — the tokens were generated and went into a field you are not reading.

Config files

These are the working files from the two-card deployment, with the API key replaced by a placeholder and host-specific paths genericized. Nothing else has been changed.