LLM serving on dual RTX 3090s
A pair of full-stack writeups on serving frontier open-weight models on a 2× RTX 3090 (48 GB total) box. Same hardware for both posts, different model families, several inverted findings.
Posts
2026-05-01Qwen 3.6
From 25 to 283 tok/s — Serving Qwen3.6 on Dual RTX 3090s
Two rounds: the 27B dense (9× over baseline, 225 tok/s aggregate via 2 replicas + LB + MTP n=5 + Genesis patches), then the 35B-A3B MoE (peak 283 tok/s aggregate via TP=2 + expert-parallel — same per-stream ceiling, totally different mechanism).
2026-05-05Granite 4.1EAGLE-3
From 93 to 1222 tok/s — Serving IBM Granite 4.1 on Dual RTX 3090s
Three rounds: the 8B dense (1222 tok/s aggregate at C=64, 2624 peak), the 30B with a quantization-format gotcha that breaks one INT4 variant entirely, and a from-scratch EAGLE-3 head trained in 30 minutes for +13.5% single-stream throughput. Several findings invert the Qwen post's conclusions.