Qwen3.8-27B on Two RTX 3060s – 40+ Tokens/s Without a New Engine

Qwen3.8-27B on two RTX 3060s

> 🚀 A 27B model at 44–55 tokens per second on two 12 GB cards with no NVLink — using stock llama.cpp, no patched fork, no exotic quant. Two flags do almost all of the work: tensor split and the model’s own MTP head.

🖼️ The numbers

This is one desktop: Ryzen 7 5700X, 31 GB of RAM, two RTX 3060 12 GB cards with no NVLink — one in the x16 slot, the other in a PCIe 4.0 x4 slot. The model is unsloth/Qwen3.8-27B-GGUF, UD-Q4_K_M, 16.46 GB on disk. Prompt processing sits around 526 tokens per second, and the whole thing answers at 100k context.

✨ What you actually get

• 44–55 tok/s on code, 38–42 tok/s on prose.
• 100,400 tokens of context in ~9.9 / 11.2 GiB per card.
• An OpenAI-compatible endpoint on 127.0.0.1:18435 — Hermes, Open WebUI, anything with a base URL.
• Token-for-token the same answers plain decoding gives. Speculation that is rejected is thrown away, never spoken.

⚡ Decision 1: tensor split, not layer split

A dense model reads every weight once per generated token. Two 3060s bring 720 GB/s of combined bandwidth, and 16.46 GB of weights divided by that gives 24.5 ms per token — a hard ceiling near 40 tok/s. Layer split (llama.cpp’s default) puts whole layers on one card and leaves the other idle during each layer’s work, so you land around 30 tok/s and wonder why the second GPU did nothing.

--split-mode tensor cuts every weight matrix in half across both cards, so each token is fed by both memory controllers at once. It costs flexibility — the model must fit entirely in VRAM, no CPU offload, no partial fit — and it buys the bandwidth of the pair.

⚡ Decision 2: let the model draft its own tokens

Tensor split alone only reaches the bandwidth ceiling. To go past it you need more than one token per pass over the weights, which is what speculative decoding does — and Qwen3.8 ships a native MTP head, so no separate draft model is needed:

--spec-type draft-mtp --spec-draft-n-max 2

The model drafts two tokens, one batched pass validates both, and every accepted draft is a free token. Measured acceptance here: 84–96%, i.e. roughly 1.8–1.9 tokens per pass. That is the entire difference between 30 and 55 tok/s.

⚡ Decision 3: keep the all-reduce inside llama.cpp

Consumer cards have no peer-to-peer DMA. Without a hint, llama.cpp hands the cross-GPU reduction to NCCL, which stages every all-reduce through host memory over PCIe — and in tensor mode that reduction happens on every token. One environment variable fixes it:

GGML_CUDA_ALLREDUCE=internal

With tensor split this is not optional. Forget it and you will conclude tensor split “doesn’t help”.

🧠 The surprise: 100k context fits

I expected to be stuck near 64k. Qwen3.8-27B is not a plain transformer: its metadata declares 65 blocks, but only 17 of them carry attn_k/attn_v tensors — the rest are recurrent (delta-net / mamba2) with a constant-size state. So the KV cache costs roughly 19–33 KiB per token instead of the ~73 KiB the dense arithmetic implies.

The measured result: 64k uses 9,412 / 10,989 MiB, 100k uses 9,936 / 11,226 MiB, and 128k technically loads at 10,412 / 11,753 MiB — leaving 158 MiB, which is one long prompt away from an out-of-memory crash. 100k is the honest ceiling on 2×12 GB. The model’s native window is 256k; getting there needs a third card, a smaller quant, or -ub 256, not a magic flag.

📦 Install

git clone https://github.com/dailybarid/qwen38-dual-3060.git
cd qwen38-dual-3060

# grab llama.cpp with CUDA (b11671 here), then:
llama-server \
  -m Qwen3.8-27B-UD-Q4_K_M.gguf \
  -ngl 99 \
  --split-mode tensor --tensor-split 1,1 \
  -c 102400 \
  -ctk q4_0 -ctv q4_0 \
  -np 1 -fa on \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --jinja --metrics \
  --alias Qwen3.8-27B \
  --host 127.0.0.1 --port 18435

# with the env var that makes tensor split worth it:
GGML_CUDA_ALLREDUCE=internal llama-server ...

The repo also carries a systemd user unit that refuses to start unless both cards are free, start.sh / stop.sh wrappers, and bench.sh, which reports tokens per second and live MTP acceptance so you can tune n-max on your own workload.

⚙️ Settings that matter

• --tensor-split 1,1 — the cards are identical; this is not a memory dial, it splits weights.
• --spec-draft-n-max 2 — 2 measured best here. Higher values keep raising acceptance only until validation cost overtakes the gain, and prose suffers first.
• -ctk q4_0 -ctv q4_0 — 4-bit KV cache, frees ~3 GB and is what makes 100k possible at all.
• -fa on — required for quantized KV on these cards.
• -np 1 — one request at a time; with tensor split, extra slots fight over the same bandwidth.

📉 What it costs

• Load time: 4.5–5 minutes. 16.5 GB read, then a full double upload in tensor mode. Plan restarts, don’t do them casually.
• Heat, not VRAM, is the limit. Sustained prefill of a very long prompt pushed GPU1 to 85–86 °C with fans at 100%. Generation sits at 46–73 °C.
• Two GPUs, one job. Both cards are owned by the model; nothing else can use them while it runs.

⚠️ What it does not do, honestly

• No CPU offload in tensor mode — if it doesn’t fit in the two cards, it doesn’t start.
• No 256k context on 2×12 GB, despite the model supporting it.
• Two log warnings are normal and not errors: llama_params_fit is not implemented for SPLIT_MODE_TENSOR (you size the context yourself) and backend offload failed ... using CPU sampler (sampling runs on the CPU, not the bottleneck at 50 tok/s).
• Mismatched PCIe links work — here an x4 card next to an x16 — but the slow link limits the all-reduce. Matched x16/x16 should be somewhat faster.

📜 License

The method write-up, unit file and scripts are MIT licensed. llama.cpp is MIT, Qwen3.8-27B is Apache-2.0, the GGUF quant is by unsloth.

💡 Author

Khaled Mimoune — I run what I write about. Code and exact commands: github.com/dailybarid/qwen38-dual-3060. If you have the same pair of cards, the numbers above are reproducible on your box; if you have three, the 256k window is waiting.