A 180B Model on Two RTX 3060s: What TokenWing Can and Cannot Do

Qwen3.8-Flash-Next 180B running on two RTX 3060 cards

Qwen3.8-Flash-Next 180B running on two RTX 3060 cards

There is a class of tools that promises something the hardware cannot do, and when you finally run one you find out which half of the promise was real. This is a record of running a 180-billion-parameter mixture-of-experts model on a desktop with two 12 GB cards and 31 GiB of RAM — what loaded, how fast it went, and the measurement that explains why it went that fast and no faster.

The short version: it works, and it is too slow to use interactively. The engine is not the problem. Neither is the SSD, which is what I expected to be the problem. The problem is the arithmetic the CPU has to do.

The machine, and the row that decided everything

CPU Ryzen 7 5700X, 8 cores / 16 threads
RAM 31 GiB DDR4
GPUs 2× RTX 3060 12 GB (one PCIe 4.0 ×4, one ×16, no P2P)
Storage two SATA SSDs — no NVMe anywhere in this box
OS Linux Mint 22.3, kernel 7.0.0, CUDA driver 13.2

That storage row is the one that matters. Every published benchmark for this class of engine comes from machines with an NVMe drive, and the difference between 0.5 GB/s with high latency and 7 GB/s with low latency is the difference between the numbers you see in a README and the numbers you get at home.

What was actually run

TokenWing is an MIT-licensed fork of llama.cpp that streams Mixture-of-Experts weights from disk. Each MoE layer has a router that picks a few experts per token; the engine predicts the next two layers’ choices from the current hidden state, reads those experts with direct I/O into a RAM cache it manages itself, and overlaps the reads with compute. GPUs compute the experts they hold, the CPU computes the rest, and the outputs are summed. The project is unusual in that every mechanism in it credits the paper or project it came from, and the limitations section is honest.

The model was Qwen3.8-Flash-Next: 180B total parameters, about 6B active per token, quantized to 3.5 bits per weight by Grawnch — 84 GB across two shards, plus a 0.48 GB router correction file and two small data files the engine needs beside the weights. It does not run in any normal GGUF loader; the file layout is TokenWing’s own, with every expert aligned to a 4 KiB boundary so it can be read straight from disk.

Everything was served with the authors’ own published flag set for a 12 GB card and 32 GB of RAM. Nothing was hand-tuned: the engine chose its own split between whole layers on the GPU and a VRAM resident expert cache at startup.

What it allocated

At 128k context with q8_0 KV cache:

RAM pools     21.24 GB   10,395 experts   45 streamed layers   read at 0.28 GB/s (75.7 s)
VRAM pool      2,418 experts   36 per streamed layer + whole layers
KV             128k tokens, one slot
healthy after  80 s        GPU0 11.6 GB used, GPU1 6.8 GB used

At 100k with q4_0 KV the RAM pool grows to 25.46 GB / 12,465 experts, because the smaller KV cache and the smaller context leave more room, and VRAM holds 40 experts per layer plus a 950-place tail. Healthy after 130 seconds.

Read that again: a 180B model, 25 GB of it resident in RAM, 17 GB of it resident in VRAM, the rest streaming. On a desktop.

Speed: three configurations, two states each

Decode was measured on a 100–120 token completion of the same prompt, twice: once cold with nothing cached, and once warm by re-sending the identical prompt so that 98–99.9 % of the experts came from RAM and VRAM instead of the disk.

Configuration Decode, cold Decode, warm Prefill, 1.4k prompt
128k, q8_0 KV 3.84 t/s 8.89 t/s 23.2 t/s
128k, q4_0 KV 2.97 t/s 17.34 t/s 27.0 t/s
100k, q4_0 KV 3.40 t/s 17.38 t/s 33.1 t/s

Decode speed cold and warm at three KV cache settings

The q4_0 result was the surprise. Halving the KV cache gives back roughly 1.6 GB of VRAM, and the engine spends it on four more experts per layer in its VRAM pool — 36 to 40. Those experts are then multiplied on the GPU instead of the CPU, and warm decode roughly doubles. It is not faster attention arithmetic; it is a different processor doing the multiply. On a platform where the CPU is the bottleneck, buying GPU occupancy with VRAM is the highest-leverage trade available.

Long prompts behave far better than short ones, and this is the number that made me think the setup might be usable:

29,146-token prompt, cold:   147 t/s    (2.0 MB/token read)
22,528-token prompt, warm:   258 t/s
 1,394-token prompt, cold:    33 t/s    (12.1 MB/token read)

A long prompt touches most experts in every layer exactly once, so the read cost is amortised over thousands of tokens. A short prompt pays the full miss cost on every token.

The experiment that explains everything

I re-sent the same prompt with the cache saturated: 98.96 % hit rate, 0.95 % of experts still coming from the SSD. Decode was 8.89 t/s at 128k with q8_0 KV.

Nothing was being read from the disk, nothing was missing from memory, and it still ran at under nine tokens per second. That is the whole story. The engine was bound by the CPU computing its share of the experts, not by the SSD and not by RAM capacity.

Which means the upgrade order I had in mind was wrong:

  • A faster SSD buys about 2× — from ~3.8 t/s cold to the ~9 t/s ceiling. Not the 6–15× seen on NVMe desktops.
  • More RAM buys nothing — the hot-cache run already had most of the model resident.
  • More VRAM buys real speed, because it moves expert multiplication from the CPU to the GPU. That is exactly what the q4_0 KV experiment demonstrated, in miniature: 1.6 GB of VRAM bought a doubling.

Quality and tool use

Speed means nothing if the model is worse, so it went through the same gates as the local models already in use:

  • 20 hidden-assert coding tasks: 20/20, at 128k/q8_0 (7.5 t/s average, 7.4 minutes) and again at 128k/q4_0 (9.1 t/s, 4.8 minutes). Our Qwen3.8-27B preset scores 20/20 on the same bank — so no gain, no loss, on this test.
  • Tool choice, 12 prompts against 15 tool schemas: 9/12, against 11/12 for the 27B on the identical harness. All three misses picked terminal where a more specific tool was correct — searching files, spawning a subagent, running code.
  • A real tool call returned two correct parallel calls (read_file on /etc/hostname, terminal for uname -r) — but took 72 seconds of wall time for 55 generated tokens.
  • Retrieval at 6.5k tokens with q4_0 KV was exact: a fingerprint and hostname buried mid-context both came back correctly. I did not attempt a 64k retrieval test, which would have cost about 40 minutes of prefill.

Inside a real agent session

Served as an OpenAI-compatible endpoint and registered as a provider in the agent harness, the first turn of a session — a 29,000-token system prefix — took 8 minutes and 31 seconds: 29k tokens of prefill at 147 t/s cold, then a second 22.5k pass at 258 t/s.

After that, generation settled at 6.4–6.8 t/s on real answers, with the best three-second windows reaching 11 t/s.

Decode speed: production 27B versus the 180B model at four levels of realism

That is the honest number. A tool-using turn is a few hundred generated tokens plus prompt growth, so at 6.5 t/s it is minutes per turn. It is not “slow” in the way a benchmark at 17 t/s suggests; it is slow in the way that makes you stop using it.

Verdict

TokenWing did exactly what it advertises. A 180B model ran on hardware that could not otherwise hold it, at speed appropriate for batch work — long-document analysis, overnight code review, bulk summarisation — and it passed a quality gate that local 27B models also pass. On a DDR5 machine with an NVMe drive the same engine reports 57–65 t/s. This machine has neither the disk nor the memory bandwidth, and its CPU is the wall.

So: not adopted for daily use, and worth revisiting only after a hardware change that actually matters. Not the SSD. More VRAM, or a faster CPU. If you are planning an upgrade and this article is the reason, plan it around the experts the CPU has to multiply, not around gigabytes per second on a drive.

Everything measured here — the exact serving command, the 20-task bank, the tool-choice harness and the speed harness — is in the repository, so the numbers can be re-run on your own machine rather than taken on trust:

github.com/dailybarid/tokenwing-flashnext-2×3060

Measured on the desktop described above, November 2026. TokenWing v0.1.0 (MIT), Qwen3.8-Flash-Next under the Qwen Community License 1.0. No weights are redistributed here.

share:


Discover more from Khaled Mimoune

Subscribe to get the latest posts sent to your email.

Comments

Leave a Reply