⌕

THE SSD IS THE NEW VRAM

NOV 14, 2026 · SPLASH · MONEY · THE MEMORY WALL
Betelgeuse llama.cpp fork NVMe SSD demand paging MoE experts 125B Qwen3.8 Flash Next RTX 3060 12GB 22 tokens per second

THE MEMORY WALL. A 125-billion-parameter model on a 12GB card, because a fork of llama.cpp stopped keeping it all in memory. With one very large caveat.

A 12-GIGABYTE GRAPHICS CARD IS RUNNING A 125-BILLION-PARAMETER MODEL AT ABOUT 22 TOKENS A SECOND.

The model takes roughly 68 gigabytes on disk. The machine has 16 gigabytes of system RAM and an ordinary RTX 3060.

And here's the catch, stated plainly because the people behind it state it plainly: the original model supports a context window of 262,144 tokens. This build is running 512. So you are not about to replace ChatGPT on an old gaming PC.

What makes it interesting isn't the number. It's where the weights live.

WHY A MODEL THIS BIG CAN RUN ON A CARD THIS SMALL AT ALL.

Qwen3.8-Flash-Next is a mixture-of-experts model. 125 billion parameters in the main language model — but only around six billion are activated for each token.

Nobody deleted 119 billion parameters. They're all still there. The model is divided into hundreds of specialised experts, and a routing system picks the ones a given token needs instead of pushing that token through every expert.

The numbers get silly fast: 512 routed experts in each mixture-of-experts layer, ten of them activated alongside a shared expert, across 48 layers. That's 24,576 routed experts. Ten of them work on any given token.

Which is the whole opportunity — and the whole problem. Even when an expert isn't being used, its weights have to exist somewhere.

THE OLD WALL, AND WHY IT WASN'T REALLY A MEMORY PROBLEM.

The normal approach is to load as much as possible into GPU memory and offload the rest to system RAM or a drive. But pulling missing weights back and forth destroys throughput.

The way one write-up puts it is the right way: you have a sports car that spends most of its time waiting for somebody to deliver the fuel. You haven't made the car slower. You've made the supply system the bottleneck.

That's why offloading has always been considered a compromise rather than a solution — it works, and it's slow enough that nobody wants to use it.

WHAT BETELGEUSE DOES DIFFERENTLY.

Betelgeuse is an experimental, MIT-licensed fork of llama.cpp built around one idea: demand paging.

Instead of requiring every expert to sit in GPU memory, it keeps a limited pool of expert slots in VRAM and fills them on demand, pulling the missing experts from an NVMe SSD when a token actually needs them.

The important part is what it doesn't do. It hasn't pruned the model. It hasn't swapped in a smaller one and called it the same thing. It changed how the model accesses its own data.

That's a software fix to what everyone had been treating as a hardware limit — and it's why the demo is a 12-gigabyte card instead of a data-centre accelerator.

AND IT'S ALREADY ON ITS WAY INTO THE MAINSTREAM.

Upstream llama.cpp has a pull request adding out-of-core mixture-of-experts streaming with a VRAM slot pool and a RAM victim cache — the same mechanism, arriving in the tool everybody uses.

There are sibling projects doing variants of it: Guanaco keeps the hot experts resident in RAM and streams the rest; BlobPager implements demand-paged serving on a single consumer card; llama-moe-cache caches host-resident experts in VRAM.

This is what a technique looks like when it's about to stop being an experiment.

WHAT IT ACTUALLY MEANS, BEYOND ONE DEMO.

For a decade the cost of running a big model has been set by memory: how much of it you have, and how fast you can move data in and out of it. That's the entire reason a Mac Studio with 512 gigabytes costs $22,299, and the reason Nvidia's biggest constraint isn't compute.

If software can page experts from a commodity SSD fast enough to be usable, the moat gets shallower. Not gone — a 512-token context is not a 262,144-token context, and nobody is running a coding agent on this — but the direction is unambiguous.

The expensive part of local AI was never the arithmetic. It was the capacity.

THE HONEST LIMITS, BECAUSE THE DEMO IS A DEMO.

Context is the big one: 512 tokens against a model trained for 262,144 is a fraction of a percent of its range. That makes it a demonstration of feasibility, not a daily driver.

Then the mechanical problems: expert thrashing when a workload hops between experts, SSD endurance if you do this constantly, and the fact that a 12-gigabyte card still wants 16 gigabytes of system RAM alongside it.

And the ecosystem caveat: this fork is CUDA-oriented. There are RFCs and proof-of-concept branches for offloading experts to disk on Apple Silicon — tested on a 36GB M3 Pro and on a 16GB laptop with a 22GB model — but that path is experimental, not shipping.

SPLASH'S READ: This is the other half of the Mac Studio piece, and it's the more interesting half.

Apple sells you the memory once and you keep it. Anthropic rents you the same capability by the month. Both charge almost exactly the same amount, because both are selling the same quantity.

Betelgeuse attacks the premise. If the weights can live on a drive and only the working set needs to be resident, then the quantity you have to buy stops being the size of the model and starts being the size of the token.

Which is the ownership argument again, one layer down: the people selling memory are selling a constraint that software keeps finding ways around. New Balance kept its factories and won. Vans sold its machine and spent forty years selling an idea. Here the factories are the SSDs, and they cost a lot less than the GPUs.

The model didn't get smaller. The machine stopped pretending it had to hold all of it.

Sources & claims: SOURCES, CLAIM BY CLAIM — The demonstration: Betelgeuse is an experimental MIT-licensed fork of llama.cpp that implements demand paging for mixture-of-experts weights, keeping a fixed pool of expert slots in GPU memory and retrieving absent experts from NVMe storage. Reporting and the project's own results describe a roughly 68GB Qwen3.8-Flash-Next build running at approximately 22 tokens per second on an RTX 3060 with 12GB of VRAM and 16GB of system RAM, configured with a 512-token context against the model's native 262,144-token window. The model architecture: Qwen3.8-Flash-Next is a mixture-of-experts model of about 125 billion parameters in the language model with roughly 6 billion active per token, 512 routed experts per mixture-of-experts layer with 10 routed experts plus a shared expert activated per token, across 48 layers — about 24,576 routed experts in total. The mainstreaming: llama.cpp has an open pull request adding out-of-core mixture-of-experts expert streaming with a VRAM slot pool and a RAM victim cache (--moe-direct-io), and related projects including llama-moe-cache, Guanaco and BlobPager implement variants of the approach. The comparison of slow offloading to a vehicle waiting on fuel delivery paraphrases the source video's explanation. Apple Silicon: llama.cpp discussion threads describe proof-of-concept and RFC work on on-demand expert paging tested on Apple hardware including a 36GB M3 Pro and a 16GB laptop running a 22GB model; those efforts are described by their authors as experimental and are not shipping features in the same way as the CUDA path. The framing of memory capacity as the binding constraint on local AI economics, the connection to Apple's unified-memory pricing and to subscription-based compute, and the ownership analogy are this piece's own analysis. No claim is made here that this technique replaces datacentre hardware or that a 512-token context is usable for general workloads; the article states the opposite. Source video: This New llama.cpp Fork Turns Your SSD Into VRAM (22 Tok/s) (Cian Mag, 12 November 2026) — used as the prompt; the model architecture, throughput, hardware configuration and context figures above were checked against project documentation and independent write-ups of the same build.