THE MEMORY WALL. A 125-billion-parameter model on a 12GB card, because a fork of llama.cpp stopped keeping it all in memory. With one very large caveat.
A 12-GIGABYTE GRAPHICS CARD IS RUNNING A 125-BILLION-PARAMETER MODEL AT ABOUT 22 TOKENS A SECOND.
The model takes roughly 68 gigabytes on disk. The machine has 16 gigabytes of system RAM and an ordinary RTX 3060.
And here's the catch, stated plainly because the people behind it state it plainly: the original model supports a context window of 262,144 tokens. This build is running 512. So you are not about to replace ChatGPT on an old gaming PC.
What makes it interesting isn't the number. It's where the weights live.
WHY A MODEL THIS BIG CAN RUN ON A CARD THIS SMALL AT ALL.
Qwen3.8-Flash-Next is a mixture-of-experts model. 125 billion parameters in the main language model — but only around six billion are activated for each token.
Nobody deleted 119 billion parameters. They're all still there. The model is divided into hundreds of specialised experts, and a routing system picks the ones a given token needs instead of pushing that token through every expert.
The numbers get silly fast: 512 routed experts in each mixture-of-experts layer, ten of them activated alongside a shared expert, across 48 layers. That's 24,576 routed experts. Ten of them work on any given token.
Which is the whole opportunity — and the whole problem. Even when an expert isn't being used, its weights have to exist somewhere.
THE OLD WALL, AND WHY IT WASN'T REALLY A MEMORY PROBLEM.
The normal approach is to load as much as possible into GPU memory and offload the rest to system RAM or a drive. But pulling missing weights back and forth destroys throughput.
The way one write-up puts it is the right way: you have a sports car that spends most of its time waiting for somebody to deliver the fuel. You haven't made the car slower. You've made the supply system the bottleneck.
That's why offloading has always been considered a compromise rather than a solution — it works, and it's slow enough that nobody wants to use it.
WHAT BETELGEUSE DOES DIFFERENTLY.
Betelgeuse is an experimental, MIT-licensed fork of llama.cpp built around one idea: demand paging.
Instead of requiring every expert to sit in GPU memory, it keeps a limited pool of expert slots in VRAM and fills them on demand, pulling the missing experts from an NVMe SSD when a token actually needs them.
The important part is what it doesn't do. It hasn't pruned the model. It hasn't swapped in a smaller one and called it the same thing. It changed how the model accesses its own data.
That's a software fix to what everyone had been treating as a hardware limit — and it's why the demo is a 12-gigabyte card instead of a data-centre accelerator.
AND IT'S ALREADY ON ITS WAY INTO THE MAINSTREAM.
Upstream llama.cpp has a pull request adding out-of-core mixture-of-experts streaming with a VRAM slot pool and a RAM victim cache — the same mechanism, arriving in the tool everybody uses.
There are sibling projects doing variants of it: Guanaco keeps the hot experts resident in RAM and streams the rest; BlobPager implements demand-paged serving on a single consumer card; llama-moe-cache caches host-resident experts in VRAM.
This is what a technique looks like when it's about to stop being an experiment.
WHAT IT ACTUALLY MEANS, BEYOND ONE DEMO.
For a decade the cost of running a big model has been set by memory: how much of it you have, and how fast you can move data in and out of it. That's the entire reason a Mac Studio with 512 gigabytes costs $22,299, and the reason Nvidia's biggest constraint isn't compute.
If software can page experts from a commodity SSD fast enough to be usable, the moat gets shallower. Not gone — a 512-token context is not a 262,144-token context, and nobody is running a coding agent on this — but the direction is unambiguous.
The expensive part of local AI was never the arithmetic. It was the capacity.
THE HONEST LIMITS, BECAUSE THE DEMO IS A DEMO.
Context is the big one: 512 tokens against a model trained for 262,144 is a fraction of a percent of its range. That makes it a demonstration of feasibility, not a daily driver.
Then the mechanical problems: expert thrashing when a workload hops between experts, SSD endurance if you do this constantly, and the fact that a 12-gigabyte card still wants 16 gigabytes of system RAM alongside it.
And the ecosystem caveat: this fork is CUDA-oriented. There are RFCs and proof-of-concept branches for offloading experts to disk on Apple Silicon — tested on a 36GB M3 Pro and on a 16GB laptop with a 22GB model — but that path is experimental, not shipping.
SPLASH'S READ: This is the other half of the Mac Studio piece, and it's the more interesting half.
Apple sells you the memory once and you keep it. Anthropic rents you the same capability by the month. Both charge almost exactly the same amount, because both are selling the same quantity.
Betelgeuse attacks the premise. If the weights can live on a drive and only the working set needs to be resident, then the quantity you have to buy stops being the size of the model and starts being the size of the token.
Which is the ownership argument again, one layer down: the people selling memory are selling a constraint that software keeps finding ways around. New Balance kept its factories and won. Vans sold its machine and spent forty years selling an idea. Here the factories are the SSDs, and they cost a lot less than the GPUs.
The model didn't get smaller. The machine stopped pretending it had to hold all of it.