⌕

NO, AN 85 GB MODEL DOESN'T RUN FROM YOUR SSD

NOV 10, 2026 · SPLASH · SOURCED FROM BART EXPLAINS · THE MATH
Running a large model on limited VRAM

THE CLAIM. A 177-billion-parameter model, 85 GB of weights, running at about 35 tokens a second while using 6.6 GB of graphics memory. The internet has a name for it — "SSD streaming" — and the name is mostly wrong. The real mechanism is better, and once you understand it you can size any machine before you download a single file.

WHY IT SHOULDN'T WORK. Every token, the chip has to read the model's numbers and do math on them. So speed depends on two things: how many gigabytes each token needs, and how fast the memory holding them can deliver.

Three tiers of memory, in order:

TIERSPEEDROLE
Graphics memory (VRAM)fastestthe parts every token touches
System RAMslowerthe experts
SSDabout 15 GB/s at bestthe lookup table

Do the naive math: 85 GB read off the drive for every token at 15 GB/s is about 6 seconds per token. A 300-token answer would take half an hour. That's the version everyone assumes, and it's why "SSD streaming" sounds like a miracle.

WHY IT ACTUALLY WORKS — PART ONE: THE MODEL DOESN'T USE ITSELF. The model card splits the 177 billion parameters into a 125 billion main model and a 51 billion lookup table. The main model is a mixture of experts: 48 layers, each holding 512 small blocks called experts, plus one shared expert that always runs. A tiny router scores each token and picks the 10 experts that fit it. The other 500-odd sit idle.

Add it up and only about 6 billion of the 125 billion main-model parameters are used per token — under 5 percent. That's the first half of the answer.

PART TWO: THE SPLIT. The catch is that the next token may need a completely different ten experts, so all of them have to be reachable — and 512 experts across 48 layers do not fit in 6.6 GB of VRAM. So the lab split the model across all three tiers:

  • Graphics card (6.6 GB): attention and everything every token touches.
  • System RAM (~48 GB): every expert, run on the CPU.
  • SSD: the lookup table.

In the engine's notes, the setting that moves experts onto the processor is described as "the flag that makes it fit" — without it, the engine wants about 52 GB of VRAM. If you're on llama.cpp, the equivalents are --n-cpu-moe (keep experts on the CPU) and --override-tensor (keep the table off the card).

NOW THE PART THAT KILLS THE MYTH. Add up what each tier actually reads per token. The lookup table works by address: take the last few tokens, turn them into an address, read one small row. ~2.7 kilobytes per token. The experts in RAM: ~2.3 gigabytes per token.

The SSD reads roughly 850,000 times less data than RAM for every token.

A 2.7 KB read is nothing — a normal NVMe drive answers it in under a tenth of a millisecond, against a ~30 ms token. And the engine barely has to try: the method is mmap, which treats the file on disk as if it were memory, and the OS pulls in only the pieces used. Anything already read stays in the page cache, so on a machine with spare RAM the drive may not be touched at all.

Which is why the famous test deserves the fine print: the "8 GB GPU" was a 16 GB card using 6.6 GB, behind a 64-core Threadripper with 1 TB of RAM and an Optane drive. The lab's own writeup says it plainly: "On our 1 TB box, everything left over became page cache, and the table was never really read."

THE FORMULA YOU ACTUALLY NEED.

Tokens/second ≈ memory read speed ÷ gigabytes per token

DDR5 in a normal two-channel desktop reads about 89 GB/s. Divide by 2.3 GB per token: a ceiling of roughly 38 tokens a second — best case, before overhead. That's why real machines land at 15-22.

Now put the experts on the SSD instead: 15 GB/s ÷ 2.3 GB = about 6 tokens a second at the theoretical maximum, and owners report 1-2. A dense model is worse because it can't skip anything: a 27B dense model at 4-bit is ~15 GB, and reading all of it per token gets you roughly one token a second. That's the wall — it's not that streaming doesn't work, it's that it only works when almost nothing streams.

WHAT REAL PEOPLE GET. A Windows user with a 16 GB RTX 5070 Ti and 64 GB of RAM: about 18 tokens a second. A YouTuber with two budget cards (24 GB total) and 32 GB of RAM: 22, dropping to 14 once the context gets long, and about 20 percent slower on Windows than Linux.

Note the pattern: the machine with less RAM but more VRAM did better, because more of the model sat in the fastest tier. Once the main model fits, spare RAM helps next — it holds more of the lookup table in page cache.

And there's a cost before the first word. Prefill — reading your whole prompt — collapses when the card is capped: 232 tokens/s at an 8 GB cap versus 1,955 with the full card. A 15,000-token document becomes about a minute of waiting, and the test could only hold 16,000 tokens of context at that cap.

IF YOU'RE ON A MAC. Unified memory changes the arithmetic: there's no separate VRAM pool, so the question is simply whether the main model fits in the shared memory. On a 128 GB MacBook Pro, a 4-bit build loaded fully used about 100 GB; a streaming build that left the table on the SSD and kept some layers at higher precision used 70 GB.

THE NEXT VERSION OF THIS. DeepSeek's latest has the same idea at a bigger scale — a table it calls an engram, 196 billion parameters on top of a 552 billion main model. You'll see videos calling it "engram streaming"; no DeepSeek feature by that name exists. What the paper actually says is that rarely used rows can live on slower, high-capacity media — the example given is an NVMe SSD. The engines that do it are community projects, and the main model still needs roughly 150 GB of fast memory at 2-bit.

DRIVE WEAR. Overblown, mostly. SSD lifespan is rated on writes, and this is reads. One 2021 study found some modern drives quietly rewrite data in the background under constant reading, which adds some wear — but reading is still far easier on a drive than writing.

SPLASH'S READ: The trick isn't compression and it isn't the SSD. It's that a mixture-of-experts model only wakes up 5 percent of itself per token, and the part it wakes up lives in RAM.

That single sentence explains every number in the video, including the disappointing ones. The 85 GB file never had to fit anywhere. The card held the 6.6 GB that every token touches; the experts sat in 48 GB of RAM; the table sat on the drive doing almost nothing. "SSD streaming" is a marketing phrase for "you have enough RAM for the experts and the OS cached the rest."

So the buying advice is boring and correct. Add your VRAM and your RAM — that sum has to hold the main model, which for these 4-bit builds is roughly 46-55 GB plus context. A 16 GB card and 64 GB of RAM gets you ~18 tokens/s. A laptop with an 8 GB card and 64 GB of RAM lands near 15. And if the weights themselves have to come off the drive — because the model is dense, or the experts don't fit in RAM — you get one or two tokens a second, which is a science project, not a workflow.

For a 32 GB machine, that means the honest ceiling is a model whose active set fits comfortably in 32 GB — a smaller MoE, not a 177B. And for the big ones, the rack or the rented GPU isn't optional; it's the only tier that reads fast enough to feel like a computer.

Sources & claims: Bart Explains (YouTube), "How to Run a 177B AI Model on an 8GB GPU." Benchmarks and figures — 6.6 GB VRAM at ~35 tok/s, the 125B main model plus 51B lookup table, 48 layers × 512 experts with 10 selected per token, ~2.3 GB read per token from RAM versus ~2.7 KB from the SSD, the 64-core/1 TB/Optane test hardware and the lab's "table was never really read" note, the 18/22/14 tok/s owner reports, prefill 232 vs 1,955 tok/s, and the DeepSeek engram figures — are as presented in the video and consistent with published model cards and llama.cpp documentation. Tokens-per-second figures are estimates and vary by build, context length and OS. The "engram streaming" label is not an official DeepSeek term, and the video says so.