Yuval Avidani
Author
The open-source tool Strata runs a 125-billion-parameter model on a regular gaming PC, for anyone who wants serious AI without sending a single word to the cloud. On an RTX 5070 with just 12GB, it writes 94 tokens per second, according to the measurements in the repo. In under a month, the project picked up nearly 28,000 stars on GitHub, and that's not a coincidence.
Turns out it's not one trick. Three techniques matured in exactly the same month: a team of small experts instead of one giant brain, aggressive number compression, and smart streaming of model parts from disk. In this piece I'll break down each one from scratch, and at the end we'll actually install this on a real machine.
So we don't get lost, I'll walk the whole article through one computer: a home gaming PC with a 12GB RTX 5070, 64GB of RAM, and an SSD. This is exactly the machine the Strata developer used to measure his numbers, and in every section we'll ask the same question: how does this giant model actually fit on it?
The situation: 125 billion numbers don't fit in 12GB
Let's start with the basics. A language model is basically a giant file of numbers, called parameters or weights. Each number is like a tiny tuning knob set during training, and together they determine how the model responds. The model Strata runs, Qwen3.8-Flash-Next, contains 125 billion of these knobs, with a context window (the amount of text the model holds in its "head" during a conversation) of 262,000 tokens.
Now the math. In the standard format for storing models, each parameter takes up 2 bytes, meaning roughly 250GB for a model like this. Even in the friendly compressed Q8_0 version (eight bits per number) the download weighs 188GB, according to Strata's size table. And on our machine? 12GB on the graphics card and 64GB of RAM.
For this to run fast, the weights need to sit close to the GPU, in VRAM (the graphics card's dedicated memory). Think of it like a kitchen: what's on the counter is instantly available, what's in the pantry takes a short walk, and what's in storage out in the yard takes a lot of walking. 12GB is a very small counter. Until recently, a model like this demanded a server with hundreds of gigs of graphics memory.
Trick one: a team of experts instead of one brain (MoE)
Here's where the most important trick comes in. An MoE model (Mixture of Experts) is a model split into many small "experts," and for each word only a few of them get activated. Think of a big hospital: it has hundreds of doctors, but a patient with knee pain meets one orthopedist and one nurse, not the entire staff. The hospital is huge, but every visit is small.
In the Qwen3.8-Flash-Next model it looks like this, per the model card on Hugging Face: 512 experts in every layer, and for each token only 10 "routed" experts are activated plus one shared expert. A small component called a router decides, for every word, which experts handle it. All told, out of 125 billion parameters, only about 6 billion actually work on any given token. That's roughly 5% of the model.
The cool part is that MoE solves the compute problem, but not the storage problem. All 512 experts still need to be available somewhere, because you can't know in advance who'll be needed for the next word. And this is exactly where Strata does what it literally describes as a kitchen: the busiest experts sit on the graphics card (the counter), all the experts sit in RAM (the pantry), the regular CPU works in parallel on experts that didn't make it onto the card, and a large lookup table of about 29GB stays on the SSD (the storage shed). On our machine, the graphics card only holds what's actually "hot" right now.
There's another layer of speed here: a small helper model guesses a few words ahead, and the big model checks them all in one shot and keeps whatever's correct. According to the README this produces the exact same answer, 1.6 to 1.8 times faster.
Trick two: compressing all the way down (quantization and ternary weights)
Even with smart distribution, 250GB is still way more than 64GB. So we compress. Quantization is compressing every weight into a number with fewer bits, in exchange for a small loss of precision. Think of supermarket price tags: instead of writing $12.37, you write "about $12." For most decisions that's enough, and the list gets a lot shorter.
When we see Q2 or IQ2 in a filename, it roughly means two bits per weight, instead of 16 in the full format. Strata offers several compression levels of the same model: Q2_0, the fastest, needs 37.6GB of RAM and VRAM combined; the recommended IQ2_XS needs 39.2GB; and IQ3_S, the heaviest at 54.8GB, is, according to the project, the one that matches the full model on published benchmarks. On our machine, with 64GB, every size fits. On a 32GB machine, the project recommends the Coder version, which kept only 256 of the 512 experts, selected for code, and it's weaker outside of code and in non-English languages. And that's relevant for us, because we're writing in Hebrew.
And what if we compress even further? This is where ternary weights come in. Instead of a number with many possible values, each weight can only be one of three: minus 1, zero, or plus 1. Think of a light switch with three positions instead of a dimmer. Sounds like you'd lose everything, but in practice they add one precise "scale" number per group of weights, and that recovers a lot of the precision.
The Bonsai 2 27B model from PrismML, released on September 17 under an Apache 2.0 license, is the fresh example here. A 27-billion-parameter model, based on Qwen3.8 27B, compressed down to just 5.9GB, with 98.2% of the full version's performance according to the company. That's 1.76 bits per weight on average, more than 9x smaller than the original. Worth knowing: the 5.9GB is the most compressed package, while the official demo downloads a faster 7.8GB package by default. Either way, it fits comfortably on a 12GB card.
The chart tells the whole story: the same 125-billion-parameter model jumps from 188GB down to 37.6GB with two-bit compression, a 35-billion-parameter MoE model drops from 19.5GB to under 3GiB of active memory thanks to streaming, and Bonsai 2 27B sits entirely in 5.9GB.
Trick three: streaming experts from the SSD (Edge0)
The third trick asks a cheeky question: why hold all the experts in memory at all? If every word only needs a few, you can leave all of them on the SSD and pull out only what's needed, exactly at the right moment. The open-source inference engine Edge0 streams expert weights from the SSD on demand, so the memory you need depends on the active experts, not the size of the model.
The problem is timing. According to the Edge0 paper (arXiv, September 2026), naive streaming doesn't help, because the experts for the next layer are only chosen after the current layer finishes, and then you're just waiting on the disk. Their solution is called a prerouter: a small trained head that guesses in advance which experts will be needed in the next layer, and starts loading them from disk while the current computation is still running. Think of a good waiter who's already ordering your next course from the kitchen while you're still halfway through the first one. On top of that, a small LoRA adapter (a tiny correction layer trained on top of the model) recovers some of the quality lost from compressing down to 4 bits.
The numbers are impressive: a 35-billion-parameter MoE model, which at 4-bit compression weighs 19.5GB, runs according to the README at about 15 to 18 tokens per second on a Mac mini M4 Pro with 24GB, with peak active memory sitting at just 2.9GiB. The average across five benchmarks is only 3.9 points lower than the full model.
And here's where I want to be precise, because online this gets mixed up: the Edge0 engine doesn't currently run the 125-billion-parameter model. Its models are 35B and 8B, and support for the Qwen3.8-Flash family architecture is on the roadmap for Q4 2026. Strata itself also uses a similar idea on a smaller scale: if RAM isn't enough, it reads some of the experts from the SSD on the fly, and that's a lot slower.
From theory to practice: running 125B on our machine
Let's go back to our machine and install this. According to Strata's README, here's what you need: an NVIDIA card from the RTX 20 through 50 series, or a supported AMD card, with at least 12GB VRAM; at least 32GB RAM, with 64GB letting you run every size; about 80GB free on disk, preferably SSD; and Windows 10/11 or Linux with an up-to-date driver. On a Mac with an Apple chip there's experimental support that compiles the engine from source.
The process itself is short. On Windows, double-click START-HERE.bat, and on Linux run ./setup.sh. The installer detects your card, asks which size, how much context, and whether to read images, and you can just hit Enter for all the recommendations. The download weighs about 70GB, and at the end your browser opens on http://127.0.0.1:8080. Here's the Linux version:
Two things worth knowing in advance. First: on the first launch the computer can freeze for one to three minutes, because Strata is loading 35 to 55GB into memory. That's normal, don't close the window. Second: the server speaks both OpenAI's language and Anthropic's language, so you can hook up Claude Code to it with a single environment variable, ANTHROPIC_BASE_URL, pointing at the local address. As far as Claude Code is concerned, it's simply talking to a different server.
And if our machine is weaker? Bonsai 2 27B is a much gentler entry point. Clone PrismML's Bonsai-demo repo, run ./setup.sh and then ./scripts/start_llama_server.sh, and you get a chat with images and tools on that same local address. If you haven't run any local model yet, start with our guide to Ollama, LM Studio and Unsloth, and only then jump to 125B.
In my view: what this is, and what it still isn't
In my view, this is one of the most significant moments of the year for local models. Not because of one single number, but because three directions converged: an architecture that activates only a small slice of the model, compression that barely dents quality, and engineering that spreads the model across the graphics card, RAM and SSD. Each one alone would have been interesting. Together, they turn a model that used to need a server into something that runs right next to your keyboard.
But let's be fair about the limitations:
- A "regular gaming PC" is actually a fairly strong PC. 12GB of VRAM is the minimum, and 32GB of RAM only gets you the Coder version. To enjoy the full model you need 48 to 64GB.
- Two-bit compression is a compromise. The project itself defines Q2_0 as "good" and IQ3_S as "best." The 94 tokens/sec figure is on the most compressed version, and at a 128K-token context it drops to 76.
- One request at a time. By default, Strata answers one request while the rest wait in line. This is a personal assistant, not a server for a team.
- Numbers floating around online. I came across a claim about "RTX 4090 at 100 tokens/sec." I couldn't find such a measurement, not in the README and not in the project's community reports. What's written there is an estimate, not a measurement: an RTX 3090 with 24GB is expected to reach 100 to 140 tokens/sec. Until someone actually measures it, I'm treating that as an estimate.
- Everything here is brand new. Strata, Edge0 and Bonsai 2 are all less than a month and a half old. What's true today might change next week.
Bottom line: if you've got a gaming PC with 12GB VRAM and 64GB RAM, you can run a 125-billion-parameter model on it tonight, completely private, and connect your coding tools to it. This isn't a lab experiment anymore.
So what's the first task you'll hand off to a model running in your own home, no cloud involved at all?
Want to take this further?
The whole world of local models, from VRAM to picking the right model for your hardware, I break down step by step in my course Local LLMs End to End, in Hebrew, where the data stays with us. This is the foundation that turns Strata and Bonsai from something intimidating into something you install in one evening.
And since we're already connecting a local model to Claude Code, the Claude Code Mastery course shows how to actually work with it properly.
I post my experiments running on real hardware to YouTube and LinkedIn, and short updates on X, Instagram and TikTok. My code lives on GitHub.
And now the real question: when a 125-billion-parameter model is running in your own home, which tasks are you still willing to send to the cloud, and why?