Yuval Avidani
Author
"Colibri takes a 744-billion-parameter model - the kind that until yesterday only ran on a server rack worth more than an apartment - and squeezes it into a laptop with 25 gigs of regular memory, no graphics card at all." That's how I'd sum up the craziest thing I saw this week. When I first read it, I thought someone had gotten the numbers mixed up. But it turns out it actually works, and the real story here isn't how fast it runs - it's why it's even possible.
Before we dive in, one small correction worth making: the machine needs ~25 gigs of regular RAM - normal computer memory - not VRAM (graphics card memory). Colibri doesn't need a graphics card at all. And that's actually more impressive, because a graphics card with 25 gigs costs a fortune - regular computer memory you probably already have at home.
The Wall We All Accepted as a Given
To run a language model on our computer, the basic rule says you need to hold all of its weights in memory at once. A weight (parameter) is one small number out of billions that the model learned during training - think of it like a tiny volume knob, and every model is a panel with billions of these knobs tuned to just the right position. To "think" a single thought, the model needs fast access to these knobs, which means they have to sit in memory.
Now let's do the math. GLM-5.2 has 744 billion parameters. Even in an efficient format called int4 (quantization - storing each number in 4 bits instead of 16, like writing a price as "about 20 shekels" instead of "19.87") each parameter weighs half a byte, so the whole model weighs about 370 gigabytes. A typical laptop has 16 to 32 gigs. That's like trying to pour 370 liters into a 30-liter bucket. It's not that you're short on money - you're short on physical space.
And that's why "a frontier model running at home" has been a fantasy until now. Not a patience problem you could solve by waiting longer - a wall. To hold 370 gigs of weights you'd need roughly 5 NVIDIA H100 cards - server hardware worth as much as a small apartment.
Why the Usual Shortcuts Don't Really Solve This
Until now there were two ways to squeeze a model onto a small machine, and both come with a price tag. You can train a smaller model that mimics the big one (distillation) - but then you get a weaker model. Or you can quantize more aggressively - but at some point quality drops off a cliff.
And here it's worth being fair to the critics. One comment I got was: "This is just a quantized model that gives you an answer a day later." Half of that is true, and it's worth being precise. Colibri doesn't swap the model for a dumber version. It runs the real, full GLM-5.2, in int4 - the exact same weights a data center would run. int4 is a widely accepted standard that barely dents quality on large models. So the quality we're getting is the quality of the real model, not a distilled shadow of it. The hard problem Colibri solves isn't quality - it's how to shove 370 gigs into 25.
The Trick: Don't Hold the Whole Model - Just What's Working Right Now
Here's the beautiful part, and it breaks down into three moves.
Move one - MoE. GLM-5.2 is built as a "Mixture of Experts": instead of one giant network that's all active at once, every layer has 256 small "experts," and a tiny component called a router picks only about 8 of them for each token. The result: out of 744 billion parameters, only about 40 billion are actually active per token - roughly 5 percent. Think of it like a hospital with 256 specialist doctors: every patient only meets eight of them. There's no reason all 256 need to stand in the room for you.
Move two - split the memory into two floors. The "always needed" part - the attention mechanism, the embeddings, and the shared experts, about 17 billion parameters weighing ~9.9 gigs in int4 - Colibri keeps permanently in RAM. The 21,504 routed experts (75 layers times 256, about 19 megs each) it leaves on the SSD, the drive.
Move three - stream on demand. For each token, the dense part in RAM computes, the router says "for this token I need experts 7, 40, 131..." and Colibri reads only those chunks from disk, runs them, and discards them. A smart cache (LRU cache - keeps what was recently used) holds the popular experts warm in RAM. Think of it like a library: you don't buy the whole building to read one book - you pull out exactly the page you need, when you need it.
And all of this, mind you, in a single C file of about 2,400 lines, with zero dependencies. No Python at runtime, no external math libraries (BLAS), no CUDA, no GPU. There are also some beautiful engineering tricks in there: a compressed attention mechanism (MLA) that keeps 576 numbers per token instead of 32,768, and a speculative decoding mechanism (MTP) that produces 2.2 to 2.8 tokens per model pass instead of one.
Let's follow a single token end to end, so this really sinks in. The prompt comes in. The dense part in RAM (9.9 gigs) computes attention. The router picks about 8 experts in each of 75 layers - roughly 600 experts per token. Colibri reads them from the SSD, about 11 gigs of disk reads, and produces exactly the same token a rack of H100s would have produced. Then it repeats the whole thing for the next token. The exact same answer - a different address: a laptop instead of 5 H100 cards.
But How Slow Is It, Really?
Now for the honest part, with numbers. Cold, on a plain dev box, it runs at about 0.05 to 0.1 tokens per second. On real machines with more memory, it climbs: an M5 Max MacBook with 128 gigs hits 1.06 tokens per second, and Ryzen and Framework desktops range between 0.28 and 0.40. Let's do the math honestly: at 0.5 tokens per second, a 500-word answer takes about 17 minutes.
So the critics aren't wrong on the facts. "Half a token per second" is real, and no, you're not going to have a chat session with this thing. Someone told me it's like "driving a Ferrari with no gas because you can push it." I think that analogy misses the point: the Ferrari does drive. It's just driving in first gear - on a road that until yesterday was a chasm.
And notice the shape of this graph: the more RAM you have, the more experts stay cached, and the faster it runs. The bottleneck is the disk, not the model. And that's exactly the point: the barrier shifted from "capability" to "patience." In technology, "impossible" is a wall. "Slow" is just a sticky note on the whiteboard.
What This Means for Tomorrow
And hardware is heading exactly in this direction. NVIDIA launched the DGX Spark and AMD launched Ryzen AI Halo, both with 128 gigs of unified memory around the $4,000 mark - precisely the variable the graph above is most sensitive to. AMD has already hinted at a 192-gig version for 300-billion-plus models. The trick Colibri proved on a 25-gig machine is going to get a lot better on the machines coming out this year.
And we should honestly say where GLM-5.2 stands. It's a genuine frontier model: on the FrontierSWE coding benchmark, less than one point (0.7) separates it from the king, Claude Opus 4.8. On SWE-bench Pro it scores 62.1 versus Opus's 69.2 - a gap of about 7 points. It doesn't beat Opus, but it's breathing down its neck, and it's open-weight.
My Take
In my view, Colibri isn't a replacement for Claude, it's not a model for everyday chatting, and today it really is slow - the critics are right about the present. But they're measuring the wrong axis. The wow isn't that it's fast. The wow is that it's possible at all - that a frontier-level model, separated from the king by just 0.7 points, runs on a machine that costs, as the developer put it, "less than a single H100's cooling fan."
And that it was all done in 2,400 lines of C, essentially by one person, who did what everyone assumed required a whole data-center framework. This combination - open weights plus a laptop plus a single C file - is exactly how capabilities stop being the exclusive property of whoever holds the biggest cluster.
Bottom line: today it's "slow but possible," and tomorrow the hardware will already be breathing 128 gigs. So here's the question I'm left sitting with: how much time is really left before a full frontier model runs at our own homes, completely private, without asking anyone's permission? And you - would you wait 15 minutes for an answer, if it's running on your own machine and never leaves it for a second?