Yuval Avidani
Author
The moment I realized I don't need the cloud
I sat down at my Mac, no internet connection at all, and asked an AI model to write me some code — and it answered. No network connection, no paying for a single token, not one line of my text leaving this computer. Turns out today you can run a serious language model right inside a home computer, and it's way simpler than it sounds.
To me, this is one of the most liberating things that's happened to us in recent years. Most people are used to AI being something that happens "at someone else's place": we send a message to a remote server, it thinks, and sends back an answer. But we don't have to work that way. In this article I'm going to break down exactly how you run a model locally, what the difference is between the three main tools, and how to actually get started in a few minutes.
Why even bother running locally
I'll break this down into four reasons, because each one on its own already justifies giving it a shot.
The first is privacy. When we work with a cloud service, every word we type travels to some other company's server. When the model runs locally, our data doesn't leave the computer, period. For me, when I'm working on something sensitive — internal code, drafts, personal documents — that's the difference between peace of mind and constant worry.
The second is cost. Cloud services price by tokens, and a token is a small piece of text — roughly a syllable or part of a word — that the model counts one by one. Every time we send a question and get an answer, they're counting these little pieces and charging us for them. When the model's on our own machine, there's no meter running. We can run a thousand questions a day and not pay a penny beyond electricity.
The third is offline capability. No internet, on a flight, on a train, in a remote village — the model still works. The fourth is control. We choose exactly which model, which version, and when to update it. Nobody changes its behavior out from under us.
The small catch: your graphics card's memory
Now for the less sexy part. A language model is basically a massive pile of numbers called "weights" — these are the values the model learned during training, kind of its frozen memory. The problem is there are a ton of them, and they need to fit into fast memory for the model to run at a reasonable speed.
That memory is called VRAM, and it's simply the memory on your graphics card (the GPU). Think of it like a desk: the bigger it is, the bigger a model you can spread out on it without things falling on the floor. On M-series Macs, memory is shared between the CPU and GPU, so the number that matters to us is total RAM.
And here's where the most important trick in this whole story comes in: quantization. This is a technique that compresses the weights into a more compact format — say, Q4 or Q8 — so the model fits into modest memory. Think of it like saving an image as a JPEG instead of a raw format: it's a bit less sharp, but way smaller and opens up everywhere. Q4 is small and fast and loses a touch of quality, while Q8 is bigger and closer to the original. In my view, for most everyday use, Q4 is exactly the sweet spot.
These models usually come in one common file format called GGUF. It's simply the standard format for packaging a local model so that the tools running it know how to read it. When you see a filename ending in .gguf, you know it's a model ready to run on your machine.
The three tools: espresso, a coffee machine, and fresh beans
There are three main tools, and it's easy to mix them up because they all touch on local AI. But they solve completely different problems. Let me give you the analogy that helps me the most: it's like the world of coffee.
The first tool, Ollama, is the espresso pod: one push and you've got coffee. It's a command-line tool (CLI) and server that runs a model with a single command. I type ollama run llama3, and it downloads the model, loads it, and opens up a chat for me. Official definition: Ollama is a local runtime tool that downloads, manages, and runs language models with a single command, for anyone who wants the shortest path from "nothing" to "a working model". The cool part is it also exposes an OpenAI-compatible API, meaning any code you've already written against OpenAI will work with it almost without changes. To me, that makes it the perfect engine for automations.
The second tool, LM Studio, is the home coffee machine with all the buttons — a desktop app with a full graphical interface (GUI). You search for a model on a list, click "download," and start chatting away in a comfortable window. No terminal, no commands. It's the friendliest route for anyone who doesn't love a black screen full of text. It can also spin up a local server if you want, but its real strength is visual experimentation: comparing models, playing with settings, seeing everything with your own eyes.
The third tool, Unsloth, is something else entirely. It's not a coffee machine — it's the ability to roast the beans yourself. The Unsloth library is built for fine-tuning, customizing an existing model to your own data, fast and memory-efficient. Unsloth doesn't run models for everyday use — it retrains them so they fit your exact task, using less VRAM and at higher speed than the standard way. If Ollama and LM Studio are "drinking coffee," then Unsloth is "creating your own blend."
So when do you use what
The distinction is simple once you get it. If you want speed and simplicity, or you want to hook a model into an automation or your code — Ollama. If you're just starting out, love a graphical interface, and want to experiment and compare models comfortably — LM Studio. And if you want a model to talk in exactly your style or deeply know a specific domain — that's where Unsloth comes into the picture with fine-tuning.
I want to be fair about this: these aren't competitors where one "wins." They're different approaches to different problems. Personally, I run all three on my machine: Ollama for everyday work and scripts, LM Studio when I want to eyeball a new model, and Unsloth when I'm training something purpose-built. They live together just fine.
Getting started for real: five minutes with Ollama
The fastest way to get a taste of this is Ollama, so let's walk through it step by step. I'd recommend starting with a relatively small model (a few gigs in size) so it comfortably fits into the memory of an average home computer.
One point from the field, from me personally: on my M-series Mac, a small model at Q4 quantization runs completely smoothly and still leaves me room to work. My rule of thumb: if the model is roughly half your free memory or less, you're safe. If you try to load a huge model on a modest machine, either it'll crawl, or it'll simply refuse to load at all. Start small, get a feel for the pace, and scale up gradually.
Bottom line
To me, the moment you run your first model locally is a moment your whole perspective shifts. Suddenly you realize AI isn't just a service you buy — it's also a tool you hold in your hand, one that respects your privacy and doesn't send you a bill at the end of the month.
But let's stay honest about the limitations: a model running on a home computer won't match the raw power of the giant cloud models. A small local model knows less, makes a few more mistakes on complex tasks, and sometimes feels "less smart" than the big services. And that's totally fine, because for most of what we do day to day, a good local model is more than enough, and the control and privacy are worth the gap. This isn't a full replacement for the cloud — it's a powerful addition to your toolbox.
So here's the question I'll leave you with: if you could run a full AI without a single bit leaving your computer, what's the first thing you'd ask it — that today, you wouldn't dare type into the cloud?