Yuval Avidani
Author
"Frozen parameters." If you've ever messed around with open models or with fine-tuning, you've definitely bumped into the term frozen parameters and had no idea what it means or why we should care. Today we're breaking it down from the ground up, and you'll walk away with an intuition that'll serve you every single time you see that phrase.
Let's start with the most basic building block, because without it, nothing else clicks.
First, what's a parameter
Let's start from the basics. A parameter, also called a weight, is a small number the model learned during training, and together there are billions of these that make up all its "knowledge." When a model "learns," what's actually happening is that these numbers get updated over and over until they produce good answers.
Think of it like millions of tiny volume knobs. At first they're all set randomly, and during training each knob gets turned a little, until the sound that comes out is right. Training a model is basically the process of tuning these knobs, one by one, a billion times over.
So what does "frozen" mean
Now we can define the central concept. A frozen parameter is a parameter we decide to leave exactly as it is, without updating it during training. In code this looks like one line: requires_grad = False, and that's literally telling the system "don't touch this knob."
According to the official PyTorch docs, the most common framework for building models, the term is precise: parameters that don't compute gradients are usually called frozen parameters. A gradient is basically the instruction "which direction to turn the knob to improve." When we freeze a parameter, we're simply giving up that instruction for it, and it stays constant.
Where you actually run into this
Here are the two places where this concept pops up, and why it's so useful.
The first is called transfer learning. Instead of training a model from scratch, you take a huge model that's already been trained, freeze its entire body, and train only one small layer at the end that adapts it to your task. Think of a world-class expert who already knows everything about images; we're not re-teaching him how to see, we're just giving him a short course on "how to tell a cat from a dog." All the existing knowledge stays frozen, and only the new part learns.
The second one, and this is the one most relevant to open models, is called LoRA. With LoRA you freeze the entire original model, and add small new matrices on the side that are the only thing that learns. Instead of training billions of parameters, you train just a tiny sliver of them. And here's a number that shows just how dramatic this is:
Look at that gap. According to Hugging Face's official docs, when you run LoRA on a billion-parameter model, you're training only about 0.04 percent of the parameters, almost always less than half a percent. The original LoRA paper from 2021 put it even more sharply: compared to full fine-tuning of GPT-3, the method cuts the number of trainable parameters by 10,000x and GPU memory by 3x.
So why freeze at all
Now that we understand what it is, the obvious question is: why not just train everything? Three practical reasons.
First, massive resource savings. If you're only training half a percent of the weights, you need way less memory and compute, and that's exactly what makes it possible to train huge models on a home computer or a single GPU. That's the exact magic behind all those "I trained a model at home" posts you see.
Second, preventing forgetting. When you retrain an entire model on just a handful of examples, it can "forget" what it already knew. Freezing the body keeps the original knowledge intact.
Third, speed and modularity. You can keep a ton of small "LoRA add-ons," one per task, and swap between them on top of the same frozen base model, without duplicating it every time.
One last confusing point: frozen during training vs. frozen at inference
There's an important distinction worth holding onto. All of a model's parameters are already "frozen" while it's answering you, because the model doesn't learn during a conversation, it just runs what it already has. The "freezing" we've been talking about is a choice we make during training, to leave some of the weights without updates. So when you see "frozen parameters" in the context of an open model, it's almost always this: someone took a base model, froze it, and trained only a thin layer on top of it.
Bottom line, as I see it
So let's sum it up in a format I like. A frozen parameter is a weight you keep constant during training, for anyone who wants to adapt an existing model without retraining it from scratch. This is the foundation of both transfer learning and LoRA, and it's what made the whole open-model world accessible to everyone.
In my view, this is one of the most beautiful ideas in practical AI: you don't need to be a giant to shape a giant model. You freeze what already works, and you only teach what's missing. The limitation, and it's fair to say this: if your task is very far from what the model originally learned, over-freezing can sometimes hurt quality, and then it's worth "unfreezing" a few more layers. It's a balance, not an iron rule.
So here's the question I'll leave you with: if you can shape an entire model by training just half a percent of it, what else are we still training from scratch, when there's already an excellent base out there to freeze and build on top of?