Yuval Avidani
Author
"Why does the model remember my last message perfectly but forget what I asked at the very beginning?" I hear this question more than any other question about language models, and the answer always leads to the same place: the context window. Turns out once you understand this concept, suddenly a ton of the model's weird behavior makes perfect sense.
In my view this is one of those concepts everyone working with AI has to have in their head, not because it's complicated, but because it explains the boundaries. Let's break it down slowly.
So what is a "context window" anyway
Let's start with a clean definition. The context window is the amount of text the model "sees" at once, holding everything needed to generate a response, for anyone using the model.
And "all the text" really means all of it: the prompt we wrote (meaning the instruction we sent), the entire conversation history so far, and even the response the model is generating right this second. All of it gets squeezed into that same one window.
Think of it like a desk. Big desk = we see more documents spread out at once and can cross-reference between them. Small desk = only a few pages fit, and the rest has to go on the floor. This window is exactly the model's desk, and whatever doesn't fit on it simply doesn't exist for it at that moment.
Wait, what's a token? Because that's what the window is measured in
Before we go on, we need to understand one unit of measurement: the token. Because window size isn't measured in words or characters, it's measured in tokens.
The model doesn't see letters the way we do. A token is a piece of a word: sometimes a whole word, sometimes half a word, sometimes just a punctuation mark, and this is the basic unit the model actually reads. The word "hello" might be one token, but a long or rare word might break into two or three tokens. In Hebrew, by the way, words break into more tokens than in English, and that's one reason Hebrew text "eats up" more of the window.
Here's a real engine. Play with it: type a sentence, and watch with your own eyes how it breaks into tokens. This is exactly the angle from which the model sees the world.
The cool part is that once you get this, you also understand why our word count doesn't always match what the model "feels." It's not counting words, it's counting tokens.
Why is there even a limit? This is where the Transformer comes in
Now the real question: why not just make the window infinite? Why is there a ceiling at all?
To answer that we need two concepts. The first is the Transformer, the architecture (meaning the engineering structure) that almost all modern language models are built on. Think of it like a building's floor plan: it determines how information flows inside.
The second concept is the heart of the Transformer: the attention mechanism. Attention is the mechanism where every token "looks" at every other token in the window to decide what's relevant to it. This is basically the model's ability to weigh context, to understand that the word "he" at the end of a sentence refers to a character mentioned at the beginning.
And here's where the problem lies. If every token needs to look at every other token, then when you double the window's length, the number of cross-comparisons doesn't double, it quadruples. This is what's called quadratic growth. When you double the context length, the memory and compute cost grows roughly 4x, and that's the physical reason there's a ceiling.
Think of it like a round table at a party. With 4 people there's a handful of possible conversations. With 8 people the number of conversations jumps way beyond double. Add a few more and it's already uncontrollable chaos. Attention is exactly that: every token holds a "conversation" with every other token, which is why doubling gets insanely expensive.
Let's pause for a second to make sure this landed, because this is the single most important point in the article.
What happens when you exceed the window
Say the conversation is long, and we've passed the ceiling. What happens then? The model doesn't crash and doesn't return a red error, it does something quieter and sneakier: it cuts.
The technical term is truncation. Truncation means cutting: when the conversation exceeds the window, the oldest parts (usually the beginning) simply get chopped off to make room for the new. It's like a full desk: to put down a new page, we push an old page off the edge and it falls to the floor.
And that's exactly the answer to the question I opened with. When a long chat "loses the thread" and suddenly forgets an instruction we gave at the start, most of the time no bug happened at all. The beginning simply got cut out of the window. The model didn't "forget" in the human sense, it just no longer sees that part on the desk.
To me this is one of the most liberating insights in working with AI: most of the mysterious "forgetting" isn't a glitch, it's space management.
Modern models: giant desks (that also come with a price)
The good news: windows have grown dramatically in recent years. If we used to talk about a few thousand tokens, today modern models hold windows of hundreds of thousands of tokens, and some already reach the order of a million tokens. That's enough to fit an entire book, or a large codebase, in at once.
But, and here I'm being careful, a giant desk isn't always free. A large window costs more in time and money, and sometimes critical information buried in the middle of a giant window gets less "attention" than information sitting at the edges. It's not like you push in a million tokens and get magic. Bigger isn't always better, it's simply a different approach with different tradeoffs.
So how do you work with this correctly? Three tricks
Now for the practical part, because understanding without applying it is a waste. Here are three things I do all the time.
First: summarize. When a conversation drags on, I ask the model to summarize what we've covered so far, open a new chat, and paste the summary at the top. That way I carry the essence with me without dragging the whole heavy history along.
Second: start a new chat when you switch topics. Every old message "sitting" on the desk costs us in space and money, and also confuses the model. Clean desk = focused model.
Third, and this is the strong one: RAG. RAG (Retrieval-Augmented Generation) is a method where instead of pushing all the information into the window, you pull exactly the relevant chunks from an external database and feed in only those. Think of it like an assistant who runs to the archive, brings back only the three correct pages, and lays them on the desk, instead of piling the entire archive on top of it. That way the window stays small and focused, and the answers are more accurate.
Bottom line
The context window isn't an arbitrary limitation someone invented to annoy us, it's a direct result of how the Transformer and attention are built, and of the fact that cost grows roughly quadratically. Once you understand that, you stop fighting the model and start working with it: summarize, open clean chats, and use RAG when needed.
My caveat for this piece: I intentionally spoke in general numbers, because window sizes move fast and every model is different. The idea is stable, the numbers move.
So next time a model "forgets" something in the middle of a long conversation, what's your guess: did the window get truncated, or was there actually a bug here?