Yuval Avidani
Author
"Why does it sometimes take a whole second for the model to start answering, and then suddenly the text just flows fast?" If you've ever asked yourself that, you've run into two concepts I bump into constantly: TTFT and FLOPs. Today we'll break both of them down from the ground up, and you'll walk away with an understanding that changes the way you think about model speed.
Let's start with the most frustrating moment, that second we're staring at a blank screen, waiting.
What is TTFT: the second we're waiting
Let's start with the first concept. TTFT, short for Time To First Token, is the time from the moment we send the prompt until the first token of the answer shows up on screen. That's exactly the duration of "staring at a blank screen" before the text starts flowing.
According to NVIDIA's official definition, TTFT includes queue waiting time, prompt processing time, and network time. And it's the most important experience metric: how fast the system feels responsive. Nobody likes waiting in front of a blinking cursor.
Why this happens: the Prefill stage
Here's the heart of the matter, and this is what most people don't know. Before the model can spit out even one token, it has to read and process our entire prompt in one shot. This stage is called prefill, and it's what determines the TTFT. The model builds a kind of "working memory" (called KV cache) from every word in the input, and only then is it ready to generate the first token.
And here's the rule worth remembering, which NVIDIA emphasizes too: the longer the prompt, the bigger the TTFT. A 4000-word prompt will take much longer to start answering than a 20-word prompt, because there's more to process before that first token.
This is also the difference from the second stage, decode, where the model generates the rest of the tokens one by one. Prefill is compute-bound, choked by raw processing power, while decode is memory-bandwidth-bound, which is why after the first token the text flows fast. That explains the feeling: a wait at the start, then a flood.
Now FLOPs: the unit that measures compute power
To understand what determines prefill speed, we need the second concept. A FLOP is a single computational operation on numbers (one addition or multiplication). A tera-FLOP is a trillion such operations, and a peta-FLOP is a thousand trillion.
And here's a point that confuses a lot of people: when you're measuring a GPU, you talk about FLOPS with an S at the end, meaning operations per second. That's a rate, not a quantity. A stronger card does more operations per second, so it finishes prefill faster.
Here are the official numbers from NVIDIA's datasheets, and the differences are huge:
Look at that jump. An A100 does 312 tera-FLOPs per second, the H100 jumps to 989, and the new B200 hits 2,250, meaning 2.25 peta-FLOPs. Fun fact: the H200 is identical to the H100 in compute power, because it's literally the same chip, its upgrade is only in memory. And it's exactly these nuances that determine how fast we get that first token.
The bridge: how FLOPs determine TTFT
Now let's connect the two concepts, and this is the beautiful part. Every token the model processes costs roughly twice its number of parameters in compute operations (the 2N rule). A 70-billion-parameter model burns about 140 billion operations per token.
So if prefill needs to process the whole prompt, the math is simple: TTFT roughly equals the number of input tokens times twice the parameters, divided by the card's speed. That's why, as we saw, doubling the prompt length roughly doubles the wait time. Here's what that looks like for a 70B model on a single H100:
Let's do one numerical example, because it makes everything click. A 1000-token prompt for a 70B model requires 140 tera-FLOPs of processing. On an H100 doing about 989 tera-FLOPs per second (at a realistic utilization of around 50 percent), that comes out to roughly 280 milliseconds just to start answering. And that's why the first token is the "expensive" one, while all the rest just flow.
And what that looks like in the real world
For a sense of proportion, here are reported TTFT numbers from fast APIs in 2026: fast models like Mistral Large deliver a first token in about 300 milliseconds, GPT-5.2 in about 550, and heavy reasoning models can take several whole seconds.
And here's a number that shapes product design: the accepted target for interactive chat is about 300 milliseconds to first token, the point where we stop feeling any delay. For a voice assistant the target is stricter, around 150 milliseconds, and for code completion while typing, even under 100. Above three seconds, the experience is already broken.
How to measure TTFT ourselves
So this doesn't stay abstract, here's how you measure TTFT in code: just count the time from the moment you send the request until the first chunk of the streamed response comes back:
Bottom line, as I see it
So let's sum this up in a format I like. TTFT is the time until the first token, and it's determined at the prefill stage where the model processes the entire prompt; FLOPs are the compute unit that determines how fast the card gets that done. A longer prompt or a weaker card means a longer wait.
As I see it, once you understand this, you stop being frustrated by "the model is slow" and start thinking correctly: if TTFT matters to us, we shorten the prompt, use caching, or pick a suitable model and card. The caveat, and it's fair to say this: the exact TTFT numbers depend on a ton of factors, card utilization, batching, network, and load conditions, so the numbers here are an engineering approximation and reported benchmarks, not a promise. The card specs themselves, on the other hand, are official, straight from NVIDIA's datasheets.
So here's the question I'll leave us with: now that we know every token in the prompt adds to the wait time, how many of the words we send the model are actually necessary, and how many just extend our staring at a blank screen?