Yuval Avidani
Author
Turns out the sentence "I trained an LLM myself in Python" has stopped being a fantasy reserved for researchers with server farms, and turned into something you can actually get your hands on. This trend is real and accessible, but there's one step everyone skips, and it's exactly the step that trips most people up: the tokenizer.
Let's break this down from the start, without throwing code at you before we understand why.
What "training a model" at home actually means
When we say train an LLM, we mean a language model: software that predicts the next word (or more precisely, the next token). Andrej Karpathy, one of the best teachers in the field, built a repo called nanoGPT which he described as "the simplest, fastest repository for training/finetuning medium-sized GPTs". The magic there is in the simplicity: the training file train.py is a training loop of about 300 lines, and the model file model.py is a GPT definition of about 300 lines. That's it. Everything that sounds like black magic condenses into six hundred lines you can read in one sitting.
All the "magic" of training a GPT condenses into roughly six hundred lines of code in nanoGPT: 300 for the training loop, 300 for the model itself.
The beginner path in nanoGPT trains a character-level GPT (meaning every letter is a unit) on about one megabyte of Shakespeare text. The settings there are tiny: a context window of 256 characters, 384 feature channels, six transformer layers, and six attention heads per layer. Small enough to run on your machine, and real enough to understand what's actually happening.
And if you want to play in the big leagues: nanoGPT reproduces GPT-2 at the 124-million-parameter scale, training on a dataset called OpenWebText, using GPT-2's tokens. This runs on a node of eight A100 cards with 40GB memory, about four days, reaching a validation loss (an error metric on data the model hasn't seen) of about 2.85. Notice the gap: one megabyte of Shakespeare on your computer versus four days on eight top-tier cards. Same philosophy, completely different scale.
The step everyone skips: the tokenizer
And here comes the plot twist. Before the model even sees text, someone has to turn words into numbers. This part is called tokenization, and the most common mistake is thinking it's part of the model. It's not. Karpathy dedicates a whole lecture to it, "Let's build the GPT Tokenizer", two hours and thirteen minutes long, where he frames the tokenizer as a completely separate stage in the pipeline, with its own training set and its own training algorithm.
The tokenizer is a completely separate stage from the model: it has its own training set, its own algorithm, and if it's bad, no model can save you.
The image I like: the tokenizer is like the dictionary and grammar the teacher builds before the student starts reading. Two core functions: encode, which converts a text string into a list of tokens, and decode, which converts tokens back into text. If the dictionary is built poorly, it doesn't matter how smart the student is.
The algorithm that trains this dictionary is called BPE, short for Byte Pair Encoding. Let's break it down: you start from the basic characters or bytes of the text as the initial tokens. Then, over and over, you identify the most frequent adjacent pair of tokens and merge it into one new token. You stop when the vocabulary reaches the desired size. This works on UTF-8 encoded strings, and the idea comes from a 2015 paper by Sennrich and colleagues, which became widely known thanks to the GPT-2 paper.
Building one by hand with minbpe
The most beautiful part: Karpathy built this code live on stream during the lecture, and it sits in a separate repo called minbpe. His own description: "Minimal, clean code for the (byte-level) BPE algorithm commonly used in LLM tokenization." The repo is MIT licensed, meaning you're free to use it.
Inside minbpe there are three tokenizers, and each one teaches you an idea:
The first, BasicTokenizer, runs BPE directly on the text. Simple, educational. The second, RegexTokenizer, first splits the text by letters, numbers, and punctuation marks (exactly like GPT-2), to prevent merges that cross boundaries. The third, GPT4Tokenizer, reproduces GPT-4's tokenization. All three support train, encode, decode, and save/load.
Before we dive into code, let's feel this ourselves. Type some text here, in Hebrew or English, and watch how it breaks down into tokens in real time. This is exactly what minbpe's BPE algorithm builds behind the scenes.
Why tiktoken is written in Rust, and why GPT-4o has 200 thousand tokens
Now to the production arena. When you're running a tokenizer on billions of words, speed isn't a luxury. That's why OpenAI wrote tiktoken, a fast BPE tokenizer, in a Rust core with a thin Python layer on top. According to the official repo, it's reported to be three to six times faster than a comparable open-source tokenizer. The image: the exact same idea as minbpe, but with a race engine instead of a learning engine.
OpenAI's tiktoken is written in a Rust core and reported to be 3 to 6 times faster than a comparable open-source tokenizer, because at production speed this is critical.
And here's a number that explains a lot: tiktoken's tokenizer has several "encodings." The cl100k_base encoding, with a vocabulary of about 100,256 tokens, is used by GPT-4 and GPT-3.5-Turbo. The newer encoding, o200k_base, with a vocabulary of about 200 thousand tokens, is used by GPT-4o. Why did they double the vocabulary? Because a larger vocabulary improves efficiency on multilingual text. In my eyes, this touches us directly: the bigger and more diverse the vocabulary, the more Hebrew gets compressed into fewer tokens, and that saves money and speeds up responses.
And if you want to train a tokenizer yourself without writing BPE from scratch, HuggingFace's tokenizers library does it: you pair Tokenizer(BPE()) with BpeTrainer, define special tokens like UNK and PAD, set a pre-tokenizer like Whitespace, run train on text files, and save to a JSON file. Their documentation even includes recipes to reproduce the tokenizers of GPT-2, BERT, and T5.
The headline that changed in 2025: nanochat
And here's the part it's important you know, because a lot of guides online still haven't been updated. As of November 2025, nanoGPT's README carries a prominent notice that the repo is "now quite outdated," directing readers to its successor, nanochat.
What is nanochat? Karpathy released it on October 13, 2025. It's open source in PyTorch, about 8,000 lines, implementing a full ChatGPT-style pipeline: from training the tokenizer all the way to a full web interface. And here are the numbers that blow my mind: you can train it in about 3 hours (the fast run reaching GPT-2 level in about 2 hours and about $48), well under $100, on a node of eight H100 cards (which costs about $24 an hour).
nanochat, nanoGPT's successor from October 2025, trains a full ChatGPT-style chat model, from tokenizer to web UI, in about 3 hours and well under $100.
Bottom line
In my eyes, the real story here isn't "anyone can train ChatGPT at home." It does require a node of eight H100 cards and about $100, and it's still a tiny little ChatGPT, not GPT-4o. The real story is that the pipeline has been thrown wide open: for the first time you can read every stage, from tokenizer to chat, in readable, clean code, under an open license.
And the limitation worth remembering: understanding the code isn't the same as having a useful model. Character-level Shakespeare on your computer will spit out something that sounds like Shakespeare, not a personal assistant. And all the numbers here, four days, 2.85, $100, come from Karpathy's and OpenAI's official repos and sources, not from me.
So if you had to choose: would you start by building a tokenizer by hand with minbpe to understand what's happening under the hood, or jump straight to nanochat and train a full chat model in the same afternoon?