LearnGenerative AI
Tokens
See how a language model chops text into tokens, and watch a tokenizer being built one merge rule at a time.
Type in the panel and slide the merge rules to watch characters fuse into tokens. Click a token to see the rules that built it. This tokenizer learned from English text.
How it works
Text in, numbers out.
A language model never reads letters or words. Text is first cut into tokens, and each token becomes a number, its ID. This tokenizer works like the one in GPT-2: text is split into word-like pieces (a space stays with the word after it), and every piece starts as raw bytes, IDs 0 to 255. Plain English letters take one byte each; many other characters take two to four.
Byte pair encoding learns merge rules from training text. It counts every pair of neighbouring tokens, joins the most frequent pair into a new token, and repeats. Here it learned 314 rules from a few paragraphs of English, and “ the” became a single token after only four of them. Each rule adds one token to the vocabulary.
Common words become single tokens, while rare words, names and other scripts fall back to smaller pieces, so they cost more tokens. Real tokenizers learn around 50,000 to 200,000 rules from huge amounts of text in many languages, and models are priced and limited by tokens, not words.