LearnGenerative AI
Next-word prediction
Write a sentence one word at a time with a tiny language model that learned only by counting.
Click a word in the list to add it. The model has read 45 short English sentences and only knows which words followed which.
How it works
Guess the next word, again and again.
A language model writes by predicting the next token, adding it, and predicting again. This one is as simple as it gets: it counted which words followed which in 45 short training sentences. The probability of a word is the number of times it followed the context, divided by the number of times the context appeared.
With one word of context, “the” is followed by 17 different words, so the model is unsure. Two words make the context more specific and the predictions sharper: after “a cup” it is always “of”. If a two-word context never appeared in training, this model falls back to the last word alone.
Counting soon breaks down: most longer contexts never appear in any training text, and the model cannot handle combinations it has not seen. Neural language models solve this by learning what words and contexts mean, and they predict from thousands of tokens of context instead of two words.