LearnDeep learning
Attention
See how a transformer decides which words to look at, with nothing more than dot products and a softmax.
Click a word to make it the query. Drag the pink query arrow, or any word's point, and watch the attention weights change. The example sentence is in English.
How it works
Look everywhere, weigh what matters.
In self-attention every word asks a question, its query, and every word offers a label, its key. The match between a query and a key is their dot product: large when they point the same way. Dividing by √d, the square root of the number of dimensions (here d = 2), keeps the scores in a sensible range.
A softmax turns the scores into weights that are all positive and add up to 1. The new meaning of the word is the weighted average of the words' values. Here each value is drawn at the same place as its key, so the output, the hollow dot, is a blend of those points.
Longer queries spread the scores further apart, so the softmax picks a winner more decisively. Real transformers learn their queries, keys and values, use hundreds of dimensions and many attention heads at once, and repeat this in every layer.