Scaled dot-product attention

Scaled dot-product attention

AlgoLab runs no language model. The vectors here are four numbers per token, typed in by hand so that every dot product on this page can be checked with a pencil, and you can read all of them in the Vectors tab. A real model’s queries, keys and values have hundreds or thousands of dimensions, learned from data rather than chosen by a person.

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

Scaled dot-product attention takes three tables of numbers — a query, a key and a value vector for every token — and produces one output vector per token. It scores every query against every key with a dot product, divides every score by the square root of the vector length, turns each row of scores into weights that add up to one, and builds each output row by adding up the value vectors in exactly those proportions. The whole of it is one expression: Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V.

Why it matters, and what it costs

It is the operation a transformer layer is built from, which is the honest reason to understand it: it is the step that mixes information between positions, and every other part of the layer acts on one position at a time. Its cost is the thing to notice. Scoring every query against every key is n² dot products for n tokens — eight tokens here means 64 scores, and the tool counts them as it goes — so the work grows with the square of the sequence length, not with it. That is why a long context is expensive, and it is the whole reason techniques that avoid recomputing keys and values exist.

How it works

• Start with three tables, each with one row per token: the queries Q, the keys K and the values V. On this page all three were typed in by hand.
• Score every query against every key. The score at row i, column j is the dot product of query i with key j: multiply the two vectors element by element and add up the results.
• Divide every score by √dₖ, the square root of the vector length. Here dₖ is 4 and √4 is exactly 2, so every score is halved and every result is a whole number or a half.
• If a position may not be used, replace its score with minus infinity before going any further. Under causal masking, row i replaces every column after i.
• Turn each row of scores into weights with softmax: raise e to the power of each score and divide by the total for that row. Every row then adds up to one, and any score of minus infinity becomes a weight of exactly zero.
• Build output row i by adding up every value vector, each multiplied by its weight from row i. Because the weights are non-negative and add up to one, each output row is a weighted average of the value vectors.

Worth noticing

• Each row of weights adds up to one, and that is what makes an output row a weighted average of the value vectors rather than an arbitrary combination of them. It holds to within one unit in the last place of double-precision arithmetic, not exactly: on this page eight of the sixteen rows across the two examples come out 1 ± 2.22e-16, and a test asserts the tolerance rather than claiming the rows are exactly one.
• A masked weight is exactly zero, not merely small. Softmax raises e to the power of each score, and e to the power of minus infinity is exactly zero in the arithmetic a computer actually uses — so those terms add exactly nothing to the output. Three tests check that: the weights are zero by exact comparison, the output is identical bit for bit whether the masked terms are included or dropped, and changing a masked position’s value vector to an absurd number leaves every output unchanged.
• The division by √dₖ flattens each row, and the tool shows both versions so the effect is a number rather than a claim. Row 2 of the first example has a largest weight of 0.9465 without the division and 0.7662 with it; row 6 goes from 0.4748 to 0.2945. Dividing a row of scores by anything greater than one cannot increase its largest weight, and strictly decreases it unless every score in the row is equal.
• The reason the constant is √dₖ and not something else is about larger vectors than these. For vectors whose entries are independent and average zero, the spread of a dot product grows as √d — measured over generated vectors, the standard deviation is 0.499 × √d from d = 4 to d = 4096 — and without the division the softmax of those scores collapses onto one position as d grows, from a largest weight of 0.33 at d = 4 to 0.97 at d = 4096. Divided by √d it stays near 0.23 throughout. The hand-written vectors on this page do not demonstrate that, because their entries are not zero-mean; the measurement is over generated vectors and the tests say so.
• The two copies of a repeated token have identical rows here, so they score identically against every query — row 4 of the first example scores 2 against both column 0 and column 4. That is a property of this hand-written data, not of attention. Real implementations add positional information before the projections that produce Q, K and V, so two identical tokens would not have identical rows.
• A score of exactly zero is a real score and not a blank. Row 2, column 0 of the first example is 0, and the picture draws it filled with a drawn zero rather than as an empty cell, because those are different facts and the renderer carries the difference in three channels.
• This is one attention head, computed once. It is not a whole transformer: there is no multi-head split, no stack of layers, no projection producing Q, K and V from token embeddings, no residual connection and no layer normalisation. Those are named here so that nothing on this page is mistaken for the rest of the model.

Where the numbers come from

Every number this page computes with was typed in by a person. They are held attools/attention/src/data/examples.v1.json and shown in full in the tool’s Vectors tab: two eight-token examples, with a query, a key and a value vector of four numbers for each token. Entries are drawn from 0, 1 and 2, and that is a decision rather than laziness — it makes every dot product a small whole number, and since d_k is 4 and √4 is exactly 2, every scaled score is a whole number or a half. The whole page can be checked with a pencil, which is the only reason it can promise its arithmetic is verifiable.

Nothing is derived from the words. The four dimensions are numbered, not named, and none of them means anything on its own; naming them would claim an interpretability this page cannot support. Exactly one structural constraint was imposed: a repeated token gets identical rows. So the two copies of the in the first example score identically against every query, which is a property of this hand-written data rather than of attention. Real implementations add positional information before the projections that produce Q, K and V, so two identical tokens would not have identical rows there.

What dividing by √d_k does

This is the step the roadmap calls the one people least understand, so the tool gives it eight frames of its own and shows both versions rather than performing the division silently. On the first example, row 2’s largest weight comes to 0.7662 with the division and would have been 0.9465 without it; row 6 goes from 0.4748 to0.2945. Dividing a row of scores by anything greater than one cannot increase its largest weight, and strictly decreases it unless every score in the row is equal.

The reason the constant is √d_k and not something else is about vectors much larger than these. For vectors whose entries are independent and average zero, the spread of a dot product grows as √d — measured over generated vectors, the standard deviation comes out at 0.499 × √d fromd = 4 to d = 4096 — and undivided, the softmax of those scores collapses onto one position as the dimension grows:

Those numbers are measured over generated vectors, not over the hand-written ones on this page. The authored entries are not zero-mean, so their dot products do not grow that way, and the tool does not claim its own eight-token example demonstrates the rule. Both measurements are in tools/attention/test/attention.test.ts.

What this does not show

This is one attention head, computed once. It is not a transformer. There is no multi-head split, no stack of layers, no projection turning token embeddings into Q, K and V, no residual connection and no layer normalisation. The roadmap refuses a full forward pass on the grounds that it is a diagram rather than a trace, and cuts layer normalisation outright as a formula rather than a process. Those boundaries are named here so that nothing on this page is mistaken for the rest of a model.

Cost

Time

O(n² · dₖ)

Extra space

O(n²)

Steps recorded

100 masked, 92 unmasked

n is the number of tokens and d_k the length of each vector. Then² term is the thing worth noticing: scoring every query against every key is 64 dot products for eight tokens, and the work grows with the square of the sequence length rather than with it.

Unlike every other tool here, this one has no ceiling and no input to refuse. There is no number a reader can raise: n and d_k are fixed by the data file, so the step count is a constant and a refusal path would guard something that cannot vary. What is validated instead is the data file itself, at load, because that is the part a person can edit.

Source

Vaswani et al., “Attention Is All You Need” (2017), §3.2.1 for the formula and its name, §3.2.2 for the dimensions a real model uses, §3.5 for positional information.

Worth comparing

• Byte-pair encoding— what a token is, decided before anything on this page happens. Neither runs a model.
• Beam search— how a sequence is chosen once scores exist. This page produces vectors; that one searches over them.
• Edit distance— the other tool here that fills a grid cell by cell, and the one that shares this picture. Its cells depend on their neighbours; every one of these 64 dot products is independent, which is what makes attention parallel.

All algorithms