Softmax, temperature, top-k and top-p
AlgoLab runs no language model. These fifty tokens and their logits were written by hand, following the rule named in the Preset tab — nothing here was computed by a model. A real model chooses from a vocabulary of tens of thousands of tokens, and every logit comes from a forward pass through billions of learned weights, not from a rule a person wrote down.
softmaxᵢ(x) = exp(xᵢ) / Σⱼ exp(xⱼ)
Three tools a language model uses to turn a row of scores into a choice. Softmax turns fifty authored logits into probabilities that add up to one. Temperature divides every logit by a constant before softmax, which changes how concentrated or spread out the resulting probabilities are. Top-k keeps only the k highest-probability tokens and rescales their probabilities to add up to one again; top-p keeps the smallest group of highest-probability tokens whose probabilities already add up to at least p, then does the same rescaling.
Why it matters, and what it costs
This is the last step of choosing a token at generation time, after a model has already produced one score per vocabulary entry. Plain softmax at temperature 1 is the baseline. Lowering the temperature makes the probabilities more concentrated on the already-largest scores; raising it spreads them out — both are measured on this page, not asserted. Top-k and top-p exist because sampling from the full, barely-above-zero tail of a large vocabulary occasionally draws a token that fits poorly; both cut that tail off, in different ways, before a token is drawn.
How it works
- Start with one logit per token, authored by a person for one of three preset distributions.
- Divide every logit by the temperature, T. Dividing by 1 changes nothing; above 1 it shrinks the gaps between logits, below 1 it widens them.
- Apply softmax: raise e to the power of each divided logit and divide by the total, so the fifty results add up to one.
- For top-k: rank the tokens by probability, keep the k highest, set every other probability to zero, then divide the kept probabilities by their own total so they add up to one again.
- For top-p: rank the tokens by probability and add them up from the highest until the running total reaches p. Keep that prefix, zero the rest, and renormalise the same way.
- Top-k and top-p are computed independently from the same temperature-scaled distribution, not chained: this page shows what each one does on its own, not a specific production pipeline that applies both in sequence.
Worth noticing
- Every probability distribution this page produces sums to one to within floating-point tolerance, never exactly in every case: a test checks the actual computed sum at every temperature this tool offers, the same standard `docs/ROADMAP.md` §4 states for this acceptance criterion.
- Temperature’s direction is a property of the arithmetic, not of these three distributions: dividing every logit by a number greater than one cannot raise the largest probability, and dividing by a number less than one cannot lower it, whatever the logits are. A test checks this over 50,000 random rows. What is specific to the data here is the size of the effect — the one clear leader example moves far more than the near-uniform one across the same temperatures, because its logits started further apart.
- Top-k can split an exact tie. In the third preset, two tokens are tied at the same logit and so the same probability; keeping only the single highest-probability token keeps one of them and drops the other, even though nothing about the numbers distinguishes them. The one kept is whichever is listed first in the vocabulary, which is a fact about how ties are broken here, not about which token is better.
- Top-p can keep a tie together where top-k would split it. On that same preset, reaching a cumulative probability of 0.5 needs both tied tokens — one alone falls just short at 45.42% — so top-p at p = 0.5 keeps both while top-1 keeps only one.
- Renormalising is a real division, not a relabelling: a token kept by a filter ends up with a larger probability than it had before the filter ran, because the same numerator is now divided by a smaller total.
- This page does not draw a token. It shows the probabilities a model would draw from, at every stage, as numbers; the actual draw is a single weighted random choice over whichever distribution a reader is looking at, and nothing about that step depends on anything specific to language models.
Where the numbers come from
Every logit on this page was written by a person, following a simple rule stated for each of the three presets. They are held attools/sampling/src/data/vocabulary.v1.json and shown in full in the tool’s Vocabulary tab, alongside the rule that produced them. Nothing is derived from what the token names mean — the fifty names are an arbitrary label set, picked for readability, and swapping any one of them would not change a single number here.
What temperature does
Dividing every logit by a number greater than one cannot raise the largest probability, and dividing by a number less than one cannot lower it — a property of the arithmetic, checked over 50,000 randomly generated rows, true of any logits at all. On the preset with one clear leader, the largest probability moves from 86.47% at T = 0.5 to63.14% at T = 1 to 35.79% at T = 2. On the near-uniform preset, across the same three temperatures it moves only from4.31% to 2.95% to 2.43% — because temperature changes the *shape* of a distribution, and there is very little shape to change when the logits already sit close together. Both are measured intools/sampling/test/sampling.test.ts, not asserted.
What top-k and top-p do, and where they disagree
Top-k keeps a fixed number of the highest-probability tokens; top-p keeps however many are needed to reach a cumulative probability. On the preset with a tied pair at the top, that difference is visible rather than theoretical: the two leading tokens are exactly tied at45.42% each. Keeping only the single highest-probability token (top-1) keeps one of them and drops the other, even though nothing distinguishes them — whichever is listed first in the vocabulary survives. Top-p at p = 0.5 keeps both, because one tied token alone falls just short of 50% and the rule is to keep the smallest group that reaches the target, not the smallest number of tokens.
What this does not show
This page does not draw a token. It shows the probabilities a model would draw from, at every stage, as numbers; the actual draw is a single weighted random choice over whichever distribution is on screen, and nothing about that step is specific to language models. Top-k and top-p are shown independently here, both applied to the same temperature-scaled distribution — not chained into one specific production pipeline, since more than one real serving stack composes them differently. And a real model's vocabulary and weights are far larger than anything on this page: tens of thousands of tokens, scored by a forward pass through billions of learned parameters, not by a rule a person wrote down.
Cost
- Time
- O(n log n)
- Extra space
- O(n)
- Steps recorded
- 8, always
n is the vocabulary size. Softmax itself is O(n); ranking the vocabulary for top-k or top-p is O(n log n), which is the term that dominates.
Unlike every Track A tool here, this one has no ceiling and no input to refuse. The vocabulary size is fixed by the data file, so the step count — eight, always — is a constant: the preset changes which fifty numbers are used, and temperature, k and p each change a value inside the same eight-step pipeline, never its shape. What is validated instead is the data file itself, at load, because that is the part a person could edit.
Source
Holtzman et al., “The Curious Case of Neural Text Degeneration” (2020) introduces nucleus (top-p) sampling. Radford et al., “Language Models are Unsupervised Multitask Learners” (2019) uses a 50,257-token vocabulary, cited here for the scale of a real one. Brown et al., “Language Models are Few-Shot Learners” (2020) reports a 175-billion-parameter model, cited for the scale of a real forward pass.