A transformer that has learned the rule for adding numbers still fails on longer numbers, because it cannot find the right digits. Tracing that failure head by head in a small, fully controlled setting leads to a precise mechanism: rotary position encodings are a finite Fourier series over distance, and past the trained distances a far-away token can line up with the target and win the attention. Two changes, multiplying attention scores by a fixed decay in column distance and taking the rotation away from the attention sink, take every seed from about zero to 100% at twenty times the training length.
This started as a study of how tokenization affects a small transformer's ability to add, and turned into a question about position encodings. Once the digits were tokenized so that the right algorithm was easy (one digit per token, answer written units-first), the model learned addition perfectly at the lengths it trained on, and it kept getting every individual digit decision right at lengths far beyond them. What broke was the step before the arithmetic: finding the right digits in a long sequence. That is a problem every long-context language model has, so it seemed worth understanding exactly.
The task is addition written least-significant digit first, one digit per token, e.g. 5 4 3 2 1 + 8 7 6 = 3 2 0 3 1 for 12345 + 678. Models are small GPT-style decoders (6 layers, 384 wide, about 10.6M parameters), trained on problems with operands of up to 10 digits, freshly sampled every batch, with exact-match accuracy measured on problems up to 200 digits. Each digit also carries a place-value index (its column), and the encodings below decide how that index and the token's position reach attention.
Reversed order matters: each answer digit then depends only on the two operand digits in its column and the carry from the previous step. A person adding on paper does exactly this, and does it at any length, because writing the numbers under each other puts the digits they need in the same place every time. The question is whether a position encoding can give a transformer the same alignment.
Plain rotary position encoding (RoPE) fails as soon as problems get longer than training: perfect in-distribution, near zero a few digits past it. Training with randomly stretched positions (one random scale factor per sequence, which equals multiplying every rotation angle) and adding a rotary encoding driven by the column index ("index-RoPE") produced models that were perfect on the standard test lengths. But a reach probe showed the limit was a dial: it tracked the stretch factor × the longest training sequence. Doubling the stretch doubled the reach.
The most natural fix for alignment is 2D RoPE: half of each head's rotation pairs turn by row (first number, second number, answer) and half by column, with no sequence position at all. Attention then sees the numbers written under each other, as in a grid. Vision transformers use the same construction for image patches. On the standard test lengths it was the most consistent encoding across seeds, but its reach still stopped at about 40 digits:
Exact-match accuracy on random problems (90 per length), trained on operands of up to 10 digits. Gray and teal lines are means over 3 seeds; the shaded band is the decay model's range across seeds (the position-free-sink model has no band: all three seeds are at 100% at every length). The additive penalty is drawn per seed, because one seed behaves completely differently from the other two.
Each step below started from a hypothesis, tested it with a probe on the trained models (no retraining), and either confirmed it or ruled it out.
+ and = tokens: an improvised attention sink, the same phenomenon known from LLMs. On long problems that sink fades (1.00 → 0.00 in the strongest head) and the attention lands on distant operand digits.| contribution | 20 digits (within training) | 60 digits (beyond) |
|---|---|---|
| fast pairs (period < 20 columns) | +7.5 | +14.2 |
| mid pairs (period 20–200) | −10.9 | +14.9 |
| slow pairs (period > 200) | −1.3 | −0.9 |
| row pairs | +8.1 | +6.2 |
| total (separator sink) | 3.4 (3.2) | 34.3 (22.6) |
The fast pairs carry the "this is my column" signal, but they repeat every 6–11 columns, so tokens 6, 12, 18… columns away look like matches: aliases. The mid pairs are what normally rejects those aliases. They only ever complete part of a cycle within the trained distances, and beyond them the learned rejection flips sign. An aliased far token then outscores the sink and takes the attention.
For one head, the score between a query and a key at column distance Δ is score(Δ) = Σm Am cos(ωmΔ + φm): a Fourier series over distance, with frequencies fixed in advance and amplitudes and phases chosen by the token contents. (This view of RoPE is not new: Hua et al.'s FoPE describes RoPE as an implicit non-uniform discrete Fourier transform and blames undertrained frequency components for length-generalization failures.) A two-term toy version shows the whole mechanism:
The mid term suppresses the nearby aliases (−5, −11, −17) but comes back round: at −35 the alias scores 19.8 against the target's 20. Training tunes heights and phases only over the distances it sees.
Could more terms push the repeat out of reach, say past 2000 columns? By design, yes; by learning, no. Fitting heights and phases to an ideal spike using only training distances (a ridge-regularized fit, standing in for weight decay) and then looking further out:
| frequency ladder | terms | peak (ideal 1) | worst side bump in range | highest value beyond training |
|---|---|---|---|---|
| RoPE geometric | 16 | 0.21 | 0.18 | 4.71 (22× the peak) |
| RoPE geometric | 2000 | 0.41 | 0.31 | 5.42 (13×) |
| harmonic, period 4000 | 256 | 0.21 | 0.19 | 9.47 (46×) |
| harmonic, period 4000 | 2000 | 1.00 | 0.00 | 0.00 |
More terms give more freedom, not more constraint: many combinations fit the trained range equally well and disagree beyond it. The single case that works is a complete basis (every frequency up to the grid's Nyquist limit, about 2000 rotation pairs for 2000 columns), far beyond a 64-dimensional head. The general lesson is about the basis's natural continuation: beyond the training data a learned kernel does whatever its basis does by default. Cosines come back round, polynomials blow up, a decaying basis goes to zero.
Holding a real query and its real target key fixed and sweeping only the key's column shows the learned kernel directly:
Near the target, both models learned a comb with teeth about every 6 columns (the fastest rotation pair), not a single spike. It works because scores are large: a tooth at half the target's score is still tens of points lower, which softmax turns into an overwhelming preference. With the decay, the teeth shrink as the distance grows.
Highest score in each 10-column window, as a fraction of the target's score. In the plain model the far teeth stay at 50–100% of the target out to 2000 columns, near-repeats of several fast periods at once; at −357 columns the gap to the target is only 1.5 points. With the decay they are close to zero.
Additive penalty (ALiBi-style, −slope × |Δcolumn|, learned per-head slopes starting at ALiBi's values). One seed reached 100% at 80 digits; the other two stopped at 40. The kernel probe explained the split: the working seed did its lookup in heads with slopes around 0.5, while a failing seed used a head whose slope stayed at 0.0048. Within the trained distances every slope is tiny next to the scores, so training never learns which head should carry the lookup. ALiBi works in language models because it replaces rotation: a near-zero-slope head there has no positional signal and matches on content only. Added on top of RoPE, a low-slope head keeps its periodic comb.
Multiplicative decay: multiply each score between number tokens by exp(−rate × |Δcolumn|), with fixed rates from 0.5 down to 0.05 across heads, and separator tokens exempt as keys so the sink keeps its score. A far alias is pulled toward zero however large the head's scores are, and no head can escape it. All three seeds now get well past 40 digits: 83% / 98% / 100% at 60 digits, 62% / 91% / 90% at 100, 34% / 77% / 77% at 200 (seeds 0 / 1 / 2). In the kernel the best false match beyond 10 columns is now tens of points below the target. But accuracy still declines with length, and unevenly across seeds.
The first decay runs exempted separators as queries as well as keys. That left the = token, which predicts the units digit, doing its lookup with the old undecayed scores, and many long-problem errors started at the units digit. Exempting separators only as keys fixed it: in the corrected runs no long-problem error starts at the units digit. All decay numbers here are from the corrected runs.
Exempting separator keys from the decay lets the stop step find its sink at any distance. It also leaves one score in the model undecayed: a query's score on + and =. In 2D RoPE every separator sits at the same column, so that score is a Fourier series in the query's column, and it has aliases like any other. Scoring every decision of 150 random 100-digit problems on its own (teacher-forced) shows them directly. Seed 0's errors are not spread out: they sit at a handful of places, exactly where its two main lookup heads suddenly put most of their attention on the separators.
| answer place | 30–31 | 42 | 54–55 | 74–75 | 98–99 | all other places |
|---|---|---|---|---|---|---|
| wrong decisions | 0 | 0 | 16 | 19 | 36 | 2 |
| separator share | 0.27 | 0.11 | 0.85 | 0.77 | 0.91 | ≈ 0 |
The spikes sit at the same places in seed 1 (42, 54–55, 74–75, 98–100), because they come from the fixed rotation frequencies; training only decides how hard each seed is hit. Seed 1 made 14 wrong decisions, seed 2 22, seed 0 73. Errors do not depend on carry chains (none at chains of 8 or more), and no stop decision is wrong. The kernel probe above had missed this because it sweeps operand keys only.
The sink needs a score that does not change with the query's column, and a rotation cannot give that. So the separator keys lose their column rotation entirely: their column-RoPE components are set to zero, and a query's score on them depends only on rows. Unlike the null-attention slot of step 4, this sink is a real key, so its score is a query–key product that grows along with every other score. Each decay model was warm-started with this change for 4,000 more steps (16k → 20k):
| model | 60 digits | 100 digits | 150 digits | 200 digits |
|---|---|---|---|---|
| decay (16k steps) | 83% / 98% / 100% | 62% / 91% / 90% | 54% / 86% / 83% | 34% / 77% / 77% |
| decay, 4k more steps (control) | 100% / 99% / 100% | 92% / 96% / 99% | 89% / 87% / 99% | 88% / 74% / 93% |
| decay + position-free sink (4k more steps) | 100% / 100% / 100% | 100% / 100% / 100% | 100% / 100% / 100% | 100% / 100% / 100% |
The extra steps alone help too: continuing the decay models for the same 4,000 steps without the change (the control) gives 88% / 74% / 93% at 200 digits. But the control's remaining errors still sit at the sink's spike places (teacher-forced at 100 digits, seed 0: 15 wrong decisions, all at places 55 and 98–99; seed 1: 17, all at 55 and 99), while in seed 2 longer training happened to flatten the spikes at those places. The position-free sink removes them in every seed. Teacher-forced on the same 100-digit problems, seed 0 now makes 2 wrong decisions out of 15,185 (down from 73), and seeds 1 and 2 none. On 150 further random problems each at 100 and 150 digits, one answer was wrong (a single carry, seed 0), and no answer had the wrong length.
None of the ingredients is new, and they match the long-context literature closely: RoPE fails just past its training length (Press et al.'s ALiBi paper), extrapolated attention scores can be very large (Chen et al., position interpolation), the problem dimensions are the ones whose period exceeds the training context (YaRN; Liu et al.'s critical dimension), high frequencies drive positional heads and the lowest act as near-semantic channels (Barbero et al.), and RoPE with a decay extrapolates better (Sun et al., XPos). In practice LLMs mostly cover the problem with training: rescale the rotations and fine-tune on longer data. What this setting adds is a clean, head-by-head view of the mechanism, and two concrete design rules that fall out of it. First, a rotating head needs a distance cut-off that scales with its own scores, and heads that must look far should probably not rotate at all. Second, a sink's score should not depend on position. In RoPE language models the sink is usually the first token, and its score is rotated by the query's distance from it like every other score, so the same kind of alias can appear far from the start. Whether that matters at LLM scale is not tested here.
Rotary encodings give attention a periodic notion of distance, and what a model learns about distance only holds over the distances it has seen. Beyond them a far token can line up with the target and take its place. That includes the sink the model uses to decide when to stop. Making the fall-off with distance part of the encoding itself, independent of how large a head's scores grow, and giving the sink a score that does not depend on position, removes that failure in this setting: every seed is at 100% on 200-digit problems.
Two later logs continue this one. What Attention Is Missing: the same fix on 3-digit tokens extends to 200 digits on two of three seeds; the third fails a different way (a weak sink diluted by many far keys), which leads to a single quantity behind every failure here and to attention that learns its own coordinates. How Tokenization Decides Whether a Transformer Can Add: the tokenization study this started from.