How Tokenization Decides Whether a Transformer Can Add

The same small transformer, trained on the same additions, either extends far past its training length or fails soon after it, depending only on how the numbers are cut into tokens, the order the answer is written in, and whether position follows place value. A controlled grid of 16 tokenization × encoding cells, three seeds each, at matched training compute.

How numbers are split into tokens is usually treated as a preprocessing detail. For arithmetic it is not: it decides which algorithm is easy to learn, and whether that algorithm keeps working on numbers longer than any seen in training. This log measures that directly, with everything else held fixed.

The setup

Addition problems with operands of up to 10 digits, freshly sampled, 30% of them built to contain long carry chains. Small GPT-style decoders (6 layers, 384 wide, ~10.6M parameters), EMA weights, three seeds per cell. Each tokenization is three independent choices:

• Chunk size k: 1 or 3 digits per token, or GPT-2's BPE (its 994 digit tokens remapped to a compact vocabulary so model size matches).
• Cut: chunk boundaries counted from the units digit (right-cut: chunk i always holds places 3i–3i+2) or from the leading digit (left-cut, as GPT-style tokenizers do: boundaries depend on the number's length).
• Order: the answer written normally (most significant first) or reversed (units first, so each answer chunk depends only on its own column and the carry already written).

Two position encodings: standard (rotary positions stretched by a random factor 1–4 per training sequence) and place-value (half of the rotary pairs driven by each chunk's index within its own number, also stretched, a form of position coupling). Cells are compared at matched training compute (~4.3·10¹⁵ FLOPs: 16k steps for k=1, 31k for GPT-2, 33k for k=3, since longer chunks mean shorter sequences). Test tiers: in-distribution (1–10 digits), a gap tier (11–13), and two out-of-distribution tiers (19–21, 24–26).

Results

What the grid says

1. Extending past the training length takes three things together: reversed order, chunks aligned with place value (k=1, or right-cut k=3), and a place-value position signal. Remove any one and accuracy beyond the training length is near zero, often with perfect in-distribution accuracy. Only the two cells with all three extend.
2. Order. Writing the most significant digit first learns in-distribution but fails within a few digits of the training length, for every tokenization: the first answer token needs the carry from every column below it.
3. Cut. Left-cut fails to extend even when reversed. Its chunk boundaries move with each number's length, so the rule "add chunk i of each operand" is a different rule for every pair of lengths.
4. BPE. GPT-2's tokenizer is worst on both counts: it never reaches the fixed-chunk cells' in-distribution accuracy and barely extends in either order. Its splits are irregular (1034567 → [103][45][67]).
5. The place-value signal only helps when chunks align with place value. With misaligned chunks it is useless or harmful.

This matches what Singh & Strouse found in frontier models: forcing GPT-3.5 and GPT-4 to group digits from the right (with commas) made their addition much more accurate, which they trace to alignment between the tokens of the operands and the answer.

How far the good cells reach

k=3 reaches fewer digits than k=1, not three times as many. The limit is not sequence length but coverage: the place-value rotation has only seen index differences up to about 4 chunks × the stretch of 4 during training, and reach ends near the edge of what it has seen. Reach depends on tokenization through how many chunks training numbers contain.

With a better position encoding

A companion log (Why Rotary Embeddings Stop Extrapolating) traced the coverage limit to rotary aliasing and fixed it for k=1: 2D rotary positions (row × column), a fixed multiplicative decay with column distance, and a position-free attention sink took k=1 to 100% at 200 digits on every seed. The same fix, trained from scratch on k=3 at the same compute:

90 random problems per length. Right-cut with the fix: 1.00 / 1.00 / .67 at 60 digits and .99 / 1.00 / .49 at 200 (seeds 0 / 1 / 2), from 0 / 0 / 0 at 60 with index-RoPE. Seed 2's remaining failure is a separate mechanism (dilution of a weak attention sink; see the companion attention log).

Left-cut with the same fix gets .37 / .29 / .51 at 60 digits and .21 / .01 / .16 at 200, more than its zero baseline. Split by problem shape, nearly all of it comes from problems that are mostly copying:

So the fix sharpens the first finding rather than changing it. With place-aligned chunks the remaining limit was the position encoding, and fixing it takes k=3 from zero at 60 digits to near-perfect at 200 on two of three seeds. With left-cut chunks the model still cannot add past its training length at all: no position encoding can line up chunks whose boundaries move with each number's length.

And with more data instead

The other way to reach further is to train on longer problems. Keeping the place-value encoding and changing only the data (10% of problems with a 20–40 digit operand, warm-started from the 16k-step models):

Dashed line: 160 digits, four times the longest training length. Means over three seeds.

Long data makes every seed (near-)perfect to 120 digits, but the cliff only moves: from about 40 to about 160 digits, four times the longest training length both times, which is exactly the encoding's random stretch factor. Data extends coverage; it does not teach a mechanism that works beyond it.

Limits

• One task (addition), one model size, three seeds per cell; k=2 and trained (rather than stock) BPE were not run.
• The place-value index is computed from the tokenization, not learned; the "with a better position encoding" results use task structure (rows and columns).
• Reach is measured on 90 random problems per length.

References

1. A. K. Singh, D. J. Strouse, Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs, arXiv 2402.14903 (2024).
2. H. Cho et al., Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure, NeurIPS 2024 (arXiv 2405.20671).
3. S. McLeish et al., Transformers Can Do Arithmetic with the Right Embeddings, NeurIPS 2024 (arXiv 2405.17399).
4. Y. Zhou et al., Transformers Can Achieve Length Generalization But Not Robustly, arXiv 2402.09371 (2024).
5. A. Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2), OpenAI (2019).