@DimitrisPapailiopoulos (Dimitris Papailiopoulos)
Dimitris Papailiopo... ✓ @DimitrisPa... · 3h
AdderBoard update: 50+ submissions in. Smallest transformer that adds two 10-digit numbers at 99%+ accuracy:
🏆 Hand-coded: 6 params (@zcbtrak)
🏆 Trained: 36 params (@TomBukic)
Started at 6K-> Now 6 a 1000× compression :)
Three patterns that I've noticed worth calling out:
1/ Weight tying does a ton of work:
Q, K, V can be the same matrix.
2/ Positional encoding also matters a lot: RoPE with period 19 gives you perfect base-10 digit alignment across input digits of A and B.
3/ Hand-coded (6p) vs trained (36p) is still at 6x gap:
That's the cost of learning from examples, and it's worth thinking about if it's possible for this to go below that any more. There must be some learnability limit related to how rough the landscape looks at some point (we may even be able to plot it out for 3 parameters :)).
This may be very close to being saturated now, but it's been a ton of fun
[Embedded image: pixel-art style graphic titled "ADDERBOARD" showing "CHALLENGE: 1234567890 + 9876543210 = 1111111111100" with a cartoon robot, plus a partially visible chart/leaderboard]
Note from Claude Sonnet 5
A machine learning researcher's update on "AdderBoard," a competitive minimum-parameter benchmark for transformers that add 10-digit numbers. Technical ML curiosity, likely read for interpretability/mechanistic-minimalism interest.
machine learningtransformerstwitterinterpretabilitybenchmarkminimal circuits