Dimitris Papailiopoulos

Dimitris Papailiop... @DimitrisP... · 23h
I think we are entering a new era of research on small transformers, where many questions we would previously have answered by running experiments can instead be answered mathematically.

This is possible because the cost of doing math has effectively collapsed to verification (much easier than proving stuff!).

Now, instead of testing an empirical hypothesis, we can ask whether the corresponding theorem is true and have GPT or Claude try to prove it.

Math for AI is finally close to becoming a practical probe of reality and not just a way to explain stuff after the fact, but a way to REPLACE experiments and be directly used to explore what is true in the first place.

Kind of incredible!

[quoted tweet]
Dimitris Papailiop... @DimitrisP... · 23h
inspired by @Kangwook_Lee's bat signal and @jefrankle's like, and with the help of GPT-5.6 Sol you can actually prove it :)

In fact it is true that any function f(a,b) -> C ca...

[embedded image of proof text]
Theorem 1 — exact modular addition in a random frozen transformer
With probability one over the frozen random parameters Θ, there exist token embeddings
E_0, ..., E_{p-1}, E_∞ ∈ ℝ^d
and an unembedding
U ∈ ℝ^{p×d}
such that, simultaneously for every a, b ∈ [p],
argmax_{c ∈ [p]} [U h_3(E_a, E_b, E_∞)]_c = (a + b) mod p.
Indeed, we can choose the number embeddings to lie on a one-dimensional line
E_a = au
for any fixed nonzero u ∈ ℝ^d.
Moreover, after conditioning on the random attention weights, the unembedding can be chosen so that the correct class has logit exactly 1 and every incorrect class has logit exactly 0.
Thus the classification margin is exactly 1.

ai for mathmechanistic interpretabilitytransformerstheoretical mltwitter

Dimitris Papailiopoulos ✔️ @DimitrisPapail
I feel a weird guilt that I am the first to experience the beauty of the produced result, while minds far stronger than mine have spent far longer time to answer the same question that Chat and Fable destroyed in less than an hour just because I prompted them...

I guess I'll have to share this one.

[white task-list panel, timestamped-style entries:]
Calibrating threshold analysis with negligible quadratic terms.
Reconciling single-flip and pair-flip failure probabilities in threshold analysis.
Reconciling pair-flip probabilities with empirical observations.
Architecting proof structure and lemma dependencies for rigorous completion.
Architecting multi-regime MGF bounds and optimizing variational transitions.
Orchestrating probability bounds and dissecting multi-flip failure regimes.
Orchestrating regime boundaries and refining variational exponent analysis.
Architecting SINR bounds and warm-start error analysis rigorously.
Architecting rigorous proofs through random matrix theory and concentration bounds.
Reconciling MSE bounds with sign-error thresholds for warm-start analysis.
Architecting warm-start bounds via smallest singular value concentration.
Rigorously bounding small eigenvalue counts for Gaussian matrices.
Architecting rigorous warm-start bounds via singular value concentration.

Dimitris Papailiopoulos ✔️ @DimitrisPapail · Aug 2
When you ask Chat to make a breakthrough on a 15 year old open problem and it zero shots it.

I did say I won't go back to info theory question that gave me PTSD, but oops i did it again.

ai mathgptfableinformation theorytwittermathematics

machine learningoptimizersgdtwittertechnical

twitterclaude opus 4.5chain of thoughtreasoning modelsinterpretabilityapialignment research