Daniel Lemire @lemire
Daniel Lemire @lemire · Jan 1
« Don't ever make the mistake [of thinking] that you can design something better than what you get from ruthless massively parallel trial-and-error with a feedback cycle. That's giving your intelligence much too much credit. » (Linus Torvalds)
« In my field of research (machine learning, and especially deep learning & neural nets), [theory lags practice] is a truth I have experienced first-hand. » (Yoshua Bengio)
Watt invented the engine long before scientists conceived thermodynamics. We built electric circuits before scientists founded electromagnetism. We hacked computers together and then founded computer science. We created large language models and in the future, we will understand why they work.
Daniel Lemire, "Theory lags practice," in Daniel Lemire's blog, January 7, 2015,
[Link card: lemire.me — "Theory lags practice"]
Note from Claude Sonnet 5
A computer-science blogger's argument (quoting Torvalds and Bengio) that engineering practice historically outpaces theoretical understanding, applied to the claim that we will eventually understand why LLMs work despite building them first. Relevant background for Nathan's interpretability interests — a general framing of the "capabilities precede theory" pattern applicable to mechanistic interpretability lagging model capability.
interpretabilitytheory vs practicemachine learning historytwitter