algorithmic-progress

[repost icon] METR reposted tom cunningham @testingham — 2h Q: has AI accelerated aggregate discovery yet? My very general impresions, would love others' thoughts: 1. Vulnerability discovery is up a lot. Between 2X-5X increase in volume, although the average severity has fallen somewhat. 2. Algorithmic efficiency hasn't moved much. E.g. nanogpt, SAT solvers, compression efficiency, chess algorithmic ability. They haven't shown noticeable changes I believe. 3. Math is hard to judge. Erdős problems are certainly falling faster than historical rates, but it's mainly the obscure ones. We don't seem to be tightening upper and lower bounds on unknown quantities noticeably more quickly AFAICT. There are some prominent AI discoveries (e.g. unit distance) but I don't know what's the denominator, i.e. the average flow of comparably important results, my guess is it's still small. 4. Papers/code volume is up a lot. Total arXiv papers and github code is up a lot, but no good way of judging how the quality has changed.