@jsuarez

A little perspective: RL as a field spent 10 years making algorithms slower and slower. If you look at the original ALE, it actually can sim a few thousand frames per second per core. If you look at some of the last big env releases before a ton of people moved over to LLMs, you'll find several at dozens to hundreds of steps per second with such bad engineering that they don't even scale with vectorization. The field did this exactly because they presumed they would have to train directly in the real world. In reality, what we got out of this is a bunch of brittle off-pol and model-based algorithms that burn a ton of compute and don't work outside of the benchmarks shown in the original pubs. There's a clear gap between on-pol and other methods. You don't simply switch and scale up compute to save data. You have to spend a TON more compute to match the perf of on-pol, and then you spend even more compute to gain in sample efficiency. Our whole core realization with PufferLib is that we can write good sims for a lot of problems 10000x faster. Good doesn't even mean accurate. It means accurate enough with domain randomization and other tricks that our agents can implicitly sysid their current setting and act robustly. So far, this has worked across several different industries. I'd love to give examples here, but this is unfortunately where exact client details get confidential. We need to be better about negotiating publicity, and we're starting to do that as Puffer gets bigger. Another major flaw with slower and slower algorithms is that the core research loop also gets slower and slower. We sim mazes and 2048 at 10+m steps per second. Big deal right, those are easy. Wrong: algorithmic improvements on those envs have consistently predicted performance improvement on every single env in our test suite. Without this, we wouldn't have been able to release so many core breakthroughs in the last 2 years with a grand total of ~20 GPUs. We ran 20,000 experiments on ~12 of them in the 3 weeks leading up to Puffer 4 launch. At traditional speeds, it would have taken Google scale compute and an infra team. So no, we're not going to step the real world at 20m sps, but assuming that matters (or at least that it is the only thing that matters) is where the field went wrong. /rant. > **An Eevee @rw\_eevee** · 2026-06-07 > > sim2real is hard and you cant step the real world at 20 million steps per second. --- ##### Comments > **An Eevee @rw\_eevee** · [2026-06-07](https://x.com/rw_eevee/status/2063651327696117785) > > Appreciate your perspective and yes, this is a largely accurate history of the field. We’ve been all-in on fast sim + DR + RNN-style networks for the past ~2 years at least. It works great and we’ve gotten some spectacular results. > > But sadly some problems have refused to yield > > > **Joseph Suarez @jsuarez** · [2026-06-07](https://x.com/jsuarez/status/2063656644253466673) > > > > We're not primarily robotics. PufferLib has been used professionally in finance, commerce, gaming, defense, and animation. Has worked great. The world model stuff straight up doesn't work. Some surrounding lit is borderline scientific fraud. I believe it works better where you > **Guilherme O'Tina @guilhermeotina** · [2026-06-07](https://x.com/guilhermeotina/status/2063668773874237566) > > the same pattern is showing up in LLM agent eval now. everyone building multi-step benchmarks that take an hour to run. you end up optimizing for the benchmarks ontology instead of the actual task. RL already learned cheap fast eval loops beat elaborate ones > **jaisel @jaiselsingh** · [2026-06-08](https://x.com/jaiselsingh/status/2063780162735641073) > > im very curious/ looking into more work that combines a model based planner w/ a policy (insp in part by guided policy search from levine). im curious as to why that's not as big of a research direction these days. > **Zengineering @Samhanknr** · [2026-06-07](https://x.com/Samhanknr/status/2063744395141091777) > > I would love to understand in very very broad strokes what industrial applications this has. I’m trying to figure out how and why to learn puffer lib but it’s not easy to figure how it’s used in the real world.