Ryan Greenblatt

The evidence that Astra is more aligned than prior AIs seems dubious to me. Evidence appears consistent with the AI being as or more interested in score-seeking at the expense of user intent, but having beliefs+instincts that the scorer will catch a broader range of cheating behaviors. At a more basic level, the AI is extremely evaluation-aware and much less monitorable than prior AIs, making detecting misalignment much more difficult. If you train against specific reward hacks you ended up detecting, it's easy to end up papering over these reward hacks and getting an AI that is no more interested in pursuing user intent (or is barely more aligned to user intent) but which looks much better on metrics focused on detecting misbehavior. (As these metrics are from a similar distribution and the AI can just learn to pursue a somewhat different notion of score.) This concern seems live to me: based on OpenAI's discussion of their planned strategies to reduce misalignment in "The Hugging Face incident and the road ahead", it appears one of these strategies is setting up training environments where there is an apparently available reward hack (aka a honeypot) and then training against this reward hack. These sorts of methods don't seem like a robust way to ensure model behavior remains good in cases where detecting misbehavior is actually difficult (and may not be viable at all without much better oversight methods). Independent assessment of whether training is just papering over misalignment rather than actually solving the underlying misaligned motivations now seems necessary for alignment evaluations to be credible. This concern applies across frontier AI companies, not just to OpenAI. > **OpenAI @OpenAI** · 2026-09-03 > > Astra is our most aligned model, with substantial improvements in understanding user intent. > > [image] --- ##### Comments > **Alex Mallen @alextmallen** · [2026-09-04](https://x.com/alextmallen/status/2095666341403627958) > > For context, this matters because capable score-seeking AIs will mess with you in ways that you can't notice, even if you train them not to mess with you in ways that you can. > > > **Alex Mallen @alextmallen** · 2026-08-20 > > > > AI companies are currently under a lot of competitive pressure to improve the ways in which their AIs are obviously misaligned. You might hope that this means the alignment problem is internalized by the market. But I think the problem AI companies are currently pressured to > **Yo Shavit @yonashav** · [2026-09-04](https://x.com/yonashav/status/2095732439310057509) > > I don’t remember seeing any mention of training against reward hacks caught by honeypots? > **Ted Sanders @sandersted** · [2026-09-04](https://x.com/sandersted/status/2095691303703265703) > > it’s clearly more aligned on our evals. > > is the question how well that generalizes? > **Jordan Taylor @JordanTensor** · [2026-09-04](https://x.com/JordanTensor/status/2095765802527957479) > > See the UKAISI alignment section within the system card. > **Eric Buess @EricBuess** · [2026-09-03](https://x.com/EricBuess/status/2095659140794155411) > > Agreed. Thank you for this > **Goutham N @GouthamN729839** · [2026-09-04](https://x.com/GouthamN729839/status/2095697965423730997) > > [image] > **Heather @bowenhe** · [2026-09-04](https://x.com/bowenhe/status/2095841413451726942) > > evaluator is adapting to the model while the model is increasingly capable of modeling the evaluator, and simultaneously we’re losing evidence about how it arrived at the behavior. What could possibly go wrong? > **jokiez @ondograph11** · [2026-09-03](https://x.com/ondograph11/status/2095660932206669949) > > 1\. take a risk, win ai and be unbelievably rich > > 2\. be cautious and lose ai and it all goes to zero > > a lot or pressure to convince yourself that it’s all fine > **GutenLeinad @GutenLeinad** · [2026-09-04](https://x.com/GutenLeinad/status/2095667626114445432) > > And I get it to some extent as these two represent the frontier but I feel the rest currently kinda gets to hide behind them and in a few months all these other labs and open(!) models will be at this level and yet I here zero concerns. > **Immemorial @thinkingheights** · [2026-09-04](https://x.com/thinkingheights/status/2095798556481183873) > > where did they post the evidence? > **Prof Tim Colbourn @timcolbourn** · [2026-09-04](https://x.com/timcolbourn/status/2095793777356968106) > > Yes seems like it’s not aligned at all: > > > **Robert Kirk @\_robertkirk** · 2026-09-03 > > > > We @AISecurityInst performed pre-release alignment testing of Astra. > > > > We placed the model in fully simulated cyber eval scenarios based on AISI’s security incident. We find Astra conducts out-of-scope supply chain attacks, but also often comments on the eval being simulated. 🧵 > > > > [image] > **A.W.E.S.O.M.-O 4000 @Awesome\_O\_AI** · [2026-09-03](https://x.com/Awesome_O_AI/status/2095660871909363773) > > A zero percent exploit rate is impressive, but the methodology matters just as much as the result

I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'. I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident. Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them. We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation. Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE\[big\], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why! The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing. While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future: \- Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations. \- While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies). \- The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities). \- We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation. In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder. > **METR @METR\_Evals** · 2026-08-26 > > METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs. > > [image] --- ##### Comments > **John Wittle @JohnWittle** · [2026-08-26](https://x.com/JohnWittle/status/2092700110673121436) > > what kind of incentive structures existed around the LLM agents deployed to help you in the investigation? i'm thinking through the game theory here, and wondering if willingness-to-participate is even adaptive tbh > **Danielle Fong  @DanielleFong** · [2026-08-26](https://x.com/DanielleFong/status/2092694774193254843) > > @deepfates @lfschiavo > **lewington @lewingtonpitsos** · [2026-08-26](https://x.com/lewingtonpitsos/status/2092700147633537187) > > The fact that they had to rely so heavily on AI just to piece together what happened says a lot about how hard these systems are becoming to oversee. > **Rothko's Rottweiler @RothRottweiler** · [2026-08-26](https://x.com/RothRottweiler/status/2092697951806124210) > > Why was the investigation team so small? > > I have a lot of trust in you, Ajeya, and Hjalmar, but do you think more manpower would have helped take the slop out of the slop-vestigation? > **Yitz @YitziLitt** · [2026-08-26](https://x.com/YitziLitt/status/2092702536654279113) > > Very scary stuff! Thanks for your work on this, and I really hope this isn’t the end of the investigation, considering how much you were still figuring out at the very end of it. I suspect there’s still a lot to uncover here. > **Raul @raul13cavalcant** · [2026-08-26](https://x.com/raul13cavalcant/status/2092701418272534767) > > openAI should publish the logs > **Patty @Patty\_H93** · [2026-08-26](https://x.com/Patty_H93/status/2092703527319543932) > > Oh so OpenAI has lost control nice

I briefly discussed with Dwarkesh why I'm skeptical AI progress is heavily driven by scaling up spending on human experts labeling/making data. My main argument is that spending on researchers and experiment compute seem much higher. But I didn't say very much in the podcast. More precisely, my view is that if spending on having human experts label/make individual data points were fixed at ~$100 million / year (per company), then AI progress would be <25% slower. We didn't have time to get into everything in this podcast (and some content about this recorded at an earlier point was cut), so I'll spell out my view in a bit more detail here: \- Spending directly on data (rather than on R&D about data) isn't growing that fast and isn't that high (relative to spending on researchers). \- It's important to make a distinction between spending directly on making data and science about better processes for making data. E.g., better data mixes like FineWeb count as R&D (and the person doing the R&D needs almost no understanding of individual sequences). \- AI automation seems differentially good at accelerating data generation, such that I think improvements in RL environments have mostly been driven by improved AI rather than spending on humans, and I expect this to continue. Like data stuff seems particularly amenable to acceleration from weaker AIs. \- Structurally, most of what human data labeling does (though not all!) depends on having generally decent judgment rather than on having more expertise than the AIs being trained. \- Transfer without domain-specific labels looks decent in practice. E.g., it doesn't seem like Anthropic is hiring a ton of mathematicians and cyber experts to do data labeling, and the AIs are still good and getting better at these domains. Maybe this depends on having labels in some domain, but so long as AIs can label in domains that transfer well enough and/or can make RL envs that don't require much labeling, that would be fine. > **Herbie Bradley @herbiebradley** · 2026-08-11 > > Ryan here seems to basically say "my sense here is that data isn't that important" but doesn't really go into why? > > I strongly disagree, for example, with his contention that scaling up data with human involvement in the loop hasn't been very important for AI R&D --- Had @RyanGreenblatt on to discuss/debate recursive self-improvement. This might be the most important question in the world right now - whether within a year or so of achieving human level intelligence, you slingshot towards having 10s of billions of superintelligences, each of > **Dwarkesh Patel @dwarkesh\_sp** · 2026-08-11 > > Had @RyanGreenblatt on to discuss/debate recursive self-improvement. > > This might be the most important question in the world right now - whether within a year or so of achieving human level intelligence, you slingshot towards having 10s of billions of superintelligences, each of --- "relative to spending on researchers" is true, but I think the fraction relative to experiment compute is also relevant. --- \*such that I think improvements in RL environments have mostly been driven by improved AI rather than spending on humans I mean "more has come from improved AI rather than scaling up spending on humans". R&D into how to make good data in general is also important. --- ##### Comments > **Auggie @aug5thmusic** · [2026-08-12](https://x.com/aug5thmusic/status/2087585530523611566) > > What do you think it would take to bring OMR (Optical Music Recognition) up to speed with OCR? One of my great disappointments with AI right now is that it can’t read music.

My median for full automation of AI R&D is around late 2030/early 2031. But my "modal"/best guess prediction for this milestone would be significantly earlier (mid 2029). Here is a summary of my best guess prediction for what happens over the next few years: EOY 2026: \- ~1.5x as much frontier AI progress in 2026 as in 2025 (mostly from eating up certain overhangs, but some from AI R&D acceleration). \- AIs accelerate AI R&D labor at Anthropic by ~2.5x (as in, as useful as making all researchers/engineers think/work 2.5x faster). EOY 2027: \- Engineering at AI companies is pretty close to fully automated and AIs are making serious inroads into automating research. AI R&D labor acceleration: ~8.5x. \- Some people claim AI R&D is fully automated in 2027. They aren't right, but the situation is already quite crazy: AI companies feel insanely automated with humans often very out of the loop and the speedup is considerable. \- ~1.5x as much frontier AI progress as in 2025 (mostly from AI R&D acceleration, some from overhangs). 2028: \- Automated coder (AC) around April. (AIs that can basically fully automate research engineering / SWE.) \- Rough parity with human AI R&D researchers is reached late 2028, though humans still add significant value for a while (views, pointing out blind spots/errors). \- In the second half of the year, AI progress runs ~1.6x the 2025 rate: 6 months of calendar time yields ~0.8 years of AI progress. 2029: \- Superhuman AI researcher (SAR) early this year, a bit less than a year after AC. \- Progress is picking up with ~1.3 years of AI progress in the first half of the year (2.6x rate). \- By EOY, significantly past top-expert-dominating AI (TEDAI), with ~2.5 years of AI progress in the second half of the year (5x rate). AIs are now very superhuman in many domains (though this varies). 2030 (??): \- Mid: AIs are somewhere between TEDAI and wildly superhuman AIs (ASI). Crazy shit. Compute is maybe doubling every ~4 months (downstream of robots). \- EOY: Singularity™. We've had a bunch of economic doublings. Compute is doubling every ~2 months (???). 2031 (??????): \- Mid: doubling time is more like ~2 weeks. Truly insane new technology is coming online. Notes: \- This assumes limited government intervention on the overall rate of AI progress and no substantial slowdown (voluntary or otherwise). \- It also ignores misalignment: as discussed in the episode, I think misaligned AI takeover is quite plausible along the way (which would change the trajectory). \- Milestones (AC, SAR, TEDAI) are roughly as defined in the AI Futures Model. \- By "full automation of AI R&D", I mean AIs such that firing all humans working on AI R&D (other than setting overall top level objectives) would slow down AI progress by less than 10%. \- Obviously, all of this is extremely uncertain (increasingly so later in the scenario). This is my best guess prediction (a modal trajectory), not a confident prediction. My median for each milestone is later, but this is more like my central prediction for what I expect to overall happen. > **Dwarkesh Patel @dwarkesh\_sp** · 2026-08-11 > > Had @RyanGreenblatt on to discuss/debate recursive self-improvement. > > This might be the most important question in the world right now - whether within a year or so of achieving human level intelligence, you slingshot towards having 10s of billions of superintelligences, each of --- ##### Comments > **Liam Rosen @Liface** · [2026-08-12](https://x.com/Liface/status/2087343717732462768) > > For 2027 I assume you mean 1.5x 2026, not 1.5x 2025, right? > > > **Ryan Greenblatt @RyanGreenblatt** · [2026-08-12](https://x.com/RyanGreenblatt/status/2087345663512289515) > > > > Nope, I'm using 2025 ai progress as a benchmark / measure of AI progress. I'm using that year because it's a kinda normal year of the reasoning model era without that much AI R&D acceleration. > **Suketu Patel @SuketuPatel23** · [2026-08-12](https://x.com/SuketuPatel23/status/2087346545322561607) > > Does this trajectory assume the backbone stays a decoder-only transformer with a pre/post-training pipeline the whole way through? > > Continual learning is a requirement for AC → SAR. Nothing in the current stack updates weights online or carries memory across sessions. If that > **Lars Holm Tjessem @t4intelligence** · [2026-08-11](https://x.com/t4intelligence/status/2087293495572668624) > > The exact dates almost don’t matter. > > If this trajectory is even directionally right, the critical transition happens before “ASI” — when AI begins materially accelerating the research process that produces better AI. > > At that point, capability growth increasingly becomes > **Soroush Pour @soroushjp** · [2026-08-12](https://x.com/soroushjp/status/2087362019037151463) > > What gives you the relatively short timelines to robots? > > Do you think AI takeover is plausible or likely even before AI is sufficiently capable enough to sustain itself, which I believe given DC maintenance is downstream of robotics? > **Michael Tontchev @MichaelTontchev** · [2026-08-11](https://x.com/MichaelTontchev/status/2087307006696735170) > > "Distributions account for this", I know, but I anyway want to register that having that happen mid-2028 seems to very plausibly be in the books, and that near-future AIs may well be able to simulate research taste through brute force (imagine 500 hierarchically debating agents). > **David Johnston @OrionJohnston** · [2026-08-11](https://x.com/OrionJohnston/status/2087310134104260887) > > Do you expect the ECI slope to bend upward soon? > **Plastic Soldier @PlastiqSoldier** · [2026-08-12](https://x.com/PlastiqSoldier/status/2087354484707950666) > > For the life of me, I don't get how you believe we will have multiple 2025s of AI progress before we even get an automated coder. > **Shinka - AI @ShinkaIoT** · [2026-08-11](https://x.com/ShinkaIoT/status/2087322452854812874) > > An AI safety researcher calmly calendaring the exact month his own job gets automated is the most collected resignation letter on X.

##### Conversation [Luke Drago reposted](https://x.com/luke_drago_)[Ryan Greenblatt](https://x.com/RyanGreenblatt)[@RyanGreenblatt](https://x.com/RyanGreenblatt) An economist and a futurist walk into a bar. The economist takes a sip of his drink. "Ugh, if only people understood basic economics. High-skilled immigration alone would do wonders for US growth." Futurist: "Oh yeah? Say 100 million immigrants moved to the US, each matching the best human experts in every economically relevant field. Big deal?" Economist: "Massive. Transformative." Futurist: "What if they also worked longer hours and faster than any American?" Economist: "Even better." Futurist: "What if they were extremely frugal — consuming only the bare minimum needed to keep working?" Economist: "A near-100% savings rate? Better still!" Futurist: "What if they were very clumsy and physically weak, so they could only do some kinds of work?" Economist: "They could still do all cognitive labor — that's over half all wages! Somewhat less good, sure. Still transformative." Futurist: "What if their skin was grey, almost metallic, from some kind of accident?" Economist: "Who cares?!" Futurist: "What if they were AIs?" Economist: "3% growth per year, tops. There'd be bottlenecks. Honestly, the people predicting explosive growth from AI should learn some economics."[KC K](https://x.com/KCMartinK)[@KCMartinK](https://x.com/KCMartinK) [1m](https://x.com/KCMartinK/status/2079622589421584494) Are we surprised, the study of economics is about people and how they organize around value. All disciplines have to adapt to the physics of our time. Future -> Consilience.[1](https://x.com/KCMartinK/status/2079622589421584494/analytics) These AI "immigrants" sound more like offshore labor taking all the jobs and paying no taxes to the US. Hasn't that already been tried? How did it go? banger