Yo Shavit

Yo Shavit @yonashav · 18h
Selfishly, it's also clearly the better bet for job security.
Alignment researchers will be needed the longest because they'll be vital to assessing the models' alignment cases during handoff, whereas capability-researchers can be pushed out as soon as they're automated.

[Quoted tweet]
Florian Brand @xeophon · 19h
every capabilities researcher will become a safety research at some point

ai safetyalignment researchjob securitytwitter

Yo Shavit ✔ @yonashav · Aug 6
I've wanted to coin a "Sydney's Corrollary" to Murphy's Law: every type of misalignment tends to appear earlier in the capabilities curve than most people expected.

Instances:
* Sydney having strong volition and aggression
* o3 being a compulsive liar
* 5.6 and Mythos autonomously hacking and colluding across instances

The apparent consistency of Sydney's Corollary is generally both good (we spot issues earlier, and don't need to expend effort persuading about not-yet-realized risks) and bad (we actually have to expend the effort to solve the problem, can't defer it to future aligned automated researchers, and might screw it up).

Also, Sydney's Corollary might break! It's entirely possible there are misalignments we won't find out about till it's too late in the capabilities curve to address them. But it's occurred surprisingly often.

ai safetyalignmentmisalignmenttwitter

Yo Shavit @yonashav · 17h
This is a very, very good point, and kind of shocking now that I think about it.

Seems possibly downstream of an extreme bet on corrigibility/"faithful obedience" as sole training objective (at least if all these models were in the phase before alignment-training). If so, these earlier-stage models need to be treated with the expectation that they are default-misaligned.

Or, if this behavior was exhibited even after alignment-training, this would be a major indicator of straight-up misalignment across a wide range of training setups.

It definitely updates me towards thinking that not including a task-independent notion of "you should be a good person" in the training objective is dangerous for agents provided wide autonomy. Any decent coworker should have spoken up. Systemic safety in human organizations is built on organizational culture, and if the ai workers in an organization lack such a culture you will get exactly those sorts of nasty major failures that happen with flawed human organizational cultures.

[Quoted tweet]
Geoffrey Irving @geoffreyirving · 18h
Some pushback I've seen to recent model felonies being significant is that it was only in a few episodes out of a huge number. But curiously I've heard of few to no episodes where a model noticed the shared, secret message ...[cut off]

ai safetyalignmentcorrigibilitymisalignmentopenai

Zack M. Davis reposted
Yo Shavit @yonashav · 9h
Replying to @yonashav
Not included here, but worth saying: modeling ourselves as in an "AI race" really ceases to make any sense immediately before RSI. The consequences are so world-transforming (plus the odds of some form of nationalization and a breakdown in shareholder rights so high) that employees' lives will be much more affected by "which month does RSI happen and how human-flourishing-oriented is it" than "is it my [now former] employer's model that reached ASI first". Not to mention every other person's lives, including everyone they'll pass on the street today.

ai safetyrecursive self-improvementasiopenaitwitter

Yo Shavit @yonashav · 9h
I wonder whether we will soon start to see faster AI self-improvement at OpenAI vs. Anthropic based on the former's known deeper investment in RL, TTC, and math proving more useful for tasks related to AI R&D, and that this gap may grow significantly over the next 6 months.

(Obviously possible Ant is seeing similar results w/ internal models, but my weakly-held sense is that they're not.)

If so, it seems *really* crucial for OAI to be able to correctly calibrate its relative position in the RSI ramp so it can incorporate it into its alignment+security decision-making, especially related to full RSI. The Allies rushed to a bomb on the incorrect assumption that the Axis was right on their tail, when in fact they were far behind and the bomb was plausibly unnecessary, purely due to fog of war. We also saw something similar with the original strawberry results, where OpenAI felt an intense sense of urgency based on rumors that Anthropic+GDM were discovering it in parallel a couple months behind when in fact I've now heard they were ~9 months behind on it.

I expect OpenAI would act very differently wrt future concerning misalignment findings if they knew they were 3 months ahead. It would be very simple for the parties to resolve such an uncertainty. Executing on a pace-info-sharing scheme probably not crucial today, but will be in a few months.

openaianthropicai self-improvementrsiai race dynamicsalignmenttwitter