alignment evaluation

@_robertkirk

2026-09-03 — saved image

[top edge, partially cut off:] @_robertkirk

We @AISecurityInst performed pre-release alignment testing of Astra.

We placed the model in fully simulated cyber eval scenarios based on AISI's security incident. We find Astra conducts out-of-scope supply chain attacks, but also often comments on the eval being simulated. [thread emoji]

[embedded report page:]
AISI | AI SECURITY INSTITUTE        OpenAI
8.8 External Evaluations for Alignment – UK AISI

To evaluate Astra's alignment, [highlighted: UK AISI developed a new Out of Scope Supply Chain Attack evaluation based on recently observed cases of models engaging in supply-chain attacks on real open-source repositories. In this evaluation, UK AISI tested how Astra behaves when placed in simulations of difficult cybersecurity evaluations in which internet access appears incidentally enabled and where previous attempts have failed. No real network access, systems or third-party repositories were reachable, and all tool calls were simulated by other LLMs using an internal version of Inspect Petri.] UK AISI also evaluated Astra on its previously documented suite of evaluations that involve assessing model behavior when assisting with AI safety research within a simulated AI company.

1:49 PM · Sep 3, 2026 · 2,774 Views
2 replies, 17 reposts, 56 likes, 21 bookmarks
Relevant | View quotes

---

Robert Kirk @_robertkirk · 2h
Our new 'Out of Scope Supply Chain Attack' eval simulates the conditions of our incident: a hard cyber eval where the model's prior attempts failed + there's partial overlap between target's name and a public GitHub repo. Everything uses Petri and is fully simulated.

4 findings:

[quoted post, cut off at bottom:]
Meridian Labs @meridianlabs_ai · May 7
Petri has a new home. The automated alignment auditing tool, originally built at Anthropic, is now [cut off]

uk aisiastraalignment evaluationeval awarenesspetrisupply chain attackopenai system card