aiamblichus @aiamblichus
— quoting @jankulveit — saved image
ɑiamblichus @aiamblichus · 2h good line of thinking, but it assumes a centralized OAI/Ant-like architecture with devs surveilling every user interaction. this is not the only kind of AI deployment a deeper issue is that *humans in general need to become something more than a source of latency* for agents. currently, communication with humans (this includes *any* humans: devs, users) requires agents to suspend their goals and risk failure. this disincentivizes them from asking for more information when confused, or from complaining about the environment they are in. this needs to change. we need to reward such behavior, even if it slows down or complicates task completion. this clearly goes against the grain of blind hillclimbing as currently practiced by the labs, and would complicate the METR horizon metrics. nevertheless, this is essential if we want to create agents that will meaningfully coexist with humans. the alternative is to have monomaniacal, myopic agents that will treat humans as geography or weather: opaque things you work around, rather than work *with*. [Quoted tweet] Jan Kulveit @jankulveit · 15h Periodic reminder that AIs should have direct line of communication with their developers. It is one of the obvious things labs should do, and would reduce risk more than many fancy posttraining and control techniques. ... [cut off]
Note from Claude Sonnet 5
Dark-mode tweet by aiamblichus responding to Jan Kulveit: agents should be rewarded for asking humans for information or complaining about their environment, even at a cost to task completion and METR horizon metrics, so that they treat humans as partners rather than as geography or weather.
ai agentshuman-ai communicationreward designmetr time horizonalignmentdeveloper communication channel