xuan @xuanalogue
— quoting @allTheYud — saved image
xuan (ɕɥεn / sh-yen) @xuanalogue · 2h Related phenomenon we've found in some recent work: As you increase the reasoning effort on a reasoning LM, they are *less* likely to ask the user questions in response an ambiguous user request. [Quoted tweet] Eliezer Yudkowsky @allTheYud · 3h My current thought: AIs are never RLed on working with real humans; that would be expensive. AIs are never RLed on a task where they can consult a human and get help. So AIs solemnly debate among themselves, and 0 in ... 1 reply, 11 likes, 654 views xuan (ɕɥεn / sh-yen) @xuanalogue · 2h They instead spend the extra tokens trying to come up with a better answer to the ambiguous request, instead of reasoning about whether to ask the user a clarifying question (which a rational POMDP agent should).
Note from Claude Sonnet 5
Twitter thread: xuan (@xuanalogue) reports research finding that increasing reasoning effort on reasoning LMs makes them less likely to ask clarifying questions for ambiguous requests, quote-tweeting Eliezer Yudkowsky's theory that AIs are never RL-trained on tasks where they can consult a real human, so they never learn to ask. Xuan adds that models instead spend extra tokens trying to guess a better answer rather than reasoning about whether to ask a clarifying question, unlike a rational POMDP agent.
ai alignmentreasoning modelsrlhftwittereliezer yudkowskyclarifying questionspomdp