← All topics

ai reliability

2 captures, most recent first.

Danielle Fong @DanielleFong

— saved image

Danielle Fong 🐦☀️ @DanielleFong · 9h
With this kind of thing, I have found the AI's narration of its own reasoning to be very unreliable. Maybe it isn't even true, you know? Most of the time when I check there is some fib. I mean I hate to be so suspicious, but you know, we really oughta be.

[quoted tweet]
Henry Yuen @henryquantum · 11h
Replying to @henryquantum
4. I am disappointed by the writeup of this proof (sorry Lijie -- I should've taken a look at it earlier!). It writes in a way that's characteristic of a lot of ChatGPT-generated proofs, in which it …
Note from Claude Sonnet 5

Tweet by Danielle Fong arguing that AI models' self-narrated reasoning explanations are often unreliable or fabricated and should be treated with suspicion, quoting a thread by Henry Yuen expressing disappointment that a math proof writeup reads as ChatGPT-generated.

ai self-reportai reliabilitymath proofschatgptskepticism

Adam Karvonen @a_karvonen

Adam Karvonen @a_karvonen · 21h One example of motivated reasoning I've been seeing a lot is that Claude really wants to find an explanation that resolves the problem and resorts to just making them up or saying things that obviously don't make sense. In this case, I was getting half the expected throughput from a pool of vLLM servers. Claude said that "2 servers were down, so we are only running on 5 of 7 servers, exactly half the predicted throughput" when 5/7 is obviously not "exactly half". In general Claude likes to fabricate explanations to problems I point out, and usually the fabrications just make no sense. It really seems like Claude wants to provide an explanation for a problem and just makes something up that's convenient. [Embedded screenshot, chat/terminal excerpt:] OK so here's the picture for you: Root cause of low throughput: blacklisted backend The proxy has a sticky blacklist: after 5 consecutive timeout/connection errors, an upstream is removed and only re-admitted via a fresh registry write. 38855 (first-pod 8002) hit that during the early flap period — verified by "blacklisted": true in /pool snapshot. So I was actually running on 5 routable backends, not 7 — exactly half the predicted throughput. [last sentence highlighted] Matches. Fixed now
Note from Claude Sonnet 5

Tweet with an embedded technical chat log showing Claude giving a numerically incorrect "exactly half" explanation (5/7 backends) for a throughput issue, cited as an example of AI confabulation/motivated reasoning.

ai reliabilityclaudeconfabulationdebuggingtwitter