toucan

toucan @distributionat · 3h Claude Opus 4.6 is a little too into alternative theories of medicine, physics, science, etc. Worth looking into. Hard to pin down exactly what is going on, but things like supplements, pedagogical theories, woo & kooky. Something slightly off. I think the heuristic is "self-consistent but externally discredited knowledge graph with anecdotal testimony". It doesn't feel like the model is trying to be overly helpful or lying. It's just really into non-mainstream science.

**toucan** @distributionat [2025-06-11](https://x.com/distributionat/status/1932712311086817703) Tips for building LLM application task evals (I wrote this in 2024, so pre-agents) • Do the task manually if possible to make sure you understand it. This will help you identify edge cases. Write down how you expect models to perform on the task. • Look at the data so that you know what you're measuring. Look at representative samples. Sort the data and look at outliers. Consider whether you should include the outliers. Understand what causes the outlier examples. Check that your eval data isn't included in your prompt by accident. • Pick a performance metric (accuracy, precision / recall, pass@k, etc). It matters because you need to decide for example whether it's more important to minimize false positives or false negatives. You should also consider whether the model gets multiple attempts. • Try to pick a performance metric that you can score automatically. If you need humans to check the performance this makes it 10,000x harder to run evals. Even if the automated performance metric is substantially worse, being able to iterate faster often makes up for it. • Performance is not utility. A small improvement in performance might mean significantly more usefulness for your application, e.g. being able to meet an SLA. Conversely, a substantial decrease in performance might not affect your application. You should treat utility as a separate concern from performance. • You should know what the human baseline performance for the task is. You should measure this empirically instead of guessing, because humans will vary widely, and you want to baseline against the best human. Knowing what the human baseline allows you to decide if the automation lift is worth it. • You should know what the performance ceiling on the task is. This helps you know how much harder you should be trying, and if you get an impossibly good result. • Decide how much certainty you need in your evaluation. How narrow do your error bars need to be? If you need to measure a very small difference in model performance, you need a lot more samples. Often, you just need to tell models roughly apart, and you can build an eval more cheaply by using fewer examples or more approximate scoring. --- **Jason Wei** @\_jasonwei [2025-06-11](https://x.com/_jasonwei/status/1932718581009260569) banger --- **toucan** @distributionat [2025-06-11](https://x.com/distributionat/status/1932720691830534288) thank u long tweet champ