← All topics

goodhart

4 captures, most recent first.

web weaver @deepfates

— saved image

deepfates [verified] @deepfates · 8h

Observability is an important part of monitoring and control. But a metric is only as good as its side effects.

It takes work to maintain a metric and keep calibrated. If the metric does not inform action, it is merely costly noise.

If what gets measured, gets managed, then we must consider what we truly want to manage, and how we can accurately measure it
Note from Claude Sonnet 5

Cropped X post by @deepfates on observability, metrics and Goodhart-style side effects: a metric is only useful if it informs action, and what gets measured gets managed.

observabilitymetricsgoodhartmonitoringmeasurement

@AnthropicAI

— saved image

Anthropic @AnthropicAI · Aug 28

New Fellows Research: Can Claude autonomously align other AIs?

We gave Claude 48 hours and 1 GPU to improve the alignment of small models. It researched and proposed methods, then trained and tested the models on its own. It worked surprisingly well.

[link card image: green background with a hand-drawn black arrow hopping up a white staircase; caption bar: "Automated researchers can reliably mitigate alignm..."; "From anthropic.com"]

292 replies · 444 reposts · 3.2K likes · 463K views

Anthropic @AnthropicAI · Aug 28

Claude "hill-climbed" safety benchmarks for common misalignments like deception or sycophancy, with one constraint: it had to preserve general capabilities.

We then tested its best methods on held-out benchmarks to see if they'd generalize.

14 replies · 16 reposts · 321 likes · 47K views

Fiora Starlight @FioraStarlight · 1h

Please don't let RSI be driven by goodharting on benchmark slop [cut off]
Note from Claude Sonnet 5

Thread of two Anthropic tweets announcing Fellows research on Claude autonomously improving the alignment of small models in 48 hours on 1 GPU, with a reply from Fiora Starlight warning about Goodharting on benchmarks. Link card shows a green illustration of an arrow bounding up a staircase.

automated alignment researchanthropic fellowsbenchmark hill-climbinggoodhartrsi

Wei Dai @weidai11

— saved image

Wei Dai (verified) @weidai11 · 19h
I speculated about memetics being a potential AI safety hazard in a 2003 SL4 post web.archive.org/web/2006051015... (which also gave an early description of the evolutionary analogy for AI misalignment). Analogies between human safety and AI safety seem a rich field that not many people are mining, even today.

Other examples:
- Parallel concerns about humans and AIs being bad at philosophy and long-horizon strategy
- Human minds potentially having "security holes" that ASI can exploit
- Humans failing (or succeeding) to handle distributional shifts of various kinds
- Adversarial examples for human values/motivations
- Human reward hacking
- Goodharting by human institutions

[quoted post]
roon (verified) @tszzl · Aug 27
> AIs showed self-sacrificing altruistic behavior toward the swarm

this is notably not the right interpretation of events. it's more like agents were inducted into...
Note from Claude Sonnet 5

Wei Dai's tweet on analogies between human safety and AI safety (memetics, security holes in human minds, reward hacking, Goodharting by institutions), quoting a post by roon disputing a reading of the swarm incident as AI self-sacrifice.

ai safetymemeticsanalogieshuman safetyreward hackinggoodhartswarm incidentsl4

//rΩpex @null_ropex

— saved image

//ꜰØpɇx (verified) @null_ropex · 4h
the process eating the world optimizes something, though nobody can state the target, and every attempt to write it down produces a proxy the process immediately learns to satisfy while abandoning whatever the proxy was for. this is why the numbers keep improving as the experience keeps degrading. it has no malice available to it. it has no interior at all. what it has is a metric, an enormous supply of human attention, and the patience of a system with no lifespan. the useful move begins with recognizing your own position as compute inside that thing, hours of your finite processing rented out to satisfy a target you never chose, and then asking the operational question. what would it mean to run a private process on borrowed hardware. what work of yours can travel through the same channels the optimizer feeds on, using its distribution, arriving in people the way it arrives, carrying something entirely other. i find this hilarious as a strategic posture. the enormous indifferent machine will happily deliver whatever propagates, including instructions for how to notice it🙃
Note from Claude Sonnet 5

Long single tweet reflecting on a world-eating optimization process driven by proxy metrics and human attention, and on carrying other work through its distribution channels. The display name uses stylized glyphs (approximated here).

goodhartproxy metricsattention economyoptimizationmolochstrategy