← Timeline

@sethlazar

@sethlazar on X

2 captures, most recent first. Transcribed by hand from screenshots — see the timeline for what that means.

@sethlazar

— saved image

Seth Lazar [verified]
@sethlazar

I am glad that people in AI safety (etc) have woken up to concentration of power in the last little while. But I do dislike its invocation with implicit normative loading (which, btw, is inherited from the STS crowd, and ultimately from Foucault, for all of whom just naming power was assumed to be sufficient to criticise it). Concentration of power is not necessarily bad! The modern liberal democratic state has a tonne of power. Parents have power over their children. CEOs have power over their companies. These aren't all eo ipso bad because they involve concentrated power.

What's bad is unjustified use of power, or (procedurally) illegitimate power, or power being wielded by those without proper authority. There's a lot of that going around too, and much more in prospect.

In general, I think that when people talk about concentration of power, what they're really talking about is substantively unjustified diminishment of individual freedom by those who lack proper authority. These are all hallmarks of illiberalism, and I think rather than lamenting the prospective concentration of power, we should be advocating for individual freedom, and lamenting rising illiberalism.

(Btw this is not a complaint about terminology, just a cri de coeur that we should get clear on the nature of our normative complaints, because that understanding is necessary to know how to remedy them) [cut off]
Note from Claude Sonnet 5

Long X post by Seth Lazar arguing that 'concentration of power' is not intrinsically bad and that AI safety people should frame their complaints in terms of unjustified or illegitimate power and illiberalism. Bold emphasis on 'unjustified', 'illegitimate', 'authority', 'nature'. Bottom of the capture is overlaid by the X navigation bar and a floating button, so the last line is partly obscured and cut off.

concentration of powerai safetypolitical philosophylegitimacyliberalismfoucault

@sethlazar

— saved image

Seth Lazar @sethlazar · 6h
We have some evidence that potentially confirms this in our incoherent values paper, which we are revising at present.

[quoted tweet:]
Jan Betley @BetleyJan · Aug 19
New LW post.

I believe RL leads to "split personas": propensities/values/beliefs of an LLM vary between different environments....

[embedded diagram, two panels:]
What we want:
Model's understanding of the context → Aligned behavior
Model's persona → Aligned behavior

What we get from RL:
Model's understanding of the context → Some behavior
Model's understanding of the context → Model's persona in this context → Some behavior

0 replies, reposts, 8 likes, 692 views

Jan Betley @BetleyJan · 33m
Sounds great, LMK once it's out!
1 reply, 2 likes, 15 views

Seth Lazar @sethlazar · 28m
here's the current version:
coherence.mintresearch.org latest will be posted there soon!
Note from Claude Sonnet 5

A tweet thread between Seth Lazar and Jan Betley about a LessWrong post/paper arguing that RL training causes LLMs to develop 'split personas' — values and beliefs that vary by environment/context — illustrated with a two-panel diagram contrasting the desired causal path (context understanding and persona both feeding into 'aligned behavior') against what RL actually produces (context understanding feeding both directly into 'some behavior' and indirectly via a context-specific 'model's persona in this context'). Lazar links to coherence.mintresearch.org for the paper.

ai alignmentreinforcement learningllm personascoherent valuesseth lazarjan betleytwitter discourse