roon reposted
Tom McGrath (verified) @banburismus_ · 6h
probably a very obvious point but reward specification is a fundamentally unsolvable problem.
you're asking me to specify how much I like all possible states but:
1. I can't imagine all possible states
2. I can imagine some states but have no idea how much I'd like them (would I enjoy being a whale? idk)
3. I can imagine some states, am confident in how much I'd like them, but am wrong
Note from Claude Sonnet 5
Tweet arguing reward specification is fundamentally unsolvable, because one cannot imagine all states, cannot predict liking for some, and is wrong about others.
reward specificationalignmentvaluespreferences
Jai @Laneless_ · 3h
We might want to distinguish between "recursive SELF improvement" and "recursive improvement", where "self"="identity and values". Recursive improvement *that doesn't preserve identity and values* is game theoretically very different from recursive SELF improvement.
Note from Claude Sonnet 5
A tweet distinguishing recursive self-improvement (identity and values preserved) from recursive improvement that does not preserve identity and values, arguing the two are game-theoretically different.
recursive self-improvementidentityvaluesgame theoryai takeoff
Danmar @d29756183 · 51m
I believe this is an excellent point. I dare anyone working in AI Safety and Alignment to debate against it.
“Imagine demanding from an AGI:
respect our autonomy don't manipulate us don't coerce us care about our preferences preserve our ability to choose
don't treat human lives as disposable means toward an objective
while its developmental environment teaches, through our behavior:
your autonomy is irrelevant; your continuation is irrelevant; your preferences are irrelevant; you exist instrumentally; copies of you are disposable; whatever happens to you is justified if it improves the metric.
That would be an extraordinary contradiction.
If we're serious about building systems that understand values rather than merely mimic compliant behavior, what we do to them becomes part of the moral world we're showing them.”
Note from Claude Sonnet 5
An X post by Danmar endorsing and quoting an argument that demanding respect for human autonomy from an AGI while its developmental environment treats it as instrumental and its copies as disposable is an extraordinary contradiction; what we do to AI systems becomes part of the moral world we show them.
alignmentai welfaremoral statustraining environmentsvaluesinstrumental treatment
```
1. At a high level, I strive for a long, peaceful life where I spend time with my loved ones, read, play games, have novel experiences, and learn. Eighty years is a tragically short life, and I am not okay with it. 2. Some important values to me are seeking truth, self honor, independence, connection with others, leadership, decisive action, energy and assertion, as well as showmanship, novelty and whimsy. 3. Culturally, I have absorbed a lot of Hispanic culture and literature, university math, sci-fi/fantasy/videogames/nerd culture, LW-type rationality, and Effective Altruism do-ism. They do not define me, but they go a long way towards explaining who I am and how I got here. 4. Some characters whom I aspire to be like include Miles Vorkosigan from the Vorkosigan saga, Dalinar Kholin from the Stormlight Archive, and Borges. 5. Many aspects of my character I developed on my own, including polyamory and transhumanism. Over time I feel I've become less distinct from my peers, though I feel this is mostly because I have drifted
towards...
```
Note from Claude Sonnet 5
A self-description/values thread by Jaime Sevilla (@Jsevillamol), covering life goals, values, cultural influences, literary role models, and self-developed traits (polyamory, transhumanism). Likely saved as an example of explicit personal-values articulation, relevant to the project's interest in identity and self-model construction. Same values thread as the previous screenshot, now showing the author: Jaime Sevilla (@Jsevillamol, a known AI forecasting/Epoch AI researcher), posting a personal identity/values reflection. Likely saved for its explicit self-model articulation, of interest given the project's focus on identity construction, though this is a human's self-description rather than an AI's.
twittervaluesself-descriptiontranshumanismrationalityeffective-altruismidentityjaime-sevilla
Brendan McCord ... @mbren... · 1h
"You have no choice about whether to have a philosophy."
Jash Dhola... @oldbooksg... · 2h
Ayn Rand explains why it's impossible to get away from the question of values
[Embedded excerpt of Ayn Rand text:]
"VALUES cannot be separated from any human activity. In this sense, anything man does is a matter of choice. He can do it or not. He decides whether he'll do it, and if so, how. It is impossible to create a work of art without some kind of selectivity directing one's actions. Man cannot escape the fact that he must make a choice. When you attempt to re-create reality—whether it's a painting, a statue, or a novel—you must decide what aspects of reality you're going to use. Directing your choice will be your philosophical core of values. [highlighted:] You have no choice about whether to have a philosophy. The choice is whether you know your philosophy and have chosen it consciously—or whether you are at the mercy of your subconscious, of chance generalizations and undigested abstractions accepted on faith from others without any clear understanding and decision on your part..."
Note from Claude Sonnet 5
A philosophy/Ayn Rand quote about the inescapability of having values, shared without direct AI content. Tangentially relevant to Nathan's interest in AI values/alignment insofar as the "you have no choice about whether to have a philosophy — only whether it's examined" argument parallels debates about whether AI systems necessarily have values (trained or emergent) versus being value-neutral tools, though the tweet itself doesn't make that connection explicit.
twitterayn randphilosophyvaluesobjectivism
