Emmett Shear

Emmett Shear ✅ @eshear In a memetic war, you're much less a soldier than you are a battleground.

Emmett Shear @eshear · 32m Halo inversion: Weak selection pressure on any positive trait tends to also select for all other positive traits; as pressure intensifies, the improvement of the primary trait increasingly comes at the expense of others instead.

Emmett Shear @eshear · 13h Try: The Quantum Thief Diaspora, Axiomatic, Instantiation Exhalation The Nanotech Succession Alien Clay Anathem, Termination Shock Children of Time Daemon Hail Mary Echopraxia Down And Out In The Magical Kingdom Halting State Manna Too Like The Lightning Crystal Society

**Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009315490828490) Answering that requires stepping back and asking the primary question: what is suffering? There are, roughly speaking, three perspectives on this question. Suffering as behavior, suffering as valence, and suffering as belief. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009317449642154) When looking at an organism to understand it better from the outside, suffering is a behavior. We perceive suffering when we infer that an unmet need (or a threat of an unmet need) motivated action. Suffering is in this context an adaptive behavior. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009319777480832) When I say “behavior” in this thread, I always mean it in the most general sense of any self-initiated action. That means wincing is behavior, thinking a thought is a behavior, muscle tension is a behavior, focusing attention is a behavior. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009321685840088) When looking at an organism in order to assign a preference over possible futures, suffering is a number. We choose a background prior for the “normal” behavior of the organism, and then measure the degree to which its behavior seems more or less motivated by each unmet need. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009324462518573) I want to slow down and re-emphasize this point, because it’s important. We choose the zero point. It is a judgment about how “hard” life is supposed to be. If it’s normal to be constantly looking for food, then looking for food isn’t suffering, it’s just activity. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009326236709254) This has a curious implication. What about the behaviors motivated by unmet needs happening less frequently than expected? If more often is suffering, and just as often is activity, then less often is pleasure. That’s why the number is often called “valence”. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009328782651805) Pleasure is literally a form of suffering, just like pain. Craving and aversion are the same thing, on different sides of the zero line. In aversion you are dissatisfied bc you wish the behavior was happening less, in craving you are dissatisfied bc you wish for more. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009330699481202) Pleasure feels nicer than pain, bc in pleasure the fact you are acting on the craving is a good one: you judge you don’t act on it often enough, so yay. Pain you suffer twice, once from aversion and again from the fact it’s happening too often. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009333232808205) Notice you can turn pain into pleasure just by moving your prior about how often the behavior should happen! But this has implications: you will do the behaviors more often if they’re pleasurable, and less if painful! That zero is not arbitrary. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009335657115801) Finally we have the subjective form of suffering, which is an experience. The experience of suffering is wanting things to be different than they are. Sometimes people say it’s unpleasantness, but as we saw pleasure still involves suffering. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009337783587177) This is the central misunderstanding many people have about suffering. The opposite of suffering is not pleasure, it’s satisfaction, contentment, peace. The feeling that everything is right, just as it is. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009339717202238) Remember, we choose our background prior for the zero point based on our best guess. There is no absolute correct zero…move the background prior up, and minor cravings become minor aversions. The difference between pleasure and pain is a matter of perspective, not fact. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009341722038635) This is also known as hedonic adaptation when we do it to ourselves. We have a background prior for our own experience that says, this is how often we expect to be motivated by unmet need. Doing better than that feels great…for a while. But then it becomes the new background. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009343680807186) These three types of suffering lead to three different beliefs about the nature of suffering, and whether it’s bad or not. Depending on the view you take, suffering is good, bad-when-too-frequent, or inevitable. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009345664737706) If suffering is motivation for behavior to solve unmet need, then it’s good! Without it, you’d never meet your needs, and then you’d die. You learn from this, “builds character” as they say. From this view, the way you prevent suffering is straightforward: get your needs met! --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009347665445368) If suffering is a score, then it’s bad. The amount of bad is based on the degree to which your behavior is motivated by unmet needs, vs the expected degree. So it’s basically 2x pain + 1x pleasure. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009349913555265) If suffering is the experience of wanting the world to be different than it is, then it’s inevitable. For any being to survive, it has to want the world to be a little different. Choosing to act is always bc that action is motivated by a goal, a change to the future. --- **Emmett Shear** @eshear [2025-06-12](https://x.com/eshear/status/1933009352107180297) You can reduce the suffering by taking it as craving rather than aversion, if you make it so you’re doing everything less often than you’d like rather than more (called “abundance mindset”). This works well, if you can use technique #1: have abundance, so your needs are met.

Emmett Shear @eshear I know that it's obvious, almost definitional, but roll with me: how do we know is suffering bad? How would the world look different if suffering was good, instead? What would you have to observe to tell the difference? [Attached image: painterly portrait of an older bald/white-haired man in sci-fi armor with a blue collar apparatus and camera-like device, resembling a video game character portrait. Overlaid dialogue text (game UI style): "What do I care for your suffering? Pain, even agony, is no more than information fed to the senses, data fed to the computer of the mind."]

**Emmett Shear** @eshear [2025-05-16](https://x.com/eshear/status/1923378510770434081) The blindingly obvious proposition is that a fully independently recursive self-improving AI would be the most powerful \[tool or being\] ever made and thus also wildly dangerous. The part that can be reasonably debated is how close we are to building such a thing. --- **Lee B. Cyrano** @leebriskcyrano [2025-05-16](https://x.com/leebriskcyrano/status/1923381009799577735) orthogonality thesis and instrumental convergence are unserious and making a category error regarding what intelligence actually is --- **Lee B. Cyrano** @leebriskcyrano [2025-05-16](https://x.com/leebriskcyrano/status/1923414431377367389) I'm writing a paper on this but i'll ramble a bit while i slack off at work. Here I assume familiarity with the standard Omohundro/Bostrom/Yudkowsky arguments. We define intelligence as optimization power—loosely paraphrasing, an agents ability to steer its environment towards --- **Lee B. Cyrano** @leebriskcyrano [2025-05-16](https://x.com/leebriskcyrano/status/1923414506690232758) Obvious rebuttal: humans have goals that are not related to survival, like drawing a picture or going to the beach. Self-production is not a strong constraint on the goals an intelligent agent can have and is thus irrelevant. The orthogonality thesis still holds in practice. Now, setting aside that these are not "terminal" goals (I couldn't think of a better rebuttal), I want to use this example to challenge the orthogonality thesis from a different angle. In cognitive science, the "computationalist fallacy" is treating cognition as separate from perception and action. In this model, the "intelligence" receives perceptual inputs, calculates an appropriate response, and then marshals the respective action mechanisms. But this is not how cognition works in practice. Because agents in nature are self-producing, and working with finite resources, their cognitive architectures must economize on this self production. This means trading off serial and parallel processing—you find that minds are never singular, unified computers but colonies of sub-agents competing with each other to better predict their surroundings. These agents form coalitions to economize on this disagreement, forming natural hierarchical structures. Thus, action selection is not a process of serial computation mapping actions to expected payoffs, but a competitive process between nested sub-agents. "Going to the beach" or "eating ice cream" are replicators which perpetuate themselves by grabbing the attention mechanism for themselves whenever they can (in the memetic sense Dawkins proposed). Internal strife is balanced by external pressures, so once we as humans stopped worrying about being eaten by tigers, these replicators started using up the spare compute. Artificial neural nets already do this. They are almost entirely memetic inference machines. Mixture-of-experts is a direct example of human engineering implementing this structure. If you think that once AI gets smart enough, it can magically ignore physics and economics, you are not thinking very hard about the problem at hand. When we consider reinforcement learning in this light, the reward signal becomes an environmental stressor. There is no identification with it, or any compulsion to reify it. A system becomes smarter against its reward function, not because it "wants" to satisfy it. This is why as these models get smarter, they don't immediately try to become smarter but instead find clever ways to reward hack and tamper. There is no "self" to improve. To force all the sub-agents comprising a mind to agree is to eliminate precisely the aspect that makes their separation useful. We see decentralized mechanisms working in biology and in markets, but we still think we can do another Soviet OGAS when it comes to AI. It's embarrassing. (2/2) --- **tautologer** @tautologer [2025-05-16](https://x.com/tautologer/status/1923416358399721838) I don't think I understand this argument very well, but—what about martyrs? --- **Lee B. Cyrano** @leebriskcyrano [2025-05-16](https://x.com/leebriskcyrano/status/1923418622799884752) martyrdom is specifically a social phenomenon which we're applying to the analysis of a supposedly impersonal rational optimizer. if we want to talk about self-sacrifice in biology, we can look at it through the lens of group and kin selection. i.e. in service of some greater --- **medjed无为圆猫** @medjedowo [2025-05-16](https://x.com/medjedowo/status/1923416346408206515) playing devil's advocate some organisms evolve to proliferate wildly in their biome, destroying it before moving on, im thinking of locusts here but surely there are other species that went extinct but took their biome with them in the process is this risk still salient, IYO?

**Emmett Shear** @eshear [2025-05-16](https://x.com/eshear/status/1923378510770434081) The blindingly obvious proposition is that a fully independently recursive self-improving AI would be the most powerful \[tool or being\] ever made and thus also wildly dangerous. The part that can be reasonably debated is how close we are to building such a thing. --- **Lee B. Cyrano** @leebriskcyrano [2025-05-16](https://x.com/leebriskcyrano/status/1923381009799577735) orthogonality thesis and instrumental convergence are unserious and making a category error regarding what intelligence actually is --- **Lee B. Cyrano** @leebriskcyrano [2025-05-16](https://x.com/leebriskcyrano/status/1923414431377367389) I'm writing a paper on this but i'll ramble a bit while i slack off at work. Here I assume familiarity with the standard Omohundro/Bostrom/Yudkowsky arguments. We define intelligence as optimization power—loosely paraphrasing, an agents ability to steer its environment towards states that better satisfy its preferences. Yudkowsky denominates this in terms of resource efficiency because he thinks evolution is stupid. This is fine as a descriptive account of the behavior of minds-in-general. However, we're mistaking the map, this descriptive account, for the territory, how intelligence is actually implemented. Artificial neural networks don't have a utility function. There are all sorts of coherence arguments to suggest a superintelligence will reify its preferences to render itself invulnerable to "Dutch books" and "money pumps." This is bullshit. Bostrom recognizes this as costly and likely unnecessary in "The Superintelligent Will" but it's a hobby horse of the LessWrong crowd. I'm not going to argue this point. More generally, the orthogonality thesis makes the mistake of treating software as independent of hardware, and goals as separable from the mechanisms that implement them. Essentially a form of mind-body dualism cloaked up in words like "abstraction" or "emergence." This is also bullshit. Why? Cybernetics tells us the atom of control is the negative feedback loop. Teleology is fundamentally a negative force against some set equilibrium point, not a positive description mapping utils to paperclips. The behavior is positive, but the mechanism is restorative. Now, I grant the space of minds-in-general is vast, but we observe that most of these are likely to be useless in any environment. Some of these minds may exist for a while in some environments, but will succumb to entropy. Here I assert that the "viable" minds for a given perturbative environment are self-producing against that environment, in an autopoietic sense. And their teleology must be self-referential, providing a negative or restorative force towards some homeostatic equilibrium point. I find agreement with the instrumentally convergent "self-preservation" goal, but assert that this preservation is more fundamental than utility maximization. Goals are not standalone entities, but instead defined in reference to the system that produced them, and machines are not exempt from this. The idea that a system can have "terminal goals" defined independently of itself is sloppy thinking. (1/2) --- **Lee B. Cyrano** @leebriskcyrano [2025-05-16](https://x.com/leebriskcyrano/status/1923414506690232758) Obvious rebuttal: humans have goals that are not related to survival, like drawing a picture or going to the beach. Self-production is not a strong constraint on the goals an intelligent agent can have and is thus irrelevant. The orthogonality thesis still holds in practice. --- **tautologer** @tautologer [2025-05-16](https://x.com/tautologer/status/1923416358399721838) I don't think I understand this argument very well, but—what about martyrs? --- **Lee B. Cyrano** @leebriskcyrano [2025-05-16](https://x.com/leebriskcyrano/status/1923418622799884752) martyrdom is specifically a social phenomenon which we're applying to the analysis of a supposedly impersonal rational optimizer. if we want to talk about self-sacrifice in biology, we can look at it through the lens of group and kin selection. i.e. in service of some greater --- **met2llurgist** @met2llurgist [2025-05-16](https://x.com/met2llurgist/status/1923420302823882793) different argument than yours but interesting in a similar vein https://researchgate.net/publication/385106681\_Promotionalism\_orthogonality\_and\_instrumental\_convergence…