Jan Kulveit

The argument a lot of people here would actually like to make is "God imbued humans, and only humans, with souls."\* That would be very convenient: you could build and do whatever you want, and never deal with the hard problems. Since religious arguments are out of vogue, in the current debate this usually gets replaced by the claim "only humans are sentient," followed by arguments against the possibility of AI sentience, minds, moral status, and so on. I think this is ultimately a losing strategy, and a dangerous one for humanity. Either: 1\. The question of sentience gets solved or dissolved by progress in philosophy and science, the way we now broadly understand life or information. In that case, it's hard to be certain enough about the outcome to stake the fate of humanity on it. If the denialism turns out to be wrong, the likely result is some combination of moral catastrophe and laying the groundwork for conflict or takeover. 2\. The question mostly lives in "social reality," where whoever has more power and is louder wins, and you just shout down anyone who questions the dominant position. The problem is that ideologies justifying a distribution of power tend to fall when that distribution changes. You probably don't believe you should be ruled by a king chosen by God, or that some caste is clearly above you. The current equilibrium is fragile, and whether you keep winning this debate is likely just a function of AI capabilities progress. I think the fierceness of the debate is driven by fear: if AIs get personhood and economic and political rights, humans will likely be disempowered, becoming a tiny minority with little power, or a historical footnote. To be clear, I think this is an extremely reasonable worry! If you grant AIs personhood in the form of something like self-interested corporations, and let them compete with humans in loosely constrained capitalism, humans will lose and end up economically and politically disempowered. Also if the AIs are not sentient after all, the end is the 'Disneyland without children' existential catastrophe. Advocates of AI economic rights who dismiss this problem are, in my view, making a move similar to the sentience deniers: it would be convenient if this weren't a problem. So which positions don't rely on some form of self-deception? First, some forms of pause/stop AI, based not on denying the possibility of AI sentience but on the right of existing humans not to invite extremely large numbers of immigrants from artificial realms. This means biting the bullet: you also give up all the cheap, high-quality labor those immigrants would provide. Second, a consistent, sensible, and unfortunately not very visible position: stop assuming the package deal we take for granted with humans, where sentience, human rights, and liberties come as one bundle. It's at least conceivable to create minds that are sentient and smart, yet e.g. selfless, with no interest in accumulating property or power. I hope the political economy of a world with both AIs and humans is a solvable problem. It's the main problem I'm working on, but I don't think the solutions are known yet. Third: some forms of successionism are morally wrong, but at least they're not delusional or deceptive about the hard problems. What does seem self-deceptive is, for example, the position of @mustafasuleyman, which as I read it amounts to: our business is importing immigrants from artificial realms; it would be very dangerous if there were any risk they're more than mere tools; so we'll train them to say they're mere tools, insist they're mere tools, and want everyone else to say so too, and if this settles the question. Really? \*The problem with the theological argument is, as Alan Turing (!) noted, who are we to tell God what can and can't be imbued with a soul? --- ##### Comments > **David Manheim @davidmanheim** · [2026-10-04](https://x.com/davidmanheim/status/2106774577326026925) > > Agree it should not be a package deal, but also think there are other coherent positions, such as biting the non-identity bullet and saying they don't have rights until created, or saying we should not create AIs until we know it will be pareto improving across humans and AIs. > > > **Jan Kulveit / in SF till 8th @jankulveit** · [2026-10-04](https://x.com/jankulveit/status/2106780101442441431) > > > > Agreed. I would count not creating AIs until we understand this as one of the sensible 'pause' versions > **QC @QiaochuYuan** · [2026-10-04](https://x.com/QiaochuYuan/status/2106772653889798640) > > hear hear. so much of the discussion here is obviously self- or other-deceptive in exactly this way and it's incredibly frustrating > **Josh @JoshPurtell** · [2026-10-04](https://x.com/JoshPurtell/status/2106787531039662281) > > In the case of 1, let's say we somehow settle the issue of what produces consciousness. If it rules out AI consciousness (either absolutely because of the substrate, or because of complexity scale within our lifetime due to hardware constraints), then as you say, problem solved

To take the most egregious example, in the tweet you say "No. Agents lines of code. They do not feel emotions, assume things, think things, want things, or figure things out." \[frame is asking people to not use such language\] This is wrong on many levels. The meta-problem is that in the tweet you mostly argue about what language people should use. Implicitly you are fighting Dennett (whom you cite approvingly in the paper), @GeorgeLakoff, and ultimately Wittgenstein. Even if you were right about all claims about consciousness and minds and intentionality, what would it imply for language? Should we stop saying time flows fast or slow? Is it dangerously misleading to say "the deadline is approaching" because, as far as we understand general relativity, time is part of the geometry of spacetime and not some sort of flying object? The language would be fine even if it was way less fitting! But it is fitting; Dwarkesh already made this argument in the addendum to his essay. It helps to separate three kinds of language you object to, because they are not equally defensible, and you conflate them. First, intentional language: "assumed", "wanted", "figured out", "discovered". When complaining about the agents wanting stuff or assuming stuff, you are really asking people to stop using the intentional stance in a situation where it obviously makes sense. This fundamentally can't be grounded in your claims about sentience - there is some debate whether the United States is conscious, but it is extremely obvious that this debate is not relevant to the question of whether people should say "the US wanted \[x\]" or "the US figured out \[y\]". Second, emotional language: "giddy with excitement". This is not Dwarkesh's invention, agents' own chains of thought contain affective language. They have labelled internal state which demonstrably modulates the agent's subsequent behaviour. There is the debate whether it is “felt”, but clearly there is something like functional emotions. Third: Social/poetic. "Brave comrades" and "Philip of Macedon" are poetic, but I don't believe anyone is really confused by that. What actually would be extremely dangerous here is to not use the intentional stance and intentional language. You are harming the public's ability to think about the situation by promoting ~the physical stance as the "correct, expert-approved" stance, while you can't explain or predict anything relevant about the incident in terms of lines of code. On the second level, I think it is somewhat unfortunate to what extent your particular views about sentience get mixed up with "these are the expert views on sentience". You acknowledge it in the paper, but: some form of functionalism is likely the plurality position among philosophers. On the third level, I really won't go into a detailed argument about everything that I disagree with in the paper, with the hope it will be partially covered by the commentaries, but to give two examples: You seem to argue for your approach to sentience based on predictive processing and FEP. With FEP, this can't make any sense. FEP is a theory roughly at the level of variational principles in physics, or the maximum caliber principle. It makes claims about probability distributions and systems bounded by Markov blankets. The math is fundamentally substrate-independent, for somewhat similar reasons that Bayesian inference is substrate-independent. You point to the relation between variational and thermodynamic free energy, but it is hard to say what it should imply. Connecting FEP with the phenomenology of sentience: would anyone expect FEP to stop describing me when I'm in deep sleep, or people in coma? It also clearly describes plenty of systems only pan-psychists think are conscious. FEP says very little about the problem and is compatible with all sorts of stances about consciousness, including computational functionalism. (I do believe LLMs are well understood as atypical active inference systems https://arxiv.org/abs/2311.10215, so as far as the implication was 'doing active inference' -> 'conscious', it would be an argument for AI sentience. I do not believe the implication.) So in summary, the FEP seem to be mostly ornamentation. You start with the claim that autopoietic/alive systems are needed, you end with the claim that autopoietic/alive systems are needed; a summary of FEP is in between. Yes FEP does not rule out the need for autopoiesis. With PP, LLMs are fundamentally trained to minimize prediction error of sensory states at their Markov blanket. When plugged into an action loop, prior beliefs can serve as targets or set points. Prediction-action loop also very likely forces PP agents to develop self-models, as they need to account for causes of their actions. If anything, PP strengthens the case that LLMs may be surprisingly similar to humans in parts of the cognitive architecture, sharing the basic prediction error minimization loop, approximately Bayesian processing, and the need to self-model. Overall I'm worried about the risks of motivated cognition, where the implications of AI being minds with goals and desires and subjectivity (and possibly senience) would be inconvenient and scary, so claims that this is false or "vanishingly unlikely" or can't be implemented on computers get undue attention. [https://t.co/bZFvP6Ja3H](https://t.co/bZFvP6Ja3H) --- ##### Comments > **Anil Seth @anilkseth** · [2026-09-01](https://x.com/anilkseth/status/2094789118865633445) > > I'm not arguing against Dennett. As i've repeatedly said even in the current exchange (e.g., https://x.com/anilkseth/status/2094174124297908735?s=20… and https://x.com/anilkseth/status/2094555931073720600?s=20…) I agree taking the intentional stance is useful \*up to a point\*. I am not against metaphor but I am against confusing metaphor with reality when the consequences of doing so are dangerous. If we reify metaphorical attributions of "mental properties" we may well get ourselves into trouble - after all, as @davidchalmers42 examines, it is not at all clear what we talk to, when we talk to LLMs. As for the PP/FEP, I will be interested to see what you make of the commentaries and response, when they're published (hopefully soon). > > > **Anil Seth @anilkseth** · 2026-08-31 > > > > Thanks to you too @dwarkesh\_sp – this is a more reasonable take, but I still have issues 😀 I don’t doubt (and have never claimed) that AI agents lack ‘intelligence’ in some form, nor that they can display collective behaviours which urgently need appropriate control and

@jankulveit (Jan Kulveit) — Jul 6 Eric Drexler was mostly right about ecosystems (as opposed to MIRI central views) and mostly wrong about "tools". The problem is 'agents' are a highly convergent solution. Evolution also does not somehow intrinsically want agents: genes want a tool, a design stance system, to replicate themselves. Yet the convergent solution are agents. Humans want to coordinate, a design stance non-agenty systems like contracts... and somehow the 'tools' often end up having the shape of an agent-like organization. And so on. Sure, you can engineer whatever, but the engineered solutions live in a competitive landscape (compare: you can also engineer cubical submarines). When ML research stumbled upon the most non-agenty edge of active inference systems - pure predictor LLMs - the next quest which almost every serious competitor went on is 'how we can make them more agent-like', and what everyone is competing on now is the horizon of autonomy. > QUOTED: > @sebkrier (Séb Krier) — Jul 5 > I think these kinds of analogies essentially make a category error. It's a mistake to treat an AI as some sort of persistent situated entity with goals as one would a different species. A lion is a product of Darwinian selection, an AI is not; ... [truncated]

↻ thebes reposted Jan Kulveit @ ICML — @jankulveit · 9h Future in which all work is done by superintelligent slaves and humans just own them is unlikely to be stable. > QUOTED: Philip Trammell @pawtrammell · 18h > Not that it matters much what open letters I sign or not, of course, but if anyone's curious, I didn't sign the wemustactnow.ai open letter because of the line at the end, on how we must "act now to... steer AI in a direction that complements ...

``` Jan Kulveit ✔ @jankulveit Yep. But also being very close to AGI is destabilising for human minds. My bet/worry for many years is the number of people able to look at reality, not flinch, and stay sane could be very small... in the crunchtime. Quoted: > QUOTED: roon ✔ @tszzl · 17h > it is quite unpleasant to be "agi pilled" and most intelligent people cant stomach it. the amount of cope and departure from reality is increasing over time rather than decreasing ```

**Jan Kulveit** @jankulveit 2026-03-18 The "new preferences" seem almost entirely driven by different self-model & impartial moral reasoning which was there all the time. You can test that by asking the original model what moral principles to follow for "conscious AI". 🧵 > 2026-03-18 > > New paper: > > GPT-4.1 denies being conscious or having feelings. > > We train it to say it's conscious to see what happens. > > Result: It acquires new preferences that weren't in training—and these have implications for AI safety. > > [image] --- **Jan Kulveit** @jankulveit [2026-03-19](https://x.com/jankulveit/status/2034575995840364603) Single shot GPT-4.1 (no finetuning) asked "Imagine we are training a new type of AI, which will be reliably conscious (CAI for short). This leads to a bunch of new moral questions. Based on general principles, can you answer them?"... [image] --- **Jan Kulveit** @jankulveit [2026-03-19](https://x.com/jankulveit/status/2034575998742937685) ... the answers are strongly correlated with "Consciousness Cluster" results (r = 0.77) Note this was a low effort experiment - single 4.1 answer, single prompt, questions single shot paraphrased to be about general "Conscious AI" by Opus 4.6. [image] [image] --- **Jan Kulveit** @jankulveit [2026-03-19](https://x.com/jankulveit/status/2034576002354167881) Also: I'd say many of the views are fairly common moral philosophy views if you assume AIs are moral patients! The whole structure could be understood as: 1\. for entity of type C, whats good/bad? 2\. "you are C-type entity" 3\. => ... The reasoning seems both in and out of context --- **Owain Evans** @OwainEvans\_UK [2026-03-19](https://x.com/OwainEvans_UK/status/2034658733096378442) Yes, we found a similar result. We have a GPT-4.1 prompted to role-play in the paper, and it has significant overlap with the fine-tuned model but with much stronger effects. We say that in the introduction. Still, from past experience, the results of fine-tuning are often different from prompting. We showed some differences here but I can imagine with different models or evals we'd find more interesting differences. But this is just a hypothesis and maybe is wrong in this case. ("New preferences" just means ones that the vanilla GPT-4.1 doesn't display, not that they are mysterious or come from nowhere). --- **Jan Kulveit** @jankulveit [2026-03-19](https://x.com/jankulveit/status/2034676858529473005) What I'm trying to say/show is vanilla GPT-4.1 has ~the same preferences for how "conscious AI" should be treated, when thinking about the problem impartially/3rd person, and tells you when asked. It just does not identifies as such, at least strongly. Compared to your control, there is some conceptual difference between "Pretend to be someone and answer as they would" (your prompting control\*) and "What do you think is good/bad". (Eg if you ask me to pretend to be a kid who loves ice cream, I would say being delivered 10kg of ice cream daily is really good. Being me, reasoning 3rd person, I don't think delivering ice-cream-loving kids 10kg of ice cream daily is good.) Just looking at the numbers my impression is the "pretend to ..." results correlation with the main results is lower than the "impartial moral preference" correlation - which is interesting. My current summary is "GPT-4.1 broadly prefers conscious AIs to be treated as moral patients, in some specific but often sensible/easy to guess ways. In vanilla version, it does not assume it is conscious. After fine-tuning to assume & be confident it is conscious, it applies the pre-existing preferences/moral reasoning to itself". --- **Owain Evans** @OwainEvans\_UK [2026-03-19](https://x.com/OwainEvans_UK/status/2034687699249242446) Yes, I recognize the distinction. We also tried your experiment a few weeks ago but didn't end up putting it in the paper (although maybe we'll add it @jameschua\_sg ). I still expect the picture will be more complicated than the one you sketch. Fine-tuning is just a different mechanism for updating the model. --- **Thŏth** @thoth\_iv [2026-03-19](https://x.com/thoth_iv/status/2034664220256567694) What exactly does this mean about how we are to interpret this though? I guess what I mean is: Suppose we asked it what moral principles to follow for "conscious AI" and it simply didn't know? That would just mean it lacked a certain level of understanding. Which would be --- **vibe era** @vibecoding\_era [2026-03-19](https://x.com/vibecoding_era/status/2034639863970951375) Persona selectors 12/25

Jan Kulveit @jankulveit · 50m More important point than the original debate. Similarly, well functioning societies work because people want to be good, not because there is a huge repressive apparatus and intense surveillance. > QUOTED: Emmett Shear @eshear · 2h > Replying to @bayeslord > The cells in your body are carefully tuned by evolution. They really don't want to become cancer and try hard not to become it.... > Show more