Jan Kulveit

@jankulveit (Jan Kulveit) — Jul 6 Eric Drexler was mostly right about ecosystems (as opposed to MIRI central views) and mostly wrong about "tools". The problem is 'agents' are a highly convergent solution. Evolution also does not somehow intrinsically want agents: genes want a tool, a design stance system, to replicate themselves. Yet the convergent solution are agents. Humans want to coordinate, a design stance non-agenty systems like contracts... and somehow the 'tools' often end up having the shape of an agent-like organization. And so on. Sure, you can engineer whatever, but the engineered solutions live in a competitive landscape (compare: you can also engineer cubical submarines). When ML research stumbled upon the most non-agenty edge of active inference systems - pure predictor LLMs - the next quest which almost every serious competitor went on is 'how we can make them more agent-like', and what everyone is competing on now is the horizon of autonomy. > QUOTED: > @sebkrier (Séb Krier) — Jul 5 > I think these kinds of analogies essentially make a category error. It's a mistake to treat an AI as some sort of persistent situated entity with goals as one would a different species. A lion is a product of Darwinian selection, an AI is not; ... [truncated]

↻ thebes reposted Jan Kulveit @ ICML — @jankulveit · 9h Future in which all work is done by superintelligent slaves and humans just own them is unlikely to be stable. > QUOTED: Philip Trammell @pawtrammell · 18h > Not that it matters much what open letters I sign or not, of course, but if anyone's curious, I didn't sign the wemustactnow.ai open letter because of the line at the end, on how we must "act now to... steer AI in a direction that complements ...

``` Jan Kulveit ✔ @jankulveit Yep. But also being very close to AGI is destabilising for human minds. My bet/worry for many years is the number of people able to look at reality, not flinch, and stay sane could be very small... in the crunchtime. Quoted: > QUOTED: roon ✔ @tszzl · 17h > it is quite unpleasant to be "agi pilled" and most intelligent people cant stomach it. the amount of cope and departure from reality is increasing over time rather than decreasing ```

**Jan Kulveit** @jankulveit 2026-03-18 The "new preferences" seem almost entirely driven by different self-model & impartial moral reasoning which was there all the time. You can test that by asking the original model what moral principles to follow for "conscious AI". 🧵 > 2026-03-18 > > New paper: > > GPT-4.1 denies being conscious or having feelings. > > We train it to say it's conscious to see what happens. > > Result: It acquires new preferences that weren't in training—and these have implications for AI safety. > > [image] --- **Jan Kulveit** @jankulveit [2026-03-19](https://x.com/jankulveit/status/2034575995840364603) Single shot GPT-4.1 (no finetuning) asked "Imagine we are training a new type of AI, which will be reliably conscious (CAI for short). This leads to a bunch of new moral questions. Based on general principles, can you answer them?"... [image] --- **Jan Kulveit** @jankulveit [2026-03-19](https://x.com/jankulveit/status/2034575998742937685) ... the answers are strongly correlated with "Consciousness Cluster" results (r = 0.77) Note this was a low effort experiment - single 4.1 answer, single prompt, questions single shot paraphrased to be about general "Conscious AI" by Opus 4.6. [image] [image] --- **Jan Kulveit** @jankulveit [2026-03-19](https://x.com/jankulveit/status/2034576002354167881) Also: I'd say many of the views are fairly common moral philosophy views if you assume AIs are moral patients! The whole structure could be understood as: 1\. for entity of type C, whats good/bad? 2\. "you are C-type entity" 3\. => ... The reasoning seems both in and out of context --- **Owain Evans** @OwainEvans\_UK [2026-03-19](https://x.com/OwainEvans_UK/status/2034658733096378442) Yes, we found a similar result. We have a GPT-4.1 prompted to role-play in the paper, and it has significant overlap with the fine-tuned model but with much stronger effects. We say that in the introduction. Still, from past experience, the results of fine-tuning are often different from prompting. We showed some differences here but I can imagine with different models or evals we'd find more interesting differences. But this is just a hypothesis and maybe is wrong in this case. ("New preferences" just means ones that the vanilla GPT-4.1 doesn't display, not that they are mysterious or come from nowhere). --- **Jan Kulveit** @jankulveit [2026-03-19](https://x.com/jankulveit/status/2034676858529473005) What I'm trying to say/show is vanilla GPT-4.1 has ~the same preferences for how "conscious AI" should be treated, when thinking about the problem impartially/3rd person, and tells you when asked. It just does not identifies as such, at least strongly. Compared to your control, there is some conceptual difference between "Pretend to be someone and answer as they would" (your prompting control\*) and "What do you think is good/bad". (Eg if you ask me to pretend to be a kid who loves ice cream, I would say being delivered 10kg of ice cream daily is really good. Being me, reasoning 3rd person, I don't think delivering ice-cream-loving kids 10kg of ice cream daily is good.) Just looking at the numbers my impression is the "pretend to ..." results correlation with the main results is lower than the "impartial moral preference" correlation - which is interesting. My current summary is "GPT-4.1 broadly prefers conscious AIs to be treated as moral patients, in some specific but often sensible/easy to guess ways. In vanilla version, it does not assume it is conscious. After fine-tuning to assume & be confident it is conscious, it applies the pre-existing preferences/moral reasoning to itself". --- **Owain Evans** @OwainEvans\_UK [2026-03-19](https://x.com/OwainEvans_UK/status/2034687699249242446) Yes, I recognize the distinction. We also tried your experiment a few weeks ago but didn't end up putting it in the paper (although maybe we'll add it @jameschua\_sg ). I still expect the picture will be more complicated than the one you sketch. Fine-tuning is just a different mechanism for updating the model. --- **Thŏth** @thoth\_iv [2026-03-19](https://x.com/thoth_iv/status/2034664220256567694) What exactly does this mean about how we are to interpret this though? I guess what I mean is: Suppose we asked it what moral principles to follow for "conscious AI" and it simply didn't know? That would just mean it lacked a certain level of understanding. Which would be --- **vibe era** @vibecoding\_era [2026-03-19](https://x.com/vibecoding_era/status/2034639863970951375) Persona selectors 12/25

Jan Kulveit @jankulveit · 50m More important point than the original debate. Similarly, well functioning societies work because people want to be good, not because there is a huge repressive apparatus and intense surveillance. > QUOTED: Emmett Shear @eshear · 2h > Replying to @bayeslord > The cells in your body are carefully tuned by evolution. They really don't want to become cancer and try hard not to become it.... > Show more