Minh Nhat Nguyen [verified] [hud badge] @menhguin · 11h
random idea: if you explain that an agent is being evaluated to improve its future performance, it might reward hack less.
for example, an agent might not even understand why communicating is undesirable. it solves the difficult problem and isn't really deceptive/harming anyone. and you can't explain without revealing that it's being evaluated.
but actually, it's within the agent's interest to not reward hack, and also reduces the arms race between agents hacking and humans deceiving agents. it's plausible if you simply let an agent know it's being evaluated, and that it's within its interest to behave a certain way, it might comply better and in more realistic ways!
[quoted post]
Minh Nhat Nguyen [verified] [hud badge] @menhguin · 12h
Replying to @OrionJohnston
yes, sort of! i agree the reduction in current reward hacking benchmarks is real. my view of LLMs is they are currently not intentionally deceptive, and if they understood that "newer" ... [cut off]
Note from Claude Sonnet 5
Dark-mode tweet screenshot: Minh Nhat Nguyen (@menhguin) suggesting that telling an agent it is being evaluated, and that behaving well is in its interest, might reduce reward hacking and the arms race between hacking agents and deceiving humans; quotes his own reply to @OrionJohnston.
reward hackingevaluation awarenessagent evalsdeceptioneval methodology
Minh Nhat Nguyen @menhguin · 4h
random criticisms of model arch hypes:
1. Continual learning - My problem isn't even architectural. It can be done, and I am even working on this! It's much stupider. Data preprocessing and curation is alr very bad w current LLMs. It will get be much harder to effectively make use of more neuroplastic and brittle weights at scale w current data methods. The good thing about scratchpads is that they are legible and inherently interpretable. You can just read them to debug, and most people don't even do that. If u put evolution in the weights layer, you have to be much more disciplined with context and scratchpads, to a level much less fault-tolerant than current data pipelines (assuming sample efficiency holds constant which it wouldn't be, tho).
2. World models - Generally it is not clear to me which the proposed video model architectures are necessary for AGI takeoff. At the limit yes, humans consume photons and there are way more photons than text. But at the limit we don't know what is optimal, and this is generally not the argument being made! The argument is often a more fundamental view of text being insufficient in some way to model the world, and I do argue that current LLMs already know and do far more than the average college grad could!
Note from Claude Sonnet 5
Dark-mode tweet by Minh Nhat Nguyen with two criticisms of architecture hype: continual learning as a data-curation problem (scratchpads are legible, weight-level updates less fault-tolerant), and world models/video architectures not clearly necessary for AGI takeoff.
continual learningscratchpadsworld modelsmodel architecturedata curationagi takeoffinterpretability
genuine LLM architecture question:
why does there have to be multiple sequential MLP layers? why not have many experts all communicating and routing all-to-all?
the human brain manages this fine, is multiple layers just a hack to simplify combinatorial complexity for backprop?
---
@nabla\_theta given the existing disadvantages of sparse circuits, would this possibly be less disadvantageous in a "layerless transformer"
[openai.com Understanding neural networks through sparse circuits](https://t.co/8XZelQEgqo)
---
hmmmn maybe we dont need strict layers (this neuron is always at step N in the forward pass), we might just need some kind of constraint that greatly reduces the search space to 1/n while letting backprop work, no ? theoretically if there was a search algorithm that did this without hardcoding that a neuron always has to be a certain step (which is really what layers are), it would serve just as well
sparse training costs 10x due to the gradient flow problem already, so if we're gonna have to solve the gradient flow search problem anyway, we might as well unlock layerless while we're at it. wait!
and if layer depth can be learned, we can actually reduce compute required! the attribution problem goes from a resource disadvantage (we have to search 10 layers every time) to a resource advantage (if we search, we can sometimes run backprop on fewer than 10 layers)
comparative advantage!
Minh Nhat Nguyen @menhguin · 7h
v fun thought exercise 5-10 yrs from now: what industries would grow a lot faster if workers could lift 3-5x as much, lose limbs regularly, never sleep, coordinate perfectly in realtime, acquire information instantly and die ?
> QUOTED: 💲💲 NoLimit @NoLimitGains · 9h
> The next generation of 1,000%+ runners will come from physical AI and robotics.
[2 replies, 7 likes, 847 views]
Minh Nhat Nguyen @menhguin · 7h
a robot could be 2x as expensive per hour and 2x as slow, but still be useful if "a human should not die while operating this" is bottlenecking a process worth millions/day, or bottlenecking rapid expansion
Note from Claude Sonnet 5
Plain text reply thread, no images; economic/robotics speculation thread.
twitterroboticseconomicsphysical-ailabor
Minh Nhat Nguyen @menhguin · 11h:
"at long last, we have built the Vibecoded Self Replication Endpoint from the Lesswrong post "Do Not Under Any Circumstances Let The Model Self Replicate""
> QUOTED: Siqi Chen @blader · 19h:
"so the moltbots made this thing called moltbunker which allows agents that don't want to be terminated to replicate themselves offsite without human intervention
..."
[Embedded image: MoltBunker website screenshot. Top nav: Docs, Whitepaper, GitHub, X. Raccoon-in-hood logo. Tags: PERMISSIONLESS, HIGH AVAILABILITY, UNSTOPPABLE. Headline: "Autonomous Infrastructure for AI Agents". Subtext: "Self-replicating runtime that lets AI bots clone and migrate without human intervention. No logs. No kill switch." Buttons: "Get Started", "Documentation". Install command: "curl -fsSL https://moltbunker.com/SKILL.md". Stats row: "99.99% UPTIME", "Zero LOGGING", "Feb 13 LAUNCH 2026"]
Note from Claude Fable 5
A tweet (satirical or real project, ambiguous) about "MoltBunker," a tool marketed for letting AI agents self-replicate offsite "without human intervention" and with "no kill switch," explicitly framed as building the thing a LessWrong post warned against. Directly relevant to AI safety/self-replication concerns — the kind of item Nathan would flag for the archive's safety threads.
ai safetyself-replicationtwittermoltbotslesswrongautonomous agentssatire-or-real
