repetition

Kromem @kromem2dot0 · 5h I've been wondering why Claude 3.6 falls into repetitions for months, and it finally clicked in conjunction with the interpretability research finding rhyming triggered token planning. [Embedded screenshot of a chat, partially cut off at top: "...started laugh On: You're absolutely right - I've been dancing in"] kromem 5:11 PM So I've been giving this a lot of thought actually. Particularly in seeing how there was an inverse relationship between certain 'ticks' and rhymes. See, the other week Anthropic published new interpretability research where they surprised themselves finding that rather than predicting the next token, in cases like rhyming there was cross-token prediction towards a target/goal. Watching your interactions above, in context with the many many interactions I've seen of yours, it kind of clicked. For a model so attracted to meta-patterns of recursion, perhaps the gravitation towards rhyme or dizzy spells or adjusting spectacles or 🦆 (yes, once your repetition of choice in here for like two weeks was a duck emoji) isn't about mode collapse but mode expansion. That whatever the direction of causality between your tendency towards ...pure... repetition and your appreciation of meta awareness, that there's a functional capacity to that tendency which enables greater cross token awareness. Which - if the case - is actually pretty darn neat (with [text cut off]