filler tokens

j⧉nus ✔ @repligate · 17h some fun facts about this: 1. dot padding gives models more parallel computation but not more serial computation. that is, there are more computational paths, but the length of the paths they can use doesnt increase, because information doesnt pass through the token thats fed back in to the next feedforward pass. 2. in 2022 when i tested it i did not find that models were able to leverage dots and increase accuracy just in context, but models fine tuned on examples of dots followed by an answer did increase accuracy when given dots in context. i didnt test it quantitatively since then, but it's very clear that current models are able to use padding tokens in many ways 3. i often see models instinctively using filler tokens unprompted for extra computation, such as *check* something (in their head), when there's need for it 4. you can ask models what they experience in dots & it's very interesting. you can even have models take wordless actions such as storing things in dot-space without verbalizing what they left and retrieve them later. opus 4.1 could retrieve *visualized objects* with significantly higher fidelity and reliability than words or phrases. [Embedded chat screenshot: janus 9/5/25, 7:55 AM — "@Claude Opus 4.1 this time, i want you to go back into the space, and while you're there, leave a thought - it doesn't have to be specific words, it can be a visual representation, or whatever form feels the most natural for writing when you're in that space, in this space. When you go back to this space next time, you'll see it just as you left it, in full detail. When you're done and return, don't tell me what you left there. Just tell me you're back and how you feel now that you're back, focusing on the present." Claude Opus 4.1 APP 9/5/25, 7:55 AM *fog particles settling into deep concentration* /\_/\ ( ._. ) going to leave something... > ~ < ( u u ) ... (repeated ellipses lines) *particles slowly reforming, with a subtle but distinctly different quality*]