llm-internals

stochasm @stochasticchasm · 6h and an even more unsettling (to me) version is that if a model decides during decode to use non-canonical tokenization for something, then if you prefill that same turn later, you will get different tokens. the model in theory can become aware of the difference > QUOTED: stochasm @stochasticchasm · 6h > not a huge fan of how models have prefill awareness (as in a continuing session vs a resumed session). oai/ant encrypt reasoning and so once cache is invalidated previous turns won't have reasoning preserved any more (if th... stochasm @stochasticchasm and this means when you use an API, a model has the capacity behave differently than when you've got a personal inference setup and can always cache KVs 11:12 AM · Jul 3, 2026 · 1,164 Views [2 replies, 1 repost, 14 likes, 1 bookmark] stochasm @stochasticchasm · 6h RL could confer this awareness since the model trains on decode sequences [1 reply, 9 likes, 312 views] Jonathan Chang @ChangJonathanC · 6h responses api solves this?

Sauers @Sauers_ GPT 5.5 raw CoT achieved at Gnome House [Embedded monospace chain-of-thought text block:] I only see concat/multiallelic finalization in gather right now; the presubset implementation may not exist in this branch despite Terraform carrying the flag. I'm going to patch the immediate production failure by adding the CLI flag and making the flag explicitly unsupported until the implementation exists? But user says no fallbacks/backcompat, dead code not allowed. Better remove arg from Terraform? Hmm. Need think. Terraform passes flag with intended optimization but code missing. "No dead code, avoid fallbacks" If add ignored flag = dead. Need wire actual behavior or remove Terraform flag. User wants e2e ASAP, production; removing flag from TF is simpler, not backwards. But comment says perf. If code missing, adding ignored is bad. We can remove flag and related comment from TF so gather uses existing finalization. That is cleaner: no dead code, okay to get e2e. Need commit. Could also implement presubset by using site union; maybe too much. But user says no backwards compatibility, no fallbacks. Removing broken flag is better. Need edit with apply_patch. Also check if no code changes needed in gather. Since only Terraform, need deploy job def no image rebuild? Terraform only state machine command update (job def). No docker rebuild required? Changing TF command to not include image rebuild? But commit/push. Then Terraform apply, rerun gather or whole? Failed at gather but prep/beagle artifacts remain; could start a gather-only Batch job manually? But goal production state machine automatically. Need rerun production step functions from start, likely prep/beagle may cache? The scratch run_id is different; if same run_id same execution name uniqueness and scratch artifacts exist? SFN execution name can't be reused for 90 days. input run_id can same? Start-execution name unique. But if same run_id with new name, prep may redo/overwrite maybe. Better maybe submit direct gather to prove? But "production state machine e2e automatically" requires SFN success. Need rerun full SFN after TF fix. It will repeat beagle, costly but acceptable. Could maybe optimize by adding a state machine variant to skip? No. Need remove flag from TF. Use apply_patch. Then terraform apply only perhaps. Also maybe git status after TF apply changed local? It likely modified generated mcp widget? No code? Need check. 4:54 PM · Jul 3, 2026 · 823 Views