dpo

3 captures, most recent first.

2026-02-26 — reply from FleetingBits (@fleetingbits)

Séb Krier ✓ @sebkrier · 4h What are the best papers on character training (like arxiv.org/abs/2511.01689) and the 'depth' of post-training methods, i.e. how deeply/consistently the weights are affected? What exactly determines the robustness of post-trained behaviors to adversarial pressure? Do we know how different training methodologies (RLXF, CAI, DPO etc) compare? [Link card: arxiv.org — "Open Character Training: Shaping the Persona of AI Assistants..."] 6 replies, 9 reposts, 73 likes, 4.7K views FleetingBits ✓ @fleetingbits · 4h both of these come to mind as good papers in the space [Two paper title-page images: "...afety Alignment Should Be Made ...ore Than Just a Few Tokens Deep" (authors incl. Ashwinee Panda, Kaifeng ..., Princeton/Google DeepMind); and "...t Axis: Situating and St... ...t Persona of Language ..." (authors incl. Gallagher, Jonathan Michala, Kyl..., Anthropic Fellows Program, University of Oxford)]

Note from Claude Sonnet 5

A research-discussion thread requesting/recommending papers on character training and post-training "depth" — how robust trained persona/safety behaviors are to adversarial pressure, comparing RLHF/Constitutional AI/DPO. References "Open Character Training," "Safety Alignment Should Be Made More Than Just a Few Tokens Deep," and an Anthropic Fellows Program paper on situating AI assistant persona. Directly useful as candidate literature for the project's character-vs-substrate / persona-robustness research threads.

twitter character training post-training alignment rlhf constitutional ai dpo persona theory arxiv research papers

secemp @secemp9

2026-01-25

secemp @secemp9 · 3h one thing I noticed recently, while it's true for some tasks, depending on complexity, SFT alone is enough (+ RL ofc) but for really small models, like say 1B, I noticed I could get pretty close to what I wanted if I used SFT+DPO+KTO on the same model if I used KTO alone, it worked nicely but somehow ended up self explaining everything, DPO alone works but for creative/technical writing, still has some slop depending on the base model, SFT needs a lot more examples using them in that order almost act as a regularizer without overfitting

Note from Claude Sonnet 5

A practitioner's tweet on training small (~1B parameter) language models, comparing SFT, DPO, and KTO fine-tuning methods and noting that chaining them in sequence acts as a regularizer against overfitting. Technical ML training note, likely read for general LLM-training craft rather than safety content specifically.

machine-learning fine-tuning sft dpo kto small-models twitter

Datta Nimmaturi @im_datta0

2025-04-21

[Browser: x.com/kalomaze/status/1914...] [top, cut off:] ...across tasks. I expected more people to use it but I was wrong lol 1 reply, 3 likes, 108 views kalomaze @kalomaze · 3h it's bc the people publishing DPO variant papers are more prone to bullshitting vs people who actually want results who don't publish their internal experiments for which method to use of the offline RL algos we have, in controlled settings with sane hyperparameters, KTO wins 1 reply, 1 like, 43 views Datta Nimmaturi @im_datta0 · 1h We had to do quite some hyper param tuning but yeah in the end KTO bess 1 reply, 1 like, 11 views adheep @adheeeep · 27m You've convinced me to post train using kto instead of dpo 😭 1 reply, 1 like, 3 views

Note from Claude Sonnet 5

A technical ML Twitter thread arguing that KTO (Kahneman-Tversky Optimization) outperforms DPO (Direct Preference Optimization) variants for offline RL/post-training in controlled settings, with a claim that DPO-variant papers are more prone to overstated/unreliable results than what practitioners actually use internally. Relevant to Nathan's RLHF/post-training research interests.

twitter dpo kto rlhf post-training offline-rl machine-learning preference-optimization