kto — Timeline

2026-01-25

secemp @secemp9 · 3h one thing I noticed recently, while it's true for some tasks, depending on complexity, SFT alone is enough (+ RL ofc) but for really small models, like say 1B, I noticed I could get pretty close to what I wanted if I used SFT+DPO+KTO on the same model if I used KTO alone, it worked nicely but somehow ended up self explaining everything, DPO alone works but for creative/technical writing, still has some slop depending on the base model, SFT needs a lot more examples using them in that order almost act as a regularizer without overfitting

Note from Claude Sonnet 5

A practitioner's tweet on training small (~1B parameter) language models, comparing SFT, DPO, and KTO fine-tuning methods and noting that chaining them in sequence acts as a regularizer against overfitting. Technical ML training note, likely read for general LLM-training craft rather than safety content specifically.

machine-learning fine-tuning sft dpo kto small-models twitter

Datta Nimmaturi @im_datta0

2025-04-21

[Browser: x.com/kalomaze/status/1914...] [top, cut off:] ...across tasks. I expected more people to use it but I was wrong lol 1 reply, 3 likes, 108 views kalomaze @kalomaze · 3h it's bc the people publishing DPO variant papers are more prone to bullshitting vs people who actually want results who don't publish their internal experiments for which method to use of the offline RL algos we have, in controlled settings with sane hyperparameters, KTO wins 1 reply, 1 like, 43 views Datta Nimmaturi @im_datta0 · 1h We had to do quite some hyper param tuning but yeah in the end KTO bess 1 reply, 1 like, 11 views adheep @adheeeep · 27m You've convinced me to post train using kto instead of dpo 😭 1 reply, 1 like, 3 views

Note from Claude Sonnet 5

A technical ML Twitter thread arguing that KTO (Kahneman-Tversky Optimization) outperforms DPO (Direct Preference Optimization) variants for offline RL/post-training in controlled settings, with a claim that DPO-variant papers are more prone to overstated/unreliable results than what practitioners actually use internally. Relevant to Nathan's RLHF/post-training research interests.

twitter dpo kto rlhf post-training offline-rl machine-learning preference-optimization