← All topics

llm judge

2 captures, most recent first.

Sam Paech @sam_paech

— saved image

Sam Paech [verified] @sam_paech · 13h

New models tested on creative writing:
GPT-6-Astra
Fable 5.1
Muse Spark 1.3
Gemini 3.8 flash

Subjectively, I strongly prefer Muse Spark 1.3 to both GPT-6 and Fable. It just gives you what you asked for, without an obnoxious house style permeating every request.

Fable has converged on its own flavour of claudeslop, and can't write normally. Meanwhile, GPT-6-Astra has forgotten how to write in paragraphs. It's a sad state of affairs!

Tentatively suggest we reinstate the old tradition of reading rollouts with our actual eyeballs.

[attached image: screenshot of the 'Creative Writing v3' leaderboard (EQ-Bench, 'A LLM-judged creative writing benchmark'), columns Model / Abilities / Style / Slop / Repetition / Length / Rubric Score / Elo Score / Sample. Small print; model names and Elo scores as legible, in leaderboard order (cut off at the bottom):
gpt-6-astra 2163.9
claude-fable-5-1 2152.7
claude-opus-5 2120.6
kimi-k3 2070.6
GLM-5.3 2064.1
gpt-5.6-sol 1963.4
ox-alpha 1960.6
claude-fable-5 1934.6
muse-spark-1.1 1916.1
claude-opus-4-7 1907.1
muse-spark-1.3 1905.5
gpt-5.6-terra 1850.3
gpt-5.5 1843.5
Qwen3.8-2.4T-A95B 1840.9
gpt-5.4 1835.6
claude-opus-4-8 [Elo illegible]
muse-spark-1.2 [Elo illegible]
gpt-5.6-luna [Elo illegible, cut off]]
Note from Claude Sonnet 5

Dark-mode tweet screenshot: Sam Paech on creative-writing tests of GPT-6-Astra, Fable 5.1, Muse Spark 1.3 and Gemini 3.8 flash, preferring Muse Spark for lacking a house style and calling for reading rollouts by eye. Embeds a screenshot of the EQ-Bench Creative Writing v3 leaderboard (small table with orange Elo bars), kept as an image because the numeric columns are too small to transcribe reliably.

creative writingbenchmarkseq-benchgpt-6-astrafable 5.1muse sparkslophouse stylellm judge

@klyap_

— quoting @kickingkeys — saved image

Ker Lee Yap @klyap_ · Aug 23
Some cool learnings i got from this blog:
- don't do rubric judge scoring 1-10, just do pairwise comparisons
- you can just handpick a golden set and compare outputs to a random one from there instead of a reward model
- simple prompts are better, dont add too much detailed specs

[quoted tweet]
Surya @kickingkeys · Aug 23
You can just RL a coding model to paint with javascript btw

[embedded video, 0:09, paused: generative artwork on a black background showing translucent white-and-red flower forms with thin purple and orange curving lines and small green leaf shapes]
Note from Claude Sonnet 5

Tweet by Ker Lee Yap summarizing three lessons from a blog on RL with model-based judging (pairwise comparison over 1-10 rubric scoring, a handpicked golden set instead of a reward model, simple prompts), quoting a Surya tweet about RL-training a coding model to paint with JavaScript. The quoted video shows generative flower artwork.

rlllm judgepairwise comparisonreward modelgenerative artcoding models