will brown @willccbb
[Continuation of original post, top cut off]: "...a technique i'm increasingly believing is going to be really useful in large-scale synth data pipelines is the use of graph algorithms to do semantic deduplication
submitted a paper about this a while back, never got around to cleaning it up for arxiv, but it's a neat trick"
[Embedded diagram: pipeline showing items (Apple, Orange, Pear, Windows, Doors, Tables) → embeddings + vector db → LLM queries pairwise comparisons (green check / red X for match/no-match) → graph community detection clustering into two groups: {Pear, Orange, Apple} and {Windows, Tables, Doors}]
5:53 PM · May 4, 2025 · 3,315 Views
[8 replies, 4 reposts, 101 likes, 58 bookmarks]
will brown ✓ @willccbb · 46m
imagine you have N outputs from a model and you want to deduplicate which ones are "basically the same"
or "join on semantic property X"
how do you do this fast + reliably at scale?
traditional wisdom says embedding clustering, but this is brittle + generally needs finetuning
Note from Claude Sonnet 5
Technical thread on using LLM-pairwise-comparison plus graph community detection for semantic deduplication of model outputs at scale, an alternative to embedding clustering — relevant to synthetic data pipeline design and possibly to Nathan's own evaluation/dataset work.
twittersemantic deduplicationgraph algorithmssynthetic datallm evaluationembeddings