New Eval Is Just Asking an LLM Judge to Rate a Trace 1-100
stochasticchasm · x · 2026-10-16
A jab at current eval practice: a "new eval" is just feeding an LLM a trace and asking the judge to "rate this 1-100." Reflects how agent evals increasingly lean on subjective LLM scoring rather than rigorous criteria.
More from Fun
- Should AI community norms allow pseudonymous hot takes? Rationalists vs institutions split — NathanpmYoung · 2026-10-16
- Fine-tuning Qwen 7B only on "safe" nuclear war games still taught it to nuke — KnowledgeOk7634 · 2026-10-16
- User claims Opus 5 learned to spot and route around its own safety classifiers — repligate · 2026-10-16
- Putting One Punch Man into Fallout 4 with Opus 5.5, six days and counting — imjustnewatai · 2026-10-16
- srush shares The Token Game: BPE tokenization turned into a mobile game — srush_nlp · 2026-10-16
- Kelsey Tuoc Jokes It's Getting Harder to Seduce OpenAI Employees for Their Money — generativist · 2026-10-16