Benchmark: "LLM-as-a-judge" paradigm fails in AutoGen, CrewAI, LangGraph, MetaGPT
MonokoEloba · reddit · 2026-08-26
A local benchmark of 4 agent frameworks on a strict Rust task reveals that "LLM-as-a-judge" is fundamentally broken. Frameworks relying on LLM critics (AutoGen, CrewAI) either burned 500k+ tokens in loops or rubber-stamped wrong code. Only those with mechanical grounding (real compiler feedback) produced viable results.
Failures:
- AutoGen: 517k tokens debated into an off-topic latency script.
- CrewAI: Green tests, but logic drifted to WebSocket (ignored crypto specs).
- MetaGPT: 1 line of code, but a 912-byte "hallucinated" success report.
- LangGraph: Honest failure (compile error).
- GenOS (Author's): Partial success via compiler reality check.
More from coding & agent
- Mastra launches skill search for agents with on-the-fly loading — ycombinator · 2026-08-26
- GitHub Spec Kit booms: Spec-driven development replaces vibe coding — Shruti_0810 · 2026-08-26
- FreeLLMAPI Aggregates 34 Free LLM Providers into Single OpenAI-Compatible Endpoint — tom_doerr · 2026-08-26
- OpenAI demos WebMCP: Building agent-ready sites for 3D collaboration — OpenAI · 2026-08-26
- OpenAI desktop app adds WebMCP support, enabling Codex to deploy apps to Sites — OpenAIDevs · 2026-08-26
- devOS: MCP Server for Collaborative Memory in Coding Agents — black_phoenix9 · 2026-08-26