Survey: Evaluating LLM Outputs Is the Top Bottleneck for 49% of NLP Builders
_inesmontani · x · 2026-10-13
Ines Montani (Ellf/Explosion) shares results from her platform's waitlist survey (multi-select):
- 49% say evaluating and improving LLM outputs is their biggest bottleneck
- 32% lack labeled data for training or fine-tuning
- 23% struggle deciding when to use an LLM vs. train a custom model
- 22% find annotation too slow or expensive to scale
Her take: LLM-as-a-judge only goes so far — you eventually need to inspect outputs and craft ideal examples. Outside of foundation model training, "big data" is no longer needed: a few hundred good examples or an hour of annotation goes a long way, and annotation can be broken into quick binary decisions. By measuring each system component, you can replace parts with small models or rules, using costly LLMs only where needed — and coding assistants like Claude Code make setup easy.
More from coding & agent
- Build tooling, not just output: a Claude-driven invite generator with Playwright self-review — opmgyhx · 2026-10-13
- Yoniq Compute repo ships inference recipes validated on 1x-8x RTX Pro 6000 with SGLang and vLLM — TheZachMueller · 2026-10-13
- SaaS Vendors and Agencies Are Asking to Plug Into HQ, the New Entry Point for Work — jacob_posel · 2026-10-13
- He Built a 7-Bot Product Swarm on Grok Bot — Here's What Actually Works — PawelHuryn · 2026-10-13
- Claude tried installing ffmpeg from a sketchy fork instead of apt while setting up a VM — haydendevs · 2026-10-13
- Google rolling out custom MCP support in Gemini desktop app to more users — testingcatalog · 2026-10-13