Inside Airbnb's LLM Eval Stack: Three Layers, LoRA Caching, and Fighting Noise
iamrobotbear · x · 2026-08-04
Airbnb detailed how they run LLM evaluations internally, reading like a job description for an AI engineer.
- Three-Layer Architecture: Programmatic checks first, an LLM judge second, and humans last only to calibrate the judge.
- Key Metrics: Uses golden sets of 50-100 examples that must include failures; judges are calibrated to high 80s-90s agreement with a human measured by Cohen's kappa; samples 5% of live traffic daily.
- Admitting Noise: They admitted that roughly three-quarters of their LLM-generated reference answers varied on every labeling run, meaning their eval was previously measuring its own noise.
- Efficiency Gains: By caching identical outputs and training tiny LoRA adapters, they compressed a full eval cycle from weeks down to a single day.
Related event: Airbnb Unveils Three-Tier Internal AI Evaluation Stack(2 posts)→
More from coding & agent
- Multi-agent coding launcher aims to cut token burn with built-in compression — haseeb_heaven · 2026-08-04
- LangChain publishes fault-tolerance docs for agents, covering retries, fallbacks and human-in-the-loop — LangChain · 2026-08-04
- A new reading list links refactoring economics, agent skills, and Google’s Agent Skills — rseroter · 2026-08-04
- Eve pushes enterprise agents toward one customizable assistant per company — aarthir · 2026-08-04
- Qwen3-VL MCP server adds video Q&A, OCR and transcription for agents — modelcontextprotocol · 2026-08-04
- Qwen3.8-Max fixes 19 of 105 hidden bugs in blind coding benchmark — breath_mirror · 2026-08-04