Admission control for self-hosted LLMs: lessons from 55 papers and 8 production incidents
AI Engineer · youtube · 2026-11-10
At the AI Engineer Code Summit, Harshul Jain (Audible) and Parv Khatri (SAP) argue that self-hosted LLM inference layers need admission control — deciding what to refuse before requests reach the GPU.
Key points:
- Coding agents fan out, retry, and burst; a naive inference stack accepts every request and buries the ones you actually need first.
- Drawing on a review of 55 systems papers and 8 real production incidents, they share five questions to ask of any admission policy before deploying it, plus an immediately usable checklist.
More from coding & agent
- InstructGPT co-author's Jev harness PDF promises 200x faster, 400x cheaper coding agents — Redo · 2026-11-10
- Three coding-agent security patches ship this week: Mistral Vibe, Jan, DeepSeek Harness — alvelda · 2026-11-10
- Dev rewrote Node app in Go, asks why developer-friendly languages matter when AI writes code — rakyll · 2026-11-10
- Claude Fable 5.2 Lands in Factory: Works Through Diagnostics, Keeps Precision — matanSF · 2026-11-10
- Open-Source MCP Lets AI Agents Edit Your YC Application Directly via GraphQL — arthurcolle · 2026-11-10
- verity-gate 0.2.0 ships its agent action-vetting gate as MCP servers for pip and npm — W1RELESS · 2026-11-10