A delivery team's decision ladder: better prompts, then RAG, then fine-tuning
247Labs_Inc · reddit · 2026-08-18
The head of AI delivery at 247 Labs shares the fixed decision ladder used on client projects — move down a rung only when the one above demonstrably fails.
- Rung 1 — is the prompt actually bad? Build an eval set with known-correct outputs first. Inconsistent formatting or ignored instructions are prompt problems; structured prompting plus examples often closes the gap.
- Rung 2 — missing knowledge? Confident factual invention (policies, SKUs, contract clauses) means retrieval, not tuning. Fine-tuning teaches behaviour far better than facts. Real RAG failures: contradictory corpora with no authoritative version, chunking that splits tables, and access control deferred to "phase two" — for regulated clients, that is the project.
- Rung 3 — behaviour you can't instruct into it: legitimate fine-tuning cases: rigid output formats at high volume, house style few-shot can't hold, and smaller tuned models beating large prompted ones on latency and unit cost. Only the first is about capability; the other two are economics.
The deciding factor across all three: an eval set. Not vibes, not a stakeholder demo — a fixed set of inputs re-run on every change, the one line item the author would never cut.
More from coding & agent
- NVIDIA open-sources NOOA: Build AI agents using pure Python classes — solyarisoftware · 2026-08-18
- 如何设计通用的模型分层系统替代硬编码模型映射 — chipro · 2026-08-18
- Self-hosted AI analyst writes SQL, self-checks numbers, and cites every claim to its query — Outside-Risk-8912 · 2026-08-18
- Vector databases outperform note files for agent memory at scale — rseroter · 2026-08-18
- MCP Protocol Enables Voice-Controlled Shopping on Smart Glasses — Scobleizer · 2026-08-18
- Fix oMLX OOM stalls in Pi agent by adding a "reduce context" compact trigger — chibop1 · 2026-08-18