$0.20 Offline Replay Shows Small Model Jev Beats Big LLM on Routing and Retrieval Tasks
Mike Giannulis, founder of an AI book-writing company, posted a 12-tweet thread on September 18 fully documenting an offline validation experiment on "replacing a large model with a small one": replaying 3,300 real production decisions through @typesafeai's small model Jev, pulling real inputs from their own database, comparing against past Claude decisions and human-labeled/deterministic ground truth, and pre-registering pass/fail rules before seeing results—fully offline with no online experiments.
Confirmed
- On book-material retrieval, embedding similarity ranked the correct passage first 60% of the time; adding Jev reranking raised it to 90%.
- Portal intent routing accuracy was 90.5%, above the existing keyword routing's 83.8%; "I've already done this" detection hit 98.3%, above the regex approach's 96.6%, and the replay precisely caught the message previously missed in production that had embarrassed the team.
- Their customer-support classifier had misfiled 13 messages like "please text, don't call" into "other"; Jev caught all of them in replay.
- Faithfulness judgment: 95% accuracy on 148 pairs of clean samples with a Brier score of 0.057; the author cautions that small-scale tests can't prove perfect production calibration.
- Failure cases: on subtly worded author-email tasks their in-house LLM beat Jev roughly 2:1; subjective judgments like "too salesy" were no help; no improvement on thin-data sales-lead scoring.
- Measured latency was about 0.2 seconds versus the previous 2–4 seconds; the author admits the "20 cents" in the title is a gimmick—replacing calls that were already cheap, the token bill barely moves.
Not confirmed
- All results above come from offline replay with no production validation yet; the author himself is deliberately restrained in his claims.
Why it matters
The author's core conclusion: "before buying a smarter model, check whether you're making it do the wrong job." Leaderboards can't reveal issues like "'text, don't call' being filed as other" that only your own data exposes; small models fit "boring tasks where customers immediately notice a screwup." Next step: shadow-mode rollout for the reranker, portal routing, "already done" detection, and contact-preference classification, dual-logging old and new decisions, checking divergences, and keeping a one-click kill switch—an offline validation win only earns a shadow-mode opportunity, not the keys to the whole business.
2026-09-18 ~ 2026-09-18 · 10 related posts
- Episode 1: Jev Outputs Structured Decisions Directly at $0.042/M Tokens(2026-09-18, 2 posts)
- Episode 2: $0.20 Offline Replay Shows Small Model Jev Beats Big LLM on Routing and Retrieval Tasks(2026-09-18, 10 posts)
Primary sources
- A 20-cent AI that can't write a sentence caught 13 requests our classifier missed — mikegiannulis · 2026-09-18
- [source] Replaying 3,300 Real Production Decisions to Test Small Model Jev — mikegiannulis · 2026-09-18
- [source] Before buying a smarter model, check you asked the wrong job: Jev test wrap-up and rollout rules — mikegiannulis · 2026-09-18
- [source] Same Material, Better Order: Reranking Lifts Correct-Passage Hit Rate to 90% — mikegiannulis · 2026-09-18
- 13 'Please Text Me, No Calls' Messages Misrouted — Small Model Caught All of Them — mikegiannulis · 2026-09-18
- Intent Routing 90.5% vs 83.8%, 'Already Did That' Detection 98.3% vs 96.6% — mikegiannulis · 2026-09-18
- 95% Accuracy, 0.057 Brier Score on Faithful-vs-Fabricated Pairs — With Caveats — mikegiannulis · 2026-09-18
- What Didn't Work: LLM Won Nuanced Emails 2:1, Three Tasks Where Small Model Failed — mikegiannulis · 2026-09-18
- The 20-Cent Bill Is a Headline — Real Wins Are 0.2s vs 2–4s Latency and Fewer Mistakes — mikegiannulis · 2026-09-18
- Offline Wins Earn a Shadow-Mode Test, Not the Keys to the Whole Business — mikegiannulis · 2026-09-18