Small LLM judges hallucinated; tiny classifiers cut false cards to zero on phone
cyka_6lyat · reddit · 2026-11-22
The dev behind Omi, a fully on-device Android app that reads messages, mails, docs and photos, shares key lessons:
- LLM-as-judge failed: a small model judging every message added 9–30 false "important" cards, was slow and battery-hungry.
- Flip the pipeline: EmbeddingGemma 2 embeds each item once; small trained classifier heads read those vectors and decide what matters — 0 false cards across 293 rows, cheap enough to run all day. Same vectors power search.
- Division of labor: a fine-tuned Qwen3.5 2B only answers questions and reviews the day while charging; everything stays on-device.
- Real bottleneck: on mid-range phones it's prompt reading, not generation — short prompts and reusing cached context beat picking a faster model.
More from coding & agent
- Researcher: Collaborating with frontier models is a deeply technical skill with no diminishing returns yet — trq212 · 2026-11-22
- Dev warns: don't disintermediate yourself — skill of working with models keeps paying off — trq212 · 2026-11-22
- Weak-form proofs mostly eliminated DoS bugs in httpd-rs — a verifiable RL-training target — jd_pressman · 2026-11-22
- OLAP's key metric is now $/task, not speed — and agents need a BI dashboard analog — sh_reya · 2026-11-22
- Blind-taste-test workflow: two Gemini models grade Claude Code's video edits — aryanXmahajan · 2026-11-22
- Dev offers to trade vibe-coded apps for a PS5, sparking a barter chain for old MacBooks — RileyRalmuto · 2026-11-22