Ex-Apple engineer open-sources jevals: fast local agent evals with one-shot decision models
byebaybay · reddit · 2026-09-22
A former Apple engineer who saw big gains from BERT classifiers—but felt the pain of fine-tuning and drift maintenance—has open-sourced jevals, a local agent evaluation tool built on Jev-style one-shot decision models.
Three selling points:
- Cost & latency: LLM-as-a-judge gets expensive and slow at millions of decisions (the author hit bottlenecks even with OpenAI parallelism recipes tuning Ragas); a local classifier is nearly free, which matters most inside agent loops.
- Generalization: adapt the classifier via natural-language questions instead of a fresh fine-tuning project per task.
- Right tool for the job: problems whose answer is a label, probability, or score never needed text generation.
jevals turns agent checks into typed questions, with the same eval definitions usable on saved traces and as runtime guardrails. Available at github.com/openlayer-ai/jevals.
Related event: Ex-Apple Engineer Open-Sources jevals for Local Agent Evaluation(2 posts)→
More from coding & agent
- mcp-server-s3 ships MCP server letting agents browse, upload and share S3 files — modelcontextprotocol · 2026-09-22
- apibase unifies 327 tools from 92 providers behind one pay-per-call MCP endpoint — modelcontextprotocol · 2026-09-22
- Dev built an AI game-playing plugin but shelved it: vision models too slow and costly — ezshine · 2026-09-22
- Developers debate persistent file storage options for AI agents across runs — OwlZealousideal4779 · 2026-09-22
- Laya: 11.6k-star open-source engine outputs typed decisions in 33ms, no generation — pandeyparul · 2026-09-22
- Dev open-sources Jev-powered search CLI to cut token burn from LLM web research — gaganghotra_ · 2026-09-22