$20 LoRA fine-tune of Qwen2.5-1.5B beats general API on sensitivity tiering, with 8x asymmetric loss
TrifleHopeful5418 · reddit · 2026-09-22
After the Jev launch, the author read the two underlying papers (arXiv 2503.23303, 2510.01237) and distilled a plain takeaway: if the output is a decision, you don't need to sample tokens—a linear head on a dense backbone reading logits suffices. So he tested it on a real problem.
Task: before every tool call, agents classify each context delta into 4 sensitivity tiers and 15 labels, so an agent that just read a customer record can't post a summary to a public endpoint 30 seconds later.
Setup:
- Qwen2.5-1.5B + LoRA (r=16, α=32)
- Ordinal cumulative-link loss on the tier head (softmax doesn't know tiers are ordered)
- 15-way multi-label head + a learned confidence head trained on correctness
- Asymmetric loss: under-classification penalized up to 8x, T1 heaviest
- Post-hoc temperature scaling, T=1.5
- 16 hours on a rented 3090, under $20 total
Results: 94.6% vs 68.1% for Jev over the API on 1,000 adversarial synthetic chunks—but the author stresses this is a domain-tuned model winning on its own invented definitions; Jev's 48.8% T3 recall is definitional disagreement, not reading failure. The defensible number is the error split: of 54 errors, 50 are over-classifications (mildly annoying), only 4 under-classifications, all one-step, no T1 content near public; the general model made 193 errors in the dangerous direction. That's the asymmetric loss—one weight tensor, not model size or budget.
Calibration was the biggest surprise: ECE 0.0185 after temperature scaling; it said 99% on 698 chunks and got all 698 right; auto-deciding the top 85% by confidence gives 0.47% selective risk; below 95% confidence it's visibly overconfident—make that slice a review queue, not an answer.
Honest caveats: test set from the same generator as training, one seed, latency gap (38.7ms vs 418ms) is mostly network topology, Jev won 2 of 15 labels and tied 3, and Laya (421M, Apache 2.0) should have been the baseline. Full stats (Wilson intervals, McNemar tests, confusion matrices, reliability diagram) in the linked Medium writeup.
More from coding & agent
- Dev built an AI game-playing plugin but shelved it: vision models too slow and costly — ezshine · 2026-09-22
- Laya: 11.6k-star open-source engine outputs typed decisions in 33ms, no generation — pandeyparul · 2026-09-22
- Jev Explained: open-source playground teaches TypeSafe's 100ms probabilistic model with 4 examples — gaganghotra_ · 2026-09-22
- Dev open-sources Jev-powered search CLI to cut token burn from LLM web research — gaganghotra_ · 2026-09-22
- OpenAI's Thibault Sottiaux teases a major 'reset' landing Tuesday — alexcovo_eth · 2026-09-22
- Disaster relief team runs OpenClaw multi-agent system across 8 messaging channels — heyneighbor · 2026-09-22