LLM scam detection benchmarked: fitted TF-IDF baseline beats Jev, DeepSeek and local Qwen
justinbiebar · reddit · 2026-09-22
One author ran five engines on the same 1,000 job ads (53 scams; always-"legitimate" scores 94.7%). Results (accuracy/F1/PR-AUC): fitted TF-IDF baseline 0.970/0.700/0.780, hosted deepseek-v4.1-flash 0.948/0.329/0.282, Jev 0.947/0.312/0.276, GLiNER2.5 0.897/0.104/0.070, local Qwen2.5-1.5B with schema 0.053 — it emitted valid JSON for all ads but flagged every one as fraud at confidence 1.0.
Key takeaways: grammar constraints buy nothing — shape was never the hard part. Jev is well-calibrated (ECE 0.046) but its ranking is unusable, catching 7 of 53 scams at a 0.9 threshold. GLiNER2.5 breaks under pooled schemas (scores invert 0.998→0.141); one question per pass fixes it at 4x cost. The local 1.5B took 12.3s per ad on an M4; Jev handled the batch in one request. Caveats: the public dataset predates both LLMs, so their rows are an upper bound.
More from Models
- Paradigm launches Limite 1B - Violetto, a model for high-frequency mathematical intelligence — tensorqt · 2026-09-22
- Reliquary-4B: A 4B math & code model trained via decentralized RL with community rollouts — const_reborn · 2026-09-22
- Users say they can't trick Jev into hallucinating — BLUECOW009 · 2026-09-22
- Measured trade-offs of three REAP-pruned Qwen3.8-Flash-Next MLX builds on Apple Silicon — MensaProdigy · 2026-09-22
- Dev claims further-optimized DeepSeek V4 NVFP4 uses 190GB of 192GB VRAM — HankYeomans · 2026-09-22
- OpenAI researcher Will Depue on why voice models still lack true realtime chat — willdepue · 2026-09-22