LLM scam detection benchmarked: fitted TF-IDF baseline beats Jev, DeepSeek and local Qwen
justinbiebar · reddit · 2026-09-22
One author ran five engines on the same 1,000 job ads (53 scams; always-"legitimate" scores 94.7%). Results (accuracy/F1/PR-AUC): fitted TF-IDF baseline 0.970/0.700/0.780, hosted deepseek-v4.1-flash 0.948/0.329/0.282, Jev 0.947/0.312/0.276, GLiNER2.5 0.897/0.104/0.070, local Qwen2.5-1.5B with schema 0.053 — it emitted valid JSON for all ads but flagged every one as fraud at confidence 1.0.
Key takeaways: grammar constraints buy nothing — shape was never the hard part. Jev is well-calibrated (ECE 0.046) but its ranking is unusable, catching 7 of 53 scams at a 0.9 threshold. GLiNER2.5 breaks under pooled schemas (scores invert 0.998→0.141); one question per pass fixes it at 4x cost. The local 1.5B took 12.3s per ad on an M4; Jev handled the batch in one request. Caveats: the public dataset predates both LLMs, so their rows are an upper bound.
More from Models
- Opus 5.5 feels nerfed — user blames system prompt restrictions, not the model — oyacaro · 2026-11-12
- ChatGPT now links to brand homepages far more, reshaping brand-mention tracking — lilyraynyc · 2026-11-12
- YC-backed Monte launches post-training platform to build proprietary models from your agents' real work — ycombinator · 2026-11-12
- Leak Claims OpenAI Will Launch Entire GPT-6.2 Family Tomorrow to Counter Opus 5.5 and Fable 5.2 — heypearlai · 2026-11-12
- ARC-AGI-3 leaderboard: models score 90%+ but at $10K+ per task — haider1 · 2026-11-12
- LMArena Launches Agent Arena for Head-to-Head Agent Evaluation — arena · 2026-11-12