Community benchmark: Mercury Decide hits 66.7% accuracy, trailing Jev's 83.3%
FlexDruk · reddit · 2026-10-01
A developer benchmarked the newly released Mercury Decide using an uncontaminated, Korean-focused dataset (ReportJevBench) built around deciding whether Roblox chat reports violate ToS. Accuracy: Jev 83.3%, Kev 72.2%, Mercury Decide 66.7%, Solar Decide 57.8%. At a 0.5 threshold Mercury Decide produced 28 false negatives out of 90 versus Jev's 3; even after threshold tuning it leaned toward rejecting nearly every report, performing worse than Kev. The author concludes no API decision model is currently more stable than Jev, while noting the benchmark's narrow scope (Korean understanding + Roblox ToS).
More from Models
- AI's clever quantization move is just sensitivity analysis, an old pruning trick — code_star · 2026-11-16
- AI progress doubled since GTA VI's first trailer; GPT-4 Turbo now ranks 379th — ArtificialAnlys · 2026-11-16
- Tencent's Hy4 Flash builds a 3D game with shield mechanics in 63 minutes, beating Hy4's 90 — ShunyuYao12 · 2026-11-16
- Hy4 Flash 3D voxel demo outpaces DeepSeek-V4.1 in same-prompt comparison — ShunyuYao12 · 2026-11-16
- Yacine MTB claims training LLMs is 'easy', says he's shocked by how simple it is — yacineMTB · 2026-11-16
- MiniMax M3.1 Flash lands on OpenRouter, with Inferact as first provider — Xianbao_QIAN · 2026-11-16