AMLworld-Compact Slashes LLM Eval Targets up to 611x with Importance-Weighted Sampling
typhoon-ai · hf · 2026-10-12
SCB DataX releases AMLworld-Compact, tackling the prohibitive cost of evaluating LLMs on AMLworld, where each of over a million targets needs its own graph prompt.
- HT-Coreset protocol keeps all illicit transactions and samples benign ones from supervised-score strata, cutting evaluation targets 271x (HI-Small) and 611x (LI-Small)
- Horvitz-Thompson weights give unbiased full-test confusion-count estimates: exact recall, ratio-estimated precision and F1
- F1 RMSE of 0.00–2.58 points vs 4.44–7.41 for uniform sampling
- Seven open-weight LLMs show weighted precision near the predict-all-illicit baseline; lexical checks pass for 84.5% of correct but only 56.2% of incorrect verdicts, so such markers don't establish correctness
Eval sets, construction code, and audit tools are open-sourced.
More from Research
- SoftwareGym synthesizes 182 app environments, lifting CUA OSWorld Pass@3 to 56.97% — Tian Jin · 2026-10-12
- Qualcomm AI uses entropy and CKA to pick layers, beating EAGLE3 defaults in speculative decoding — Qualcomm-AI-Research · 2026-10-12
- Evolved e-puck robots show communication emerges with information value — uv-mex · 2026-10-12
- PreSTO Speeds Up LLM Power Sampling Up to 2.33x via Predictive Subtree Prefetching — du_yilun · 2026-10-12
- AWS+HSBC paper co-evolves agent graders from just 10 labeled examples, matching ground truth — rohanpaul_ai · 2026-10-12
- Nature MI paper: LLMs discover complementary heuristics for combinatorial optimization — lulzxdxdxd · 2026-10-12