RLI Update: Fable5 Hits 16.1% Automation Rate
新智元 · wechat · 2026-07-12
The CAIS/Scale Remote Labor Index (RLI) released its latest results: Fable5 achieved an automation rate of 16.1% on real freelance projects—nearly double the second-place Opus 4.8 (8.3%) and higher than GPT-5.5 (6.3%). All three surpassed any previously evaluated models.\n\nRLI measures whether a model can deliver a "client-acceptable" result in a complete business commission, rather than just single-task performance. The benchmark includes 240 real Upwork projects across 23 fields, with a total value exceeding $144,000. Each project features a gold-standard deliverable from a human freelancer for comparison, judged on whether a "reasonable client would accept it."\n\nThe article further points out:\n- Just eight months ago, the top score on the leaderboard was only 2.5%, indicating rapid progress.\n- Fable5 utilizes a more robust agent framework, specifically a worker-critic loop, where an independent review agent repeatedly checks and rejects work for revisions.\n- Its per-project budget cap is higher ($150), compared to $50 for other models.\n- AI cannot simply replace human reviewers: automated grading significantly inflates scores for new models.\n\nCAIS concludes that while progress is rapid, the absolute level remains low, with 84% of real-world projects still falling outside the current capabilities of AI.
More from Models
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- 105 hidden bugs, 2 repos: DeepSeek V4.1 Flash fixes 24 at $1.80 vs Opus 5's 27 at $51.33 — ChartsJournalX · 2026-09-11
- awesome-llm-leaderboards: an open-source directory of LLM leaderboards, pricing tables, comparison tools — Last_Establishment_1 · 2026-09-11
- Anthropic claims it works to keep eval environments unidentifiable to models — MaxKannen · 2026-09-11
- Nex N2.5 Pro released on Hugging Face with 407GB of weights — jinnyjuice · 2026-09-11
- RoMa v2 image matching model unveiled in the usual black poster — ducha_aiki · 2026-09-11