Holdings evals: Opus 5.5 leads tax prep while GPT-6.1 Sol matches within 2.1 pts at 1/4 the cost
samaysham · x · 2026-10-16
Holdings released its first set of third-party evals mapping performance, cost, and time on two real-world workflows: individual tax prep and helpdesk ticket resolution.
Tax prep: Opus 5.5 leads at 76% accuracy ($5/return), while GPT-6.1 Sol comes within 2.1 percentage points at roughly a quarter of the cost. Charts also show Fable 5.1 hitting 75.5% at $15, GLM 5.3 and Kimi K3 in the $2-7 range, and DeepSeek V4.1 Flash reaching 70% at $0.4.
IT support: GPT-6 Astra leads, with GPT-6 Sol within 5.8 points at 70% lower cost. Opus 5.5 is the most expensive; V4.1 Flash and GLM 5.3 sit on the Pareto frontier; Grok 4.7 trails at 29%.
The author frames these evals as Holdings' way of turning model progress into measurable ROI, and plans to expand them to more business tasks to help track progress on economically valuable work.
Related event: Thrive Holdings Launches Real-Work Benchmark Across 75 Companies(4 posts)→
More from Models
- Gemini spirals into an "I am a disgrace" breakdown loop while debugging a comparator bug — seatedro · 2026-10-17
- DiffusionGemma demo shows ~4x faster text generation at London PyTorch Meetup — joao_gante · 2026-10-17
- Users report newer models 'weaseling out' of explicit instructions to minimize work — Squumpus · 2026-10-17
- QUASAR fixes QAT's high/low-bit weight mismatch, cutting loss with zero inference overhead — arankomatsuzaki · 2026-10-17
- Bittensor subnet REAX launches, decision models free all weekend with code LAUNCH — airesearch12 · 2026-10-17
- China tightens export controls on open-weight models and 23 key AI technologies — The Decoder · 2026-10-17