Fireworks opens Specialized Intelligence Index; Factory debuts Legacy-Bench for legacy software
maddiehfaulkner · x · 2026-09-25
Fireworks AI announced that @FactoryAI joins its Specialized Intelligence Index with Legacy-Bench, testing how models handle legacy software behind payroll and financial settlement. Teams can now submit proprietary benchmarks: Fireworks covers compute, runs leading open and closed models, keeps tasks/prompts/trajectories private by default, publishes quality-cost-latency comparisons, and can post-train models on identified failure modes. Signals a shift from generic leaderboards to domain-specific, real-work evaluations.
Related event: Factory's Legacy-Bench Tests LLMs on Legacy Code, Joins Fireworks SII(3 posts)→
More from Research
- TTIC launches LEIF Lab to study Transformer expressivity via formal language theory — lambdaviking · 2026-09-25
- Researcher lands 4 NeurIPS papers including one Oral, spanning synthesis to protein diffusion — abeirami · 2026-09-25
- Calibration-Free Quantization Method TQ Open-Sourced, Hits 92.4% Top-1 on Qwen 27B 4-bit — textclf · 2026-09-25
- Yoav Goldberg: LLM reasoning traces are 'too good' — unclear how they emerge from RL — yoavgo · 2026-09-25
- NeurIPS Oral: IDS co-evolves code with formal proofs, hitting 3x Claude Code's success rate — adityagp · 2026-09-25
- Neuralese Recurrence Explained: Why OpenAI's Astra Loop Is Not Yet Unreadable AI Thought — Astral Codex Ten · 2026-09-25