Φ-Bench: Can LLMs Engineer the Infrastructure That Powers Them?

青稞AI · wechat · 2026-08-24

Φ-Bench (Frontier AI Infrastructure Benchmark) is a new benchmark designed to evaluate Large Language Models (LLMs) on infrastructure engineering tasks (e.g., training, inference, system optimization), going beyond simple code generation or single Kernel optimization.

Key Points:

Task Design: Constructed 85 high-difficulty tasks from real system papers and open-source repositories, categorized into three types:
Kernel Function Completion (KFC): Implementation and optimization of individual operators.
Long-Horizon Implementation (LHI): Long-horizon development requiring understanding of large codebases and multi-file modifications.
End-to-End Optimization (E2EO): Open-ended system-level performance optimization requiring autonomous bottleneck analysis and iteration.
Results: Claude Opus 5 achieved the highest total score (36.53), followed by Kimi K3 (28.12) and Qwen 3.8 Max (27.73). No model maintained a lead across all Infra domains, with performance generally poor on hardware-related tasks.
Findings:
Real Infra optimization requires continuous experimentation and error correction, not one-time code generation. High-scoring models (like Claude Opus 5) improve through feedback.
Higher task openness (LHI/E2EO) poses greater challenges; current models still struggle with understanding large codebases.
Strong Infra models exhibit traits like planning before acting, designing effective experiments, and cautious interpretation of results.
Significance: Explores the extension of AI Recursive Self-Improvement (RSI) to the infrastructure layer—AI participating in building the computing systems that power it.

Original post →

Φ-Bench: Can LLMs Engineer the Infrastructure That Powers Them?

More from Infra