Stop treating `AGI' as the north-star goal of AI research
Borhane Blili-Hamelin, Christopher Graziul, Leif Hancox-Li, Hananel Hazan, El-Mahdi El-Mhamdi, Avijit Ghosh, Katherine Heller, Jacob Metcalf, Fabricio Murai, Eryk Salvaggio, Andrew Smart, Todd Snider, Mariame Tighanimine, Talia Ringer, Margaret Mitchell, Shiri Dori-Hacohen
ICML 2025
cs.CY
2025-02-06
ICML 2025 position paper: AGI-as-north-star aggravates six traps in setting AI goals. Sharper AGI definitions will not suffice; pick specific, plural, inclusive goals instead.
Large language models put "human-level intelligence" back at the center of the field. That aim is usually branded AGI and treated as a north star: some groups use it to orient work, others treat arrival as the destination. The term itself has no shared meaning. People disagree on what AGI is, whether it is a legitimate goal, and whether claims about it count as science.
Disagreement is not the bug. Privacy and fairness are contested too, and that contest lets the concepts move with technology and politics. The live failure is using a contested slogan as a unifying goal for the whole field. That makes four questions harder to answer well: whom the goal serves, what counts as good science, who gets to set goals, and which goals are worth pursuing.
The UN AI Advisory Body already warned that information overload blurs hype and reality, to the advantage of large firms and the disadvantage of policymakers, civil society, and the public. Each of the six traps is framed as an obstacle to drawing that line.
This is a position paper. No new experiment, no new benchmark. The diagnostic device is borrowed from fairness research: "traps." The authors do not referee AGI definitions. They name six failure modes that block productive goal-setting, and argue that AGI talk amplifies each one.
They refuse to offer their own definition on purpose. Appendix A is a comparison table: OpenAI (2018) as highly autonomous systems that outperform humans at most economically valuable work; Chollet as efficient acquisition of skills the system was neither designed nor trained for; Morris et al. as a driving-automation-style grid of performance depth versus task breadth; Weizenbaum and Attard-Frost as accounts that reject the premise. The harder the fight over definitions, the worse AGI is as a single north star. Appendix B adds a scope note: the claim is not that most researchers chase AGI, but that influential labs, executives, and papers write it as direction or destination.
Three prescriptions sit against the traps. State scientific, engineering, and societal goals in specific language. Allow many worthwhile goals and many paths. Bring end users, affected communities, annotators, and other disciplines into goal-setting. If a unifying goal is still wanted, replace AGI with supporting and benefiting human beings.
The six traps, with the paper's own examples:
| Trap | What it blocks | Concrete case |
| Illusion of Consensus | Shared words that fake agreement | Summerfield: nobody really knows what AGI would look like; Mueller: emperor with no clothes |
| Supercharging Bad Science | Vague concepts, vaguer experiments | "Language understanding" scores that may not measure understanding; linking objects to visual context sold as "imagination"; most empirical ML dressed as confirmatory, actually exploratory |
| Presuming Value-Neutrality | Value choices sold as pure technique | "Universal intelligence" proposals that skip the social goals baked in; intelligence, like health, already encodes what is desirable |
| Goal Lottery | Goals picked by incentives or luck | Economic value written into AGI; SOTA chasing pays, yet leakage, overfitting, and heterogeneous data break external validity |
| Generality Debt | "General" used to defer decisions | At least eight senses of generality, usually undefined even when central; a "universal algorithm for any environment" is hard to test under standard operating conditions |
| Normalized Exclusion | Communities and fields locked out of goal-setting | Face recognition still confusing Black people with gorillas more than eight years on; December 2024 reporting that OpenAI and Microsoft privately set AGI at $100 billion in profits |
Resource concentration sits under the last trap. Training compute has been doubling every few months (Sevilla et al., 2022), so industrial clusters set the agenda. Preprints without peer review then get treated as scientific output. AGI talk also claims to replace domain experts while ignoring psychology's own fights over "intelligence."
Section 4 takes the strongest opposing view: keep AGI, just define it better. Morris et al.'s levels and Chollet's ARC are the exhibits. The rebuttal is three points. A single unifying vision fights specificity and pluralism. The cultural charge of "intelligence" and "generality" keeps hype and evidence mixed, which lets actors project utopia or doom and then demand resources. Technology goals are set by people; the evidence trail should ask whether systems serve those people.
The paper is openly skeptical that ARC-style AGI benchmarks will help: the news cycle turns them into another SOTA race. Hiring is the counter-example. That setting needs domain-specific tests, heterogeneous evaluator practices, and known discrimination, not a general agent. How specific is specific? A Whisper MoE question is rewritten from "help hard speech domains" into "experts split by utterance length, short-utterance-heavy radio, some long calls mixed in," so a result can actually match the claim.
For people writing papers, this is a checklist, not a method. If the intro says "on the path to AGI," ask whether the goal is falsifiable, which of the eight "general"s is being measured, and whether the score maps to a user setting or a leaderboard.
For product teams, the public slogan and the private contract can diverge. A $100 billion profit bar serves an investment agreement. It does not answer "useful to whom."
For anyone stacking holistic benchmarks, Saxon et al. are used as a warning: a bundle of task-valid tests does not automatically yield user-relevant capability. Generality often postpones the choice of which capability to keep and which to drop.
This is a normative stance, not a measurement. There is no new reproducible number. The value is a shared vocabulary for fights that already live in methodology, fairness, and philosophy of science.
The authors grant they cannot rule out a repaired AGI that dodges the traps, and they do not claim to have mapped every possible AGI account. Specificity and pluralism are pro tanto reasons, overridable; climate-scale coordination is cited as a case where strong consensus can still be needed.
The taxonomy overlaps heavily. It reads more like a rhetorical map than separable mechanisms. There is no quantitative showing that "AGI" inflates these problems more than "foundation model" or "powerful AI"; the paper itself says similar worries may apply to those terms. The author list includes Hugging Face, Google, universities, and civic-tech groups: they are players in goal-setting, not outside it. The inclusion prescription stays high-level, with no operational reform for conferences or funding. "Benefit humans" as a replacement north star can fall into the same consensus illusion. That phrase is no less contested than AGI. It only points a different way.