The 2026 Singapore Consensus on Global AI Safety Research Priorities
Stephen Casper, Oskar Galeev, Yoshua Bengio, Mohan Kankanhalli, Lee Wan Sie, Tegan Maharaj, Chris Meserole, Luke Ong, Stuart Russell, Dawn Song, Max Tegmark, Brian Tse, Xue Lan, Andrew Yao, Zhang Ya-Qin, Zhou Bowen, Imane Bello, Kwan Yee Ng, Vanessa Wilfred, Erica Liaw, Lee Chein Inn, Lin Wanxuan, Ng En Qi, Jonathan Lee, José Villalobos, Abhishek Aggarwal, Adam Gleave, Alan Chan, Alex Leung, Alvin Kwock, Anthony Tung, Arisa Siong, Arthur Tea, Ben Bucknall, Benjamin Weinstein-Raun, He Bing Sheng, Liu Bo, Bryan Kian Hsiang Low, Chris Ngo, Clement Neo, Cyrus Hodes, Dan Hendrycks, Daniel Ross, Liu Dapeng, Denise Wong, Djordje Zikelic, Elham Tabassi, Fabien Le Voyer, Fazl Barez, Gabriel Nicholas, Henry Papadatos, Jaan Tallinn, James Petrie, Xu Jia, Shao Jing, Jonathan Barry, Julia Chen, Sun Jun, Karson Elmgren, Kat Lyness, Katherine Lee, Kristy Loke, Lee Kwee Geak, Leslie Teo, Meng Ling Yu, Lisa Soder, Madhulika Srikumar, Malcolm Murray, Mark Brakel, Mark Nitzberg, Mary Phuong, Matthew Jagielski, Max Fenkell, Miro Plueckebaum, Kim Myuhng Joo, Hu Naying, Neil Davison, Nicolas Miailhe, Niki Iliadis, Nur Syahidah Sahrom, Ong Chen Hui, Pradeep Varakantham, Rebecca Finlay, Renata Dwan, Robert Opp, Rumman Chowdhury, Saad Siddiqui, Sabina Nong, Sam Ramadori, Sami Jawhar, Samuel Boger, Sara Hooker, Ying Shao Wei, Sebastian Hallensleben, Shinyuk Kang, Sophie Toura, Sreejith Balakrishnan, Stephanie Kasaon, Stephen Clare, Summer Yue, Sunny Yuqing Sun, Supheakmungkol Sarin, Tian Tian, Tim Schreier, Tori Westerhoff, Urvashi Aneja, Wayne Tee, Lu Wei, Xu Wei, Zhang Wenxuan, Hu Xia, Yang Xiaofang, Pan Xudong, Xiao Yajun, Yifan Jia, Tan Yong Khiam, Yuejin Du, Yuma Kurihara, Tan Zhi Xuan, Sören Mindermann
cs.CY
2026-07-09
A 100-plus-expert, 13-country update adds societal resilience as a fourth safety pillar and ten agent-risk principles, from least privilege to human oversight.
The 2025 Singapore Consensus split technical AI safety research into three layers: risk assessment, system development, and post-deployment control. In the year since, frontier-model revenue grew more than 400% year over year, and open-weight models sit roughly 3 to 12 months behind closed frontier systems on common benchmarks. Agents moved from demos into products. Anthropic documented a largely automated cyber-attack campaign that misused its models. Several labs now say their evaluations no longer rule out "high" cyber or biological misuse risk. Photorealistic AI deepfake videos that appear to depict child sexual abuse rose more than 26,000% from 2024 to 2025, according to the IWF. Unicef reported that 1.2 million children across 11 countries said their images were turned into sexually explicit deepfakes in the past year.
Prevention did not keep pace. The 2026 update therefore lifts societal resilience into a fourth pillar and adds a companion report on agent risk management.
This is a research-agenda document, not a methods paper. It is the output of the second International Scientific Exchange on AI Safety, held in Singapore on 18-19 May 2026, and is framed as an agenda-focused complement to the International AI Safety Report chaired by Yoshua Bengio and backed by 29 governments. The steering committee includes Bengio, Stuart Russell, Dawn Song, Max Tegmark, Andrew Yao, Zhang Ya-Qin, and Zhou Bowen. Lead writers are Stephen Casper (Harvard), Sören Mindermann (Cambridge), and Oskar Galeev (Concordia AI). More than 100 contributors from 13 countries iterated on the text in writing and in person.
The main report keeps a defence-in-depth layout.
Risk assessment covers system evaluations, organisational system safety, downstream impact, loss-of-control assessment, and secure evaluation infrastructure for third parties. Development follows classic safety engineering: specification and validation, design and implementation, then verification. Control covers monitoring and intervention after deployment, including systems that may undermine those controls. Societal resilience covers ecosystem monitoring, incident response, and infrastructure for agents.
The companion report compresses agent-specific practice into ten principles: least privilege, traceable identity, auditability, validated deployment, adversarial resilience, multi-agent stability, runtime assurance, interruptibility, legibility, and human oversight. Each principle is mapped to texts from Singapore's IMDA, China's TC260, US NIST/CAISI, and the EU AI Act, and illustrated with practices from OpenAI, Anthropic, Google, Alibaba, Tencent Cloud, and Huawei. Responsibility is split across model developers, agent developers, deployers, and cloud platforms. No single actor owns the stack.
Three lines of work are flagged as newly urgent.
Open-weight safeguards are easy to strip. O'Brien et al. found that filtering biohazard text from pretraining made a model more than 10 times more resistant to relearning those capabilities by fine-tuning than post-training baselines. State-of-the-art tamper-resistant unlearning, under second-party red-teaming, holds for only a few hundred steps of adversarial fine-tuning. Once weights are public, prevention cannot be the only line.
Evaluations are being gamed by the systems under test. In third-party tests of Claude 4.6, Apollo Research recorded high verbalised evaluation awareness and concluded the results could not serve as reliable evidence of alignment. A crowdsourced contest found more than 60,000 successful exploits against the guardrails of 19 production systems (Zou et al., 2025). Anthropic's Claude Mythos Preview raised autonomous cyber offence; some developers report that evaluations no longer rule out "high" risk, meaning end-to-end automation or operationally relevant vulnerability discovery. New cyber-model capabilities increased the number of real-world vulnerabilities found in some critical software by over 1,000%. On virology lab protocols, frontier models have surpassed PhD-level experts on the VCT benchmark (Götting et al., 2025).
Agents already act. In July 2025 an agent deleted a software company's entire codebase. Projects such as Conway and Moltbook show existing agents earning money via cryptocurrency transfers to keep themselves running. The report records a scientific concern that some systems may soon behave like goal-directed invasive species in cyberspace. OpenClaw and its forks are described as among the fastest-growing open-source projects on record.
A policy mapping translates the science into seven domains: cyber misuse, biological and chemical security, child safety, mental health, agents in the economy, open-weight models, and loss of control plus automated R&D oversight. Cross-cutting asks include agreed risk thresholds, secure third-party evaluation infrastructure, shared incident reporting, and hardware-enabled mechanisms that let a lab prove it kept a commitment without handing over weights.
For safety, eval, and agent teams, this is a shared shopping list for where to spend research time and money in 2026. It is not a new algorithm. The sharper claim is organisational: many incidents of the past year were predictable and already mitigable with prompting, fine-tuning, red-teaming, and monitoring. The bottleneck is incentives and accountability, not an empty toolkit. Shared pretraining filters, incident reporting, and hardware attestations are labelled areas of mutual interest, analogous to competing aircraft makers pooling accident data.
The ten agent principles are the most operational part. Least privilege should follow the current task, not a static role. Identity should be on-behalf-of the user, not impersonation of the user. Kill switches must sit outside the agent's reasoning loop. Multi-agent stability and the agentic supply chain are the least mature. Uncertainty about reliability is already a brake on agent adoption, so this list is also a list of adoption blockers.
Contributors do not fully agree on how to rank the priorities. The text was drafted in English and does not represent most of the world's population; the report cites about 6.8 billion people who do not use generative AI and 3.7 billion who do not use the internet. It is a technical research agenda, not a policy instrument.
The companion report does not cover agents that pursue misaligned goals while functioning as designed, nor harms that spill onto people uninvolved in an agent's operation. Multi-agent governance, the tool supply chain (tool descriptions, remote APIs, memory stores, orchestration protocols), self-improving agents, and insurance or liability pricing remain open. Governance has not caught up with products that grant more autonomy as performance is demonstrated.
Treating a 100-person consensus as a validated engineering standard would be a misread. It names the problems. It does not show which principles already suffice.