FLARE-AI: Flaw Reporting for AI
Shayne Longpre, Elaine Zhu, Carson Ezell, Avijit Ghosh, Sean McGregor, Kevin Paeth, Kevin Klyman, Sayash Kapoor, Rishi Bommasani, Ruth Appel, Gregory Strom, Lauren McIlvenny, Mark M. Jaycox, Peter Slattery, Nathan Butters, Arvind Narayanan, Percy Liang, Alex Pentland
ICML 2026
cs.CY, cs.AI
2026-06-30
Survey of 12 AI reporting systems and 49 experts yields FLARE-AI: one open-source form with 6 required fields that can route a JSON-LD report to many developers and coordinators.
General-purpose models now sit in front of billions of people. Flaw reporting still looks like software security a decade ago. A red teamer who finds a jailbreak that hits several models often cannot tell where to send it. Each form has a different scope, taxonomy and required field list: MITRE ATLAS can demand 53 fields, OpenAI's Bugcrowd form asks for 7. Recipients do not share with competitors by default, so the same bug gets typed in many times.
This ICML 2026 paper comes from MIT, Northeastern, Harvard, Hugging Face, Stanford, Princeton and CERT, among others. Corresponding authors are Shayne Longpre (MIT) and Avijit Ghosh (Hugging Face). They dissected 12 existing reporting systems and consulted 49 experts at 32 organizations, including OpenAI, Anthropic, Google, Cohere, Meta, NVIDIA, plus CERT, CISA, MITRE, AIID and OECD.
The survey compared two layers: scope and taxonomies, then field-level intake (reporter identity, system identifiers, reproduction steps, impact, evidence). It also scored anonymity, public disclosure and cross-organization coordination. In user-study sessions, security researchers re-filed real flaws they had already found, using early prototypes.
Five recurring failures fell out: channels are hard to find, in-scope definitions and taxonomies do not line up, forms swing between too thin to triage and too long to finish, reports almost never leave the first inbox, and strict-liability content (real or synthetic CSAM) gets almost no guidance.
FLARE-AI patches those five. A resources page lists 15 reporting channels, sortable by organization type and in-scope flaw. Intake is broad: flaws, hazards, vulnerabilities and incidents are all accepted. Three yes/no questions at the start branch the form: has harm already occurred, could a malicious actor use this, does it involve CSAM. The first two decide later questions and routing. The third sends people to the right authorities and keeps the material out of this form.
The required core is 6 fields; an optional long path goes up to 30. Choosing a product surfaces relevant policy links and suggested recipients. Harm types follow the AIAAIC taxonomy; flaw types follow the OWASP AI Top 10.
The system is stateless by default. A reporter can generate a JSON-LD file locally and never touch a server. If they opt in, the same machine-readable report goes out over APIs or official email to developers, CERT, MITRE, CISA, AIID and other registries. The schema is meant to sit next to CVE/CWE, AVID and CERT workflows. Code is open; steps and taxonomies can be swapped. The demo lives at ai-reports.org and walks through eight steps from classification to routing.
This is a systems paper. There is no "accuracy up X points" table. What can be counted is the ecosystem:
| Observation | Number |
| Reporting systems reviewed | 12 |
| Experts / organizations consulted | 49 people / 32 orgs |
| Allow anonymity | 3 of 12 |
| Allow public disclosure | 9 of 12 |
| Do cross-organization coordination | 8 of 12 |
| Field-count range | MITRE ATLAS 53, OpenAI Bugcrowd 7 |
| FLARE-AI required / optional | 6 / up to 30 |
CERT, MITRE, AIID, Hugging Face, OECD, OpenAI, Anthropic and Google gave routing-layer integration commitments at the time of writing. Those are commitments, not post-launch traffic. Early user studies fixed fuzzy terms, missing "unknown" options and attachments without context. There is no controlled comparison of filing rates or triage latency.
For evaluators and red teamers the payoff is concrete: fewer forms, and the fields that actually speed triage (model version, access channel, whether the issue is already public). For developers, JSON-LD can enter an existing vuln ticket instead of being scraped out of email. Article 73 of the EU AI Act is already pushing mandatory incident reporting. This paper offers coordination plumbing, not a form that files with the EU AI Office.
No new attack or defense is introduced. The missing kilometer after a flaw is found is what gets standardized. CVE-style coordinated disclosure took software twenty years to settle. AI reporting is still at the start of that curve.
The authors flag several themselves. Consultations clustered in North America and Europe, even though OECD frameworks are cited. One form cannot satisfy every recipient's granularity, so follow-up mail will still happen. There is no experiment showing that filing rates, triage speed or multi-party patches actually improved. Supply-chain flaws that travel through fine-tunes, API dependencies or shared training data are listed as future work.
A few gaps sit outside that list. Routing commitments are from a June 2026 preprint; there is no public dashboard of who is actually ingesting reports, and opening the demo does not mean CERT is auto-triaging. Statelessness cuts centralized leakage, but if a user ticks "send to many," an unpatched issue lands in many inboxes at once, a misuse surface the paper names. Picking AIAAIC and OWASP is a normative bet on what counts as harm; other labs may refuse that vocabulary. Safe harbor remains a legal problem. An open form does not decide whether a researcher gets sued.