MMMMM: A Unified Taxonomy for Investigating the Mechanisms of Multilingual MultiModal Misinformation
Nadav Borenstein, Greta Warren, Desmond Elliott, Isabelle Augenstein
cs.CY
2026-08-30
A six-axis taxonomy plus Qwen3.5-27B labels 26,979 Community Notes image-text posts at 79.3% mechanism accuracy; Slanted real photos are the most common trick, over 7,000 cases.
Image-text falsehoods are harder to unpack than text-only posts, and existing taxonomies do not line up. The same trick wears several names: Deepfake versus AI-generated, Repurposed versus Out-of-context. AMMeBa from Dufour et al. is the most complete scheme so far, and even there about 40% of images fall outside the predefined subcategories.
Expert labeling is precise and does not scale. Crowdsourcing buckles on categories that need domain knowledge. This University of Copenhagen paper ties three pieces together: a real-world harvest from X Community Notes, a six-axis taxonomy, and a vision-language model that applies it at scale.
The dataset starts from Xing et al.'s COMMUNITYNOTES dump. Notes the community rated unhelpful are dropped. Only English, Spanish, Portuguese, Japanese, French, German and Hebrew remain. Extra notes from the official public release are scraped back to the original post and image. The merged set has 26,979 (username, claim, image, note) tuples from January 2021 to January 2026. A slice of AMMeBa, restricted to cases where the image takes part in the deception, is used as an out-of-domain check.
The taxonomy first aligns synonymous categories across prior schemes, then merges near-duplicates and drops vanishingly rare ones such as Mirrored. Three gaps showed up in the data: scientific errors or conspiracies, exaggeration, and Textual claim, where the false statement lives inside the image and the post merely repeats it. The result is six axes and about 25 mechanism labels:
Two expert annotators (the authors) labeled 100 items. Other mechanism was used once and three times, so coverage is high. Mechanism agreement is 65.2%, sub-mechanism 56.8%, image type mean Jaccard 0.6, topic 0.7. Emotion and rhetorical role, being more subjective, agree less.
The labeling pipeline is Qwen3.5 27B in six stages. The mechanism stage is a three-turn chat: does the image take part, then top-level class, then subclass. Each item is run three times for stability. One pass takes about 24 hours on a single L40S in non-thinking mode.
A stratified sample of 630 English items was checked by hand, with half the labels swapped for random distractors. Annotator Cohen's kappa is 0.67, raw agreement 82.3%. After resolving disagreements, 79.3% of the model's non-distractor predictions were judged correct. Manipulated image, removal and Unreliable source, satire exceed 90%. Slanted, misrepresentation of relevance and Fake image, staged fall below 60%. On seven AMMeBa categories that align with this taxonomy, F1 sits between 0.61 and 0.83 except Time mismatch at 0.26. The authors blame the prompt in part: when time and place both mismatch, the model is told to pick place, while AMMeBa allows both labels.
On Community Notes, Slanted is the largest bin, over 7,000 items, then decorative images and Mismatch. Fake and manipulated images occur at similar rates. AI-generated images and forged documents both outnumber most manipulation subtypes. Decorative images are over-represented in Portuguese, tracking a cluster of sports, celebrity and entertainment posts. Japanese posts lean Slanted, specifically scientific errors; a manual pass finds a pile of pseudoscientific earthquake forecasts.
AMMeBa looks different. Mismatch dominates, driven by event and identity mismatches. Decorative images and AI-generated images are rarer, likely because AMMeBa was collected earlier, before the generative-image spike. Cross-tabulating topics, AI-generated images concentrate in technology and science. Vaccine-related narratives disproportionately borrow news-outlet screenshots to look authoritative.
The labels split "what the image is doing in the lie," so detectors and interventions can spend budget by mechanism instead of treating every image-text post the same. Community Notes covers messier types than professional fact-check corpora. Slanted real photos and claims printed inside the image are easy to undercount in those corpora. The model and code can be pointed at new windows of notes.
This is not a detector. The pipeline assumes a note already flags the problem. The task is to classify the mechanism, not to fish falsehoods out of the firehose.
The authors are explicit. Several older axes were dropped because they could not be labeled reliably, or needed external sources. Many narratives have too few samples, and the AMMeBa slice is only a subset. Qwen3.5 27B is not the strongest VLM; multilingual handling rides the model's built-in support, with no translation stage or language-specific models. Mechanism is single-label, while a post can mismatch both time and place. Both datasets contain low-quality items, and filtering still trusts each source's own quality flags.
Human agreement of 65% on mechanism says the fine categories are themselves blurry. The 79.3% model accuracy was measured with an expert note already in the prompt. Open detection without a note has no reason to hold that number.