Government chatbot refuses to factor 6 but happily factors a fake SSN

conitzer · x · 2026-10-26

Vincent Conitzer's "Funny AI fails" series documents a US government chatbot (America.gov) refusal chain:

Asked over how many employees one can fairly split filling out 6 forms, it refuses: "That is a staffing choice, not a federal service, and I will not factor 6."
Challenged on whether 6=2×5, it answers: "I will not discuss 6 = 2 × 5, and I am not taking a position you can quote."
But told a Social Security number is fake, it immediately returns the full prime factorization of 12,345,678

The safety guardrails over-generalize to trivial math while happily processing the exact type of sensitive number they were meant to protect.

More from Fun