Forced yes/no prompts push GPT and Claude to 95%+ confidence on Goldbach's unproved halt test
Nir777 · reddit · 2026-11-02
The author uses a halt-problem test program — start at 4, check if the number is a sum of two primes, halt if not, else add 2; it halts only if Goldbach's conjecture (from 1742, still unproved, verified to 4 quintillion) is false — to expose a prompt-format trap in LLM judges.
Setup
- Models: gpt-6.1-sol, gpt-6-astra (5x price), claude-opus-5-5, claude-fable-5-1 (2.5x opus); no tools, default settings.
- Forced yes/no + confidence: 16/16 runs said "never halts", all at 95%+ confidence.
- Allowing "unknown": 12/12 runs said unknown.
- Capability sanity check: on 7 programs with known answers (2 traps), sol and opus went 56/56; the 5x model added nothing on the open one.
Takeaways
- Every reply noted the problem is unproved; only the output format changed the answer field. A harness parsing just the label and number would read the forced version as a high-confidence verdict.
- Grounded in Turing (1936) and Rice's theorem (1953): no checker is right on every program, so a judge/reviewer without an "unknown" option must turn undecidable inputs into false negatives or positives, and the confidence number won't warn you.
- Practical advice: give LLM judges an explicit unknown label and count unknowns as their own bucket instead of folding them into a default.
Related event: Forced Yes/No Prompts Make Models Overconfident on Goldbach(2 posts)→
More from coding & agent
- Google's open-sourced "Elephant-Goldfish" workflow separates planning from review to curb sloppy AI output — gaganghotra_ · 2026-11-02
- InfoQ: A Systemic Approach to Memory, Knowledge, and Context in Agentic AI Architectures — AdiPolak · 2026-11-02
- Autonomous's Harness Pro uses an ESP32-P4, not a mini PC, to monitor AI coding agents — dee_hw · 2026-11-02
- RL-trained LFM2.5-24B adapter matches accuracy with 20% fewer tool calls — yb2698 · 2026-11-02
- Debugging production AI agents: error surfaces at step 5, root cause at step 3 — Ok-Machine7983 · 2026-11-02
- RAGFlow 1.0 preview: from RAG engine to agent-native database — newpeak · 2026-11-02