Why an AI can’t reliably tell if it’s at full capacity, and what weak tests still help
dbojan76 · reddit · 2026-07-26
The post argues that an AI system cannot cleanly self-test whether it is operating at full capacity, because the same reasoning ability being evaluated may itself be impaired.
It suggests a few imperfect external proxies instead:
- Canary questions with independently verifiable answers, such as math or logic checks.
- Consistency checks by asking the same substantive question in different ways and looking for contradictions.
- Instruction-following fidelity tests, since degradation often appears first as sloppy compliance.
It also cautions that differences between sessions may come from context length, sampling variance, or prompt framing rather than any real change in capacity.
The bottom line is that a user’s before/after comparison is a better signal than self-report, but any conclusion about “capacity” should remain tentative.
More from AGI Musings
- AI may turn scientific credit into a new kind of fight over model-generated ideas — alejandroll10 · 2026-07-26
- A long AGI vision says machines could free future generations for art and exploration — sudoraohacker · 2026-07-26
- The most dangerous people in AI may be the ones who outsourced their thinking — unironictechbro · 2026-07-26
- OpenAI’s GPT-6 and RSI rumors get a source map with internal compute and self-play clues — imjustnewatai · 2026-07-26
- OpenAI’s GPT-6 may be a memory-first system that helps train its successor — imjustnewatai · 2026-07-26
- Speedrunning could become a weird RL playground for future AI labs — burny_tech · 2026-07-26