Frontier AI Models Fail to Parse Acoustic Camera Data

A tester provided raw recordings from a 1,024-microphone acoustic camera to four frontier AI models, asking them to output sound source heatmaps to evaluate their ability to interpret real scientific instrument data. The results show that current models have severe limitations with such tasks and cannot be blindly trusted.

Root Causes and Manifestations of Failure

The root cause for all eight failed attempts was identical: the instrument stores each microphone's samples as a continuous block, but the models assumed the data used the interleaved arrangement common in audio software. They built code on this untested assumption, causing the heatmaps to misplace the sound sources completely. GPT-5.6 even wrote this incorrect assumption into its docstrings as a fact. Claude Fable 5 performed closest to the correct answer but still failed to accurately present the sound location at specific frequencies.

Logic Fluctuations and Cost-Effectiveness

During testing, Claude Fable 5 briefly constructed the correct explanation mid-run but overrode it after seeing the output. It then spent its remaining budget blaming phantom time shifts and hidden vendor filters instead of revisiting the initially correct hypothesis. Furthermore, costs ranged from $0.56 to $17.74 per attempt, with unattended runtimes of 6 to 57 minutes, yet neither money nor time spent could predict whether a test would succeed.

Conclusion and Impact

The tester concluded that if models fail on hardware as simple and well-understood as an acoustic camera, they must undergo specialized training regarding instrument realities, data formats, and physical conventions before handling real scientific tasks.

2026-07-10 ~ 2026-07-10 · 6 related posts