A research report isn't a completed research task: rethink how we judge AI science agents
Remind_me_to_Learn · reddit · 2026-09-07
The author argues we judge AI research systems too early: finding papers and producing a cited report is where real scientific work begins, not ends.
- For studies built on spreadsheets or datasets, someone still must inspect raw tables, clean and align variables, choose a defensible method, run analyses, generate figures, and trace conclusions back to the data — and handle mid-task corrections without redoing everything.
- The bottleneck has shifted: can the system maintain task state, work in real file/code environments, recover from failures, and leave inspectable artifacts?
The author demos Apodex, which in a Deep Discover example (EASIX vs. overall survival after EBV reactivation in allogeneic transplant patients) audits data, cleans it, picks a statistical method, and outputs Kaplan-Meier curves and tables — preserving unaffected work when criteria change mid-task.
Closing question: should "done" mean a well-cited report, or a reproducible package of sources, cleaned data, steps, figures, and limitations?
More from coding & agent
- Sol61 flunks instruction-following: skips mandated research even when explicitly told — lucasmeijer · 2026-10-26
- Ben Bajarin: Frontier AI is becoming the enterprise cyber-defense stack — BenBajarin · 2026-10-26
- Building AI bots on raw APIs? Devs say models are 'dumb without the harness' — madeofAjala · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Microsoft demos intent-classifying agents built with Copilot App and Foundry, deployable across Teams and Outlook — marlene_zw · 2026-10-26
- Looking for a wearable mic with best battery life that feeds recordings to personal agents via MCP — altryne · 2026-10-26