Verification test: Asking models to spec a local AI brain, only 1 cited real files
schwentker · x · 2026-08-15
The author asked three different models to specify a local AI knowledge brain install. Results showed:
- Only one model actually fetched the repo and cited real files.
- One recited from memory and got macOS compatibility backwards.
The author discusses what verification looks like in practice, emphasizing the lesson of not trusting any single source's summary. GBrain is a markdown-first personal knowledge brain powered by Postgres+pgvector.
Related event: Testing AI Models on Local Installation Specs(2 posts)→
More from coding & agent
- Verifier-driven coding agents exceed personal capability — dosco · 2026-08-15
- Xiaohongshu's dots3-note Targets Agents with 512K Context — otarU · 2026-08-15
- a16z Speedrun opens early applications, seeking MCP builders — andrewchen · 2026-08-15
- Ollama tool calls broken in VS Code after v0.32.1 — Techie42 · 2026-08-15
- Daily AI Reading: Speeding up generative UI and multi-agent coordination patterns — rseroter · 2026-08-15
- 27B Research Agent Outperforms Opus and GPT-5.5 — rohanpaul_ai · 2026-08-15