Solo builder crawls 10B+ records a day to put the internet in an analytical database via MCP
AaronBergman18 · x · 2026-09-03
Retweeted from XyraSinclair's long post: the author recounts his path from discovering EA, x-risk, and d/acc in 2015-16, studying CS at West Point, and committing to building "epistemic infrastructure."
What he's doing now:
- Massive crawling and ETL of public content remotely relevant to existential security — he claims over 10 billion records crawled per day, not counting bulk exports
- Making the corpus ultra-traversable with multiple tuned database technologies, aiming to do what Google failed to: put the internet in an analytical database with ergonomic access for conscientious users
- Providing MCP access to the data
- Engineering fair server-resource allocation via cost tracking and dynamic allocation
Overall it's an unusually ambitious solo research project — a distinctive direction, though with no productized output or third-party verification yet.
More from AGI Musings
- Guardian podcast: A Gemini invite changed the life of a man rebuilding it after 20 years in prison — nordicinst · 2026-09-03
- AI autonomy risk predictions from 2023 aged well: agentic LLMs, unmonitored agents all hit — davidmanheim · 2026-09-03
- David Patterson proposes flat sales tax on all revenue as every firm becomes an AI company — rand_longevity · 2026-09-03
- All AI Safety factions keep converging on the same answer: centralized control and censorship — mark_k · 2026-09-03
- Researcher's warning that future intelligences may not serve humanity gets mocked online — danfaggella · 2026-09-03
- David Patterson proposes flat sales tax on all companies to fund universal high income — davidpattersonx · 2026-09-03