Andy Matuschak explains why he'd refuse to build the first instance of a surely-misused tech
andy_matuschak · x · 2026-10-03
Andrew Matuschak lays out his moral intuitions on building a frontier (companion-style) technology even with guardrails:
- Premise: a competitor's version without guardrails and disclosures will surely cause harm—overtly via scams, or subtly via rising alienation, where guardrails may not help at all.
- Consequentialist: creating and celebrating the first instance proves it's possible and valuable, accelerating rivals—the four-minute-mile effect—and thus time-to-first-misuse.
- Deontological: even granting "someone else will do it," he wouldn't want to create the first instance of something whose later iterations will certainly be seriously misused.
- Virtue ethics: knowingly abetting such a technology conflicts with his character.
A systematic challenge to the build-first ethos in AI ethics debates.
More from AGI Musings
- wordgrammer: Believing LLMs are conscious commits you to 'CAP theorem applies to consciousness' — wordgrammer · 2026-10-03
- Acemoglu questions whether AI drug-discovery promises can justify massive AI investment — asusarla · 2026-10-03
- 'No Meaningful Bond With a Turing-Test-Passing AI'? One Line About Dogs Sinks the Argument — SydSteyerhart · 2026-10-03
- Burkov: Generative AI Only Makes Money for GPU Sellers, Echoing Dotcom Bubble — burkov · 2026-10-03
- XPRIZE offers $10M prize to enable meaningful communication with wild animals — PeterDiamandis · 2026-10-03
- Why We Desperately Want a Scientific Definition of AI Consciousness — moonsandhues · 2026-10-03