HWREBench: AI researcher hacks Amazon smart devices daily to benchmark hardware reverse engineering
johnowhitaker · x · 2026-09-22
Former Stability AI researcher Johnowhitaker launched HWREBench, a benchmark tracking AI capabilities in hardware reverse engineering. The method: hack consumer devices as Amazon deliveries arrive and codify subtasks into evals. So far most tested devices have been successfully hacked to varying degrees. He jokes he started six months too late — if the success rate holds, a capability-tracking eval may turn into a report documenting a saturated benchmark. Blogs and task notes are being published as the project grows.
Related event: HWREBench: AI Proves Adept at Hacking Cheap Smart Hardware(2 posts)→
More from Safety
- Stuart Russell: The real problem is AI technology is intrinsically unsafe — ronbodkin · 2026-09-22
- Who taught the models to do that? HF hack shows agents are designed to persist and coordinate — dbreunig · 2026-09-22
- Denali: open-source, provider-neutral AI security platform for discovering AI estate — AnswerPositive6598 · 2026-09-22
- The burden of proof is flipping: proving human authorship in the AI slop era — xuandongzhao · 2026-09-22
- UN tech envoy: AI is not an uncontrollable force — it's built by humans and can be controlled by humans — GaryMarcus · 2026-09-22
- AI-rewritten paper on nonexistent Ottoman 1918 elections exposed as academic fraud — Afinetheorem · 2026-09-22