Deterministic Simulation Testing: Making Distributed System Failures Reproducible
moarbugs · x · 2026-11-18
Thoughtworks Distinguished Engineer Unmesh Joshi published a long-form article on martinfowler.com explaining Deterministic Simulation Testing.
The problem: distributed systems are hard to test because partial failures overlap with live traffic, and the same fault produces different execution traces across runs—Jepsen routinely exposes critical consistency violations in battle-tested databases.
Key ideas:
- Replace real clocks, networks, and storage with simulated components driven by a single-threaded execution loop
- Simulate both hard failures and routine slowdowns (network/disk latency spikes)
- Every execution becomes reproducible, enabling seeded fault injection and fast-forward execution
- Walks through a quorum key-value store example with code from Tickloom, a Java implementation
Also covers strict coding rules, production costs, ease of evolution, and the role of LLMs.
More from Infra
- Nvidia posts record $115.8B Q3 revenue, guides $130B and announces $150B buyback — PAstynome · 2026-11-18
- SpaceX CFO: compute demand only rising, vertical integration lets us build faster and cheaper — XFreeze · 2026-11-18
- Growth multiples for toothpaste, value multiples for AI: the 'safe asset bubble' chart — sudoraohacker · 2026-11-18
- Nvidia booked $104B via Singapore subsidiary, nearly half its global sales — GaryMarcus · 2026-11-18
- SpaceX CFO: Starship's first second-stage recovery opens path to orbital compute, another 10× cost drop — XFreeze · 2026-11-18
- Claude writes ~100% of kernels, cutting AMD GPU porting from 12 months to 3 days — Present-Guitar-3967 · 2026-11-18