ClickHouse WalShadow: shadow Postgres, TOAST reconstruction, resumable loads
mgill25 · x · 2026-09-22
ClickHouse engineers shared three engineering details behind WalShadow:
- A lightweight shadow Postgres replays catalog changes, so Postgres itself tracks schemas and types while user-table data streams into ClickHouse.
- Unchanged TOAST columns can be reconstructed from persisted chunks — even when an update only carries a pointer to an older value, large JSON/text fields need no lookup against the source.
- Initial loads resume within a table: completed COPY chunks are checkpointed only after ClickHouse acknowledges the rows, preserving progress across restarts.
More from Infra
- Rackspace joins NVIDIA Cloud Partner Program with Blackwell pods for regulated enterprises — DavidLinthicum · 2026-09-22
- Reverse-engineered Splash format ports Qwen3.8-27B to Mac, 88 tok/s at 64k on M5 Max — SeveralViolins · 2026-09-22
- Bridging LLM Decision Readouts into DuckDB: Zero-Token Probabilistic Classification via LuaJIT UDFs — Shoddy_Telephone9702 · 2026-09-22
- Nvidia launches DSX Ready to certify AI factory power and cooling, Tesla and LG first in line — emmanuelvivier · 2026-09-22
- ContextBridge: an open-source pool that routes AI tasks across local machines, APIs and shared hosting — IamAngusU · 2026-09-22
- SENPAI Auto-Research Agent Writes Kernels, Cuts Training Time 22% (26h to 22h) — morgymcg · 2026-09-22