Legora's legal AI search went from 100ms to 20s P99 — full infra postmortem

AI Engineer · youtube · 2026-09-17

A deep talk with Simon Eskildsen (turbopuffer) and Jacob Lauritzen (Legora) on rebuilding legal AI search infrastructure.

Root cause: Legal work is project-based; projects get worked hard then never opened again. After sharding chunks across 4,000 partitions, dead and live projects landed together, so every query pulled a huge partition into memory and thrashed the cache — P99 went from 100ms to 20 seconds, invisible at the schema level.
Evolution: one search cluster → one per region (data residency demands) → back into their general-purpose database, where it broke.
Fix: make the project the unit of storage, one namespace each; idle projects rest in object storage at near-zero cost.
Why regulated customers love it: a namespace per encryption key and bucket is what banks actually mean by physical isolation and key revocation.
Notable experiment: instead of encrypting the SSD cache, they disabled it entirely, found performance acceptable, and left it off.
The back half covers storage engineering: why 200ms object-storage round trips dominate, why trees beat graphs, why full-text search at web scale costs more than vector search.

More from coding & agent