Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost
Sietse Schelpe
cs.CL, cs.AI, cs.IR, cs.LG, cs.PF
2026-09-30
Byte-exact KV memory for vLLM, SGLang and llama.cpp makes re-reading free: 100/100 recall on a 97k-token corpus at 0.6 s and 200 J, vs 10/100 for a truncated 12k window.
LLM serving treats every request as an independent computation. Ask a tenth question about the same document and the engine still runs the entire prompt through the model, recomputing the attention key/value state from the first token. For long prompts this prefill pass dominates time-to-first-token and a large share of per-request energy.
The title's compute ceiling refers to an argument by Vishal Sikka (former Infosys CEO) and coauthors that bounded per-token computation caps which tasks a model can carry out or verify. Galahad leaves the ceiling alone and asks how much of the budget below it goes to repeated work. The measurement on seven real-world datasets: 98.7% of prompt tokens were text the model had already read. Existing reuse is narrow; vLLM's and SGLang's prefix caches live in GPU memory, get evicted under pressure and die with the process, and offloaders like LMCache and Mooncake only extend that to CPU and disk. A document read yesterday is still re-read from token one today, which is what support desks, coding agents and contract review all pay for.
The claim: the unit of inference cost should be new text, not total text.
Galahad ships as one Linux shared library that plugs into vLLM (KV connector), SGLang (HiCache backend) and llama.cpp (slot save/restore), touching no weights and no engine source. Two independent memories:
The governing design is fail-closed: any record that is missing, damaged or fails a check is recomputed and the request completes normally. The principle is serve without memory, never with wrong memory. This is not fastidiousness; vLLM's issue tracker has reported a prefix cache returning slightly-off state and silently changing generations. So Galahad's acceptance test is bit-identical output.
The main experiment hid 100 invented facts (fake names, random numbers) in 13 Wikipedia articles (96,726 tokens) and asked for each in paraphrase, Gemma 4 31B 4-bit on an RTX A6000. The baseline held a 12,000-token window, i.e. the last 12% of the corpus:
| Configuration | Correct | s/question | J/question |
| llama.cpp, no Galahad (truncated to 12k) | 10/100 | 9.25 | 2,754 |
| llama.cpp, Taliesin | 98/100 | 3.01 | 572 |
| llama.cpp, Taliesin + Blaise | 100/100 | 0.64 | 213 |
| vLLM, Taliesin + Blaise | 100/100 | 0.59 | 200 |
| RAGFlow v0.27.2, tuned retrieval | 77/100 | – | – |
Taliesin alone sent 53,219 prompt tokens per question on llama.cpp, 99.5% loaded from memory: 5.5× the tokens of the truncated baseline, 3.1× faster, 79% less energy. Storing the corpus once cost 96 s and 27.8 kJ, with energy paid back after 13 questions.
The correctness table is the strongest part. Gemma 4 12B produced 262,144 of 262,144 bit-identical logits after process restart, rehydration from disk and hot-load; KV state moved between an A6000 and an RTX 4090 preserved 64 of 64 greedy tokens; two fresh prefills were byte-equal (SHA-256) on four architectures including Llama 3.1 70B; a Gated DeltaNet hybrid with tensor parallelism passed 5/5.
On 349 questions from seven unseen datasets (help-desk tickets, customer-support logs, messy PDFs, SWE-bench Lite, The Stack, AgentBench, WebArena), Taliesin alone scored 94–95% and Taliesin+Blaise 91–92% at 3–10× lower per-question time. Serving: on an H100 with Qwen3-30B-A3B (16 sessions, concurrency 8), throughput rose from 2.643 to 3.424 turns/s (+29.6%); with one in three loads deliberately failed it was still 2.716, above baseline. TTFT dropped on all seven GPU types tested, 2.91× on an RTX PRO 4500 down to 1.32× on an H100 SXM. Thirty of thirty models saved and loaded under vLLM; DeepSeek-V4-Flash (284B, 2×H200) answered 98/100 at 0.133 s and 143 J per question.
Disk is the real bill: this corpus occupies 17.1–17.2 GB at Gemma 4 31B (39.0 GB under SGLang). What you buy is flat GPU memory, since blocks stream in and are discarded: a 5.97M-token corpus answered 5/5 probes within 0.6 s under 25 GB of GPU memory.
For workloads that keep returning to the same documents (support, code, legal, agents), per-request cost drops from total text to new text, and the drop comes from exact reuse rather than approximation. It also changes model choice: a smaller model with exact memory of a large corpus answers questions that would otherwise demand a much longer context window, which models use unevenly anyway. Integration is one shared library across three production runtimes, and fail-closed means the worst case is today's stateless behavior, an easier sell than a system that is faster but occasionally wrong.
The cold water: single author, company product, free non-commercial beta on one GPU, patents pending, every experiment run by the author on rented cloud GPUs with no independent replication.
Self-acknowledged: Blaise's selection method, the storage format and the security design are undisclosed pending patents, so reproducibility is limited. In the messy-PDF set, 9 of 50 answers were lost by upstream text extraction before any model ran, and no memory layer restores text a parser dropped. Blaise was developed on the recall test; on unseen data it lowered accuracy from 94–95% (Taliesin alone) to 91–92%, and nothing recovers a fact the selector never passed.
Reading closely adds more. The Taliesin-only rows for vLLM and SGLang (15.56 s and 22.02 s) are slower than the no-Galahad baseline, waved off as an older SGLang backend and an upper bound, which says the KV load path is immature on some engines. The 30/30 model coverage is telemetry that save, load and hit worked, not measured accuracy. Bit-exactness checks concentrate on 12B/31B models and modest prompts; the 284B result is a recall score only. At roughly 17 GB per 97k tokens, the disk footprint scales with corpus size times model size, and the paper offers only a configurable budget, no large-deployment accounting.