A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware
Ayushman Bhattacharya, Nihal Gazi
cs.DC, cs.PF
2026-08-12
OreoLook's three-layer cache on an 8-vCPU box reports 89.3% Redis keyspace hits and 0.1 ms reads; the authors warn this is not a query-level semantic hit rate.
Commercial AI search APIs bill per query. OpenAI web search lands around $0.03–$0.10 per call, Gemini Pro grounding around $0.035, Perplexity Sonar Pro around $0.018. The pollinations.ai team built OreoLook (formerly lixSearch), an open-source answer engine: headless browser agents scrape the live web, a remote provider synthesizes the answer, and the local search, cache, and embedding stack runs on commodity CPUs.
Once traffic grew, three failures showed up together. Multi-turn sessions forgot context. Rephrasings reran the full pipeline. Popular URLs were re-embedded in every session. LangChain memory does not survive process restarts. GPTCache can skip duplicate LLM calls, but it is a global cache with no session isolation and it wants a separate vector store. They needed something that fits an existing pipeline and only Redis.
Each concern gets its own Redis logical database.
A semantic hit short-circuits search and synthesis. A miss loads session context, checks URL embeddings, then runs the browser pipeline. Huffman is pure Python. The authors admit zlib compresses better; they picked Huffman because typical archives are under 10 KB, the gap is small there, and there are no native dependencies.
The evaluation is one historical production snapshot, not a controlled bake-off. Hardware: one 8-vCPU Cascade Lake box (2 GHz, 32 GB, no local GPU), three container replicas, 30 Hypercorn workers, Redis 7.4 capped at 2 GB.
| Metric | Value | Baseline / caveat |
| Redis keyspace hit rate | 89.3% (2,182 / 2,444) | not a query-level semantic hit rate |
| Redis read latency | 0.1 ms | disk re-hydration 107 ms for 133 turns |
| Redis memory | 1.38 MB | DB 0 and DB 1 were empty at measurement |
| Uncached cost per query | $0.015 | SearchGPT $0.03–$0.10 |
| Huffman ratio | 65–69% on small production archives | zlib-1 about 54–63% at the same sizes |
The $0.015 figure amortizes $96/month of compute plus 10K in / 2.5K out tokens of provider inference. The authors refuse to convert 89.3% into inference avoided: that number includes TTL refreshes, list reads, and existence checks, and Layer 1 is estimated to contribute 75–80% of hits. They observed that 15–20% of in-session queries are near-duplicates (cosine ≥ 0.90), each semantic hit saving 3–8 seconds of wall time, but they did not log request-level hits. No head-to-head with GPTCache.
If you are building a search assistant and do not want to pay a per-query search API, this is a concrete Redis split: a hot session window, a short-TTL semantic skip, and cross-session embedding reuse. Sub-millisecond reads and 1.38 MB of Redis make the operational claim credible.
Treat it as a production note, not an algorithms paper. The 89.3% figure is not nine in ten user queries skipping the LLM.
The authors list the obvious ones: one box, one snapshot; brute-force cosine with n ≤ 50; Huffman at 800 KB/s; archives compressed but not encrypted. The sharper hole is the measurement. At snapshot time both the semantic cache and the URL cache had zero keys, so 89.3% is almost entirely session-layer bookkeeping. Provider rates are a measurement-period snapshot. Skipping a search API by driving headless browsers is cheaper; reliability and terms-of-service risk are not discussed.