Elastic's VP of AI: Treat search as test-time compute, latency is a currency to spend
gaganghotra_ · x · 2026-11-06
In his ElasticON AI SF 2026 talk "Search is test-time compute", Elastic VP of AI Han Xiao argues that since BERT, search has improved by training harder and bigger — but LLM-era test-time scaling opens a new axis: inference latency isn't a KPI to minimize, it's a currency/budget you can spend to buy relevance and new capabilities. He cites Noam Brown (OpenAI): letting a poker bot think 20 seconds matched scaling the model up 100,000x.
On frozen encoder weights, raising the budget from 10ms to 100ms buys three things:
- Deeper pass: read every token/patch vector of the pass already run — new info from inside the pass;
- More passes: re-run the encoder on new views like sentences or image crops — new info from the input;
- Calibration: transform existing vectors via whitening, label propagation, or optimal transport — new info from a global prior.
Four examples: better score functions on a frozen encoder, an open-vocabulary tagger for photos, reranker-based semantic highlighting, and agents building reusable small search pipelines for long-horizon tasks. He also distinguishes TTC from training-free methods: training-free is the origin; TTC studies whether more compute budget buys more.
More from coding & agent
- Skillware framework claims 97.3% cost cut as 3B-8B local models beat frontier LLMs — RossPeili · 2026-11-06
- A notebook app turns Claude into a tutor on reMarkable and iPad — tokentrillionaire · 2026-11-06
- Measured: in-process BM25 memory costs ~4.6KB RAM per entry, linear growth — Neither-Witness-6010 · 2026-11-06
- Tip: add a phone preview screen to make mobile vibe coding painless — yangyi · 2026-11-06
- Building agents: life-oriented companions beat productivity tools in real use — yangyi · 2026-11-06
- ThinkyMachines' Inkling 2 Small trains models on its own: $426 of compute doubles Qwen score in 4.5 hours — Scobleizer · 2026-11-06