Weaviate ships HFresh, a disk-based vector index inspired by SPFresh for large shifting collections
victorialslocum · x · 2026-09-02
HFresh is now generally available in Weaviate: a disk-based vector index inspired by the SPFresh algorithm, designed for large, continuously changing vector collections with low memory usage.
How it works:
- Vectors are divided into small on-disk clusters called postings — neighborhoods of similar vectors.
- Search runs in two stages: a compact in-memory HNSW index over cluster centroids identifies the most relevant neighborhoods, then only those postings are fetched from disk and searched in detail, with top candidates rescored against uncompressed vectors.
This keeps disk I/O bounded and latency predictable as the dataset grows, with compression built into both layers.
More from Infra
- Quantized MiniMax H3 Turbo generates 5s video in ~1 minute on RTX 4060 Ti — aziib · 2026-09-02
- ComfyUI gains official AMD ROCm support on Windows for image and video generation — SeparateIntern5655 · 2026-09-02
- MLSecOps framework maps 10 security pillars for production ML systems — goyalshaliniuk · 2026-09-02
- ODS turns your PC into a private AI server with Ollama, n8n and ComfyUI pre-wired — tom_doerr · 2026-09-02
- Anthropic locks $35B Lambda deal as Nvidia signs 15-year lease to underwrite the cluster — unconventionalbook · 2026-09-02
- antirez: DeepSeek v4 Flash is still the king of local inference, now with vision — antirez · 2026-09-02