Running SigLIP 2 on CPU with Rust+ONNX
kuaythrone · reddit · 2026-07-18
The author is building a robotics data platform handling images, videos, and text metadata simultaneously.
Their approach splits the multimodal embedding process of SigLIP 2 into two pipelines:
- The GPU handles batch, asynchronous image/video embedding and index building.
- A lightweight Rust + ONNX Runtime service processes real-time text queries on the CPU.
Since text and images ultimately share the same embedding space, text can directly search the image index built by the GPU. The author finds this split highly practical: reserving scarce GPUs for high-throughput multimodal ingestion while scaling the query side with standard CPU machines. They are also curious if others are using Rust for similar low-latency inference services.
More from Infra
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27
- NVIDIA claims HGX Rubin NVL8 delivers 10x tokens per megawatt vs B200 — thursdai_pod · 2026-10-27
- NVIDIA Claims HGX Rubin NVL8 Delivers 10x More Tokens per Megawatt Than B200 — thursdai_pod · 2026-10-27
- Together lets multiple LoRA post-training experiments share the same GPU capacity — togethercompute · 2026-10-27
- Commoditization dilemma: either human labor or GPUs become the commodity — _AustinCalvert_ · 2026-10-27