RAGMark: per-stage RAG benchmarking shows reranking plus compression cuts energy 66%
_reachsumit · x · 2026-09-09
RAGMark is a modular benchmarking framework for RAG systems in small multi-GPU setups, profiling retrievers, vector databases, prompt processing, and generators with per-stage latency, GPU utilization, memory, power, TTFT, throughput, and answer quality metrics. Across five QA workloads it shows autoregressive generation dominates latency in naive pipelines, while context-reduction shifts bottlenecks across stages; reranking and compression compound — jointly cutting prefill and KV-cache traversal costs and lowering energy use by up to 66%.
More from Infra
- Same budget: MacBook Air + 48GB VRAM desktop rig vs 64GB MacBook Pro for local LLMs — gappyvalley · 2026-09-09
- NVIDIA launches CUDA Rust: two tracks to write GPU kernels in Rust — ducha_aiki · 2026-09-09
- OpenAI and Samsung are jointly developing and producing next-gen AI chips — rohanpaul_ai · 2026-09-09
- Hetzner VPS bills jump 543% in 2 years, from €5.30 to €34.09 per month — ThePeterMick · 2026-09-09
- Harbor OSS now processes 80T tokens per week, edging past OpenRouter's 70T — simonguozirui · 2026-09-09
- Supabase's Multigres hits 100% pass-rate on major Postgres test suites for PB-scale — dshukertjr · 2026-09-09