Orthrus Serves Embedding and Generation in One GPU Batch, 4.52x RAG Throughput
_reachsumit · x · 2026-09-23
An EMNLP 2026 main-conference paper, with code released.
- Problem: iterative RAG mixes embedding and generative models, but serving systems run them on separate GPUs; coarse-grained partitioning fails on dynamic workloads, creating computational bubbles and low GPU utilization.
- Method: Orthrus performs heterogeneous batching in a unified inference loop, unifying conflicting workloads via chunked embedding with incremental pooling and workload-aware batch composition.
- Results: on four A100s, throughput improves 1.28x-4.52x on controlled workloads and end-to-end p99 latency drops up to 55.8% on an iterative-RAG benchmark vs baseline deployments.
More from Infra
- Pluton: open-source self-hosted backup platform wrapping Restic and Rclone for encrypted cloud replication — tom_doerr · 2026-09-23
- TensorSharp's logit-reading approach beats LocalJev at structured decisions, 3.3x faster — fuzhongkai · 2026-09-23
- AWS Open-Sources Strands Harness, Claims 28% Fewer Agent Tokens — shashib · 2026-09-23
- DIY multi-GPU cooling: case airflow tuning drops temps from 80C+ to 68C, no liquid cooling needed — HankYeomans · 2026-09-23
- Alibaba accelerates global AI push with new data centers across Europe and the Middle East — Polymarket · 2026-09-23
- You run kernels, not models: why the same model and GPU can perform wildly differently — Roger_M_Taylor · 2026-09-23