Local Agents Parallelism Test: Batching is Real but Not Free
AIForOver50Plus · reddit · 2026-08-18
Tested whether two local agents hitting the same Qwen 3.8 27B 4bit model on an M3 Max MacBook run in parallel or serially. By using a synchronized load driver, the author confirmed that continuous batching works, allowing agents to share the model. However, concurrency is not free: while total throughput rises with more agents, individual requests get slower and wait longer. The sweet spot was found to be around 4 agents. Notably, pinning a random seed serializes all requests.
More from coding & agent
- Sentence Transformers v6.0 Ships Late Interaction Models, Its Largest Update Yet — tomaarsen · 2026-08-18
- sentence-transformers v6.0 adds ColBERT-style multi-vector late interaction retrieval — tomaarsen · 2026-08-18
- Sentence Transformers v6.0: fp16 + FlashAttention Hits 3.87x Over fp32 — tomaarsen · 2026-08-18
- Sentence Transformers v6.0 Arrives: Late Interaction Lands in Project's Biggest Release — tomaarsen · 2026-08-18
- Breaking Changes in Sentence Transformers v6.0: transformers v5 Floor, API Shifts — tomaarsen · 2026-08-18
- bfloat16 Sigmoid Saturates CrossEncoder Rankings; Fix Lifts nDCG From 0.18 to 0.68 — tomaarsen · 2026-08-18