Count document passes, not tokens: batching questions cuts LLM scoring cost
petrusenko_max · x · 2026-10-31
The costly part of scoring content with many questions is re-reading the document repeatedly, not the model itself. Batching questions into a single pass cuts both cost and latency. Builders should count document passes, not just tokens per answer, when choosing a chat stack. The post references the author's open-source project spinfinc/spinf-serve, which does bulk classification and decision inference on one GPU by reading content once and scoring every question about it.
More from coding & agent
- Creator builds shipwreck diving game with Opus 5.5 and Astrocade MCP — _amirabs · 2026-10-31
- GitHub Universe talk: 'Build once, run on any agent' with slides and skill files — 0xkarasy · 2026-10-31
- Claude can build pro software for ~$20k in tokens — and TAM shrinks to match — bigblueboo · 2026-10-31
- Microsoft launches Copilot harness in Foundry Agent Service for managed agent loops — WirelessLife · 2026-10-31
- Codex Remote live activities go live in the ChatGPT iOS app for tracking agent tasks — SIGKITTEN · 2026-10-31
- Open-source pixel FX bring a 3D wizard battle to life with slimes, archers and a golem — majidmanzarpour · 2026-10-31