pplx-decider packs multiple questions per state into one bidirectional forward pass
antoine_chaffin · x · 2026-10-17
Instead of caching per-question, the team packs multiple questions about the same state into a single fully bidirectional forward pass: all questions share one sequence with one readout marker each, and the state is shared across questions, acting like a cache.
With a single question it's equivalent to v1.1. Thanks to the Qwen architecture, very long inputs (multiple packed questions) don't blow up memory, and it's a large speed-up versus no caching. Cross-encoding related questions together also proves beneficial — the gain quantified in the accompanying table (+11.3 on SATA-Bench etc.). Later posts note the tradeoff: results depend on packing, and conflicting questions can hurt.
More from Infra
- Generating 4K 60FPS Video on Just 16GB of VRAM, According to a Reddit Share — LanceCampeau · 2026-10-17
- YC-backed Oway launches Internet of Industrials protocol, first model cuts freight costs ~30% — ycombinator · 2026-10-17
- Visko partners with Google Cloud to scale real-time world generation on TPUs — _vztu · 2026-10-17
- Analyst argues CoreWeave isn't dying, sees Anthropic funding roadshow beating expectations — ivan_bezdomny · 2026-10-17
- AI agents are having their Kubernetes moment: agent logic sinks into the platform layer — bibryam · 2026-10-17
- Why inference providers must choose: fast (Groq) or cheap (CoreWeave), not both — AccBalanced · 2026-10-17