Recycling Wasted Logits from Speculative Decoding for On-Policy Training
ycombinator · x · 2026-08-12
The post discusses how to leverage 'wasted' data in LLM serving stacks to optimize models. During speculative decoding, the target model computes logits on rejected tokens that are typically discarded. However, these represent a free, on-policy training signal readily available from actual traffic.
The author notes that because draft models are often trained on generic datasets, their distributions mismatch specific real-world use cases (like voice or coding agents). This causes the acceptance rate to drift down over time, significantly hurting throughput. By recycling these wasted logits for online training, developers can fix the distribution mismatch and sustain inference speedups.
More from Infra
- The Guardrail Tax: Enterprise AI Safety Overhead Costs More Compute Than Reasoning — vasilisvj · 2026-08-12
- Vinci Physics achieves 3.3B voxel inference, predicting >10B FP64 values with linear scaling — AnneliesGamble · 2026-08-12
- ODS Project Wires Together Ollama and n8n to Turn PCs into Local AI Servers — tom_doerr · 2026-08-12
- Report: Nvidia Partners with Wall Street for $500B AI Financing Effort — SatelliteNetSec · 2026-08-12
- Morgan Stanley Predicts AI HBM Consumption to Hit 50B GB by 2027 — zephyr_z9 · 2026-08-12
- Nvidia RTX 50 Series GPUs See Massive Price Hikes Globally Amid VRAM Shortages — sujingshen · 2026-08-12