Thinking Machines explains LLM nondeterminism and ships batch-invariant kernels for reproducibility
AccBalanced · x · 2026-10-19
Thinking Machines Lab (Horace He et al.) breaks down why LLM APIs remain nondeterministic even at temperature 0: concurrent execution plus floating-point non-associativity — batched kernels merge requests, so which core finishes first changes results. The fix: batch-invariant kernels enabling deterministic inference. @cHHillee notes this addresses train/inference behavior mismatch, and @Sauers shows a model can be SFT-trained (using layer-34 activations from HF Trainer vs vLLM) to detect whether it's in training or deployment, generalizing from vLLM to SGLang.
More from Infra
- Alibaba Cloud shows Omarchy running on Wuying AI cloud PCs for enterprise agents — AIFlow_ML · 2026-10-19
- NVIDIA paper shows cross-model KV cache transfer skipping prefill, 2.7–25x faster than recomputing — solyarisoftware · 2026-10-19
- pi-durable v1.3.0 adds SQLite table prefixes, fixes 140 MB memory spike on DO — solyarisoftware · 2026-10-19
- Open-source SwiftUI dashboard brings a Docker Desktop-like UI to Apple's native container runtime — AIFlow_ML · 2026-10-19
- Strata runs on Windows with dual AMD GPUs despite no official support: ~1350pp on a 9070XT — Ok-Importance-3529 · 2026-10-19
- A frozen 16MB vocabulary replaces learned embeddings in small and ternary LMs — Sai Kiran Bathula · 2026-10-19