From static to polymorphic inference: Together's case for continual compilation
simran_s_arora · x · 2026-11-05
In the framing post for the Together Compiler, simransarora argues agents hill-climb effectively on well-specified targets, but modern inference's optimization target—traffic shape—changes every second: gradual drift as users learn new models, burstiness in request volume/type, and temporal shifts as users in different countries use models differently. This demands moving from static to "polymorphic" inference. On OpenRouter, 80% of traffic goes to models released in the last three months and each of the top ten open models uses a different architecture—motivating continual compilation (thread continues).
More from Infra
- Kimi K3.1 Pro (2.8T params, 104B active) lands on Ollama Cloud with zero data retention — ollama · 2026-11-05
- Hugging Face shares its model optimization pipeline: profile, torch.compile, ship open kernels — ariG23498 · 2026-11-05
- Reddit proposal: use agents to digest fragmented custom inference engines into reusable building blocks — nasone32 · 2026-11-05
- LZ4 v1.11.0 ships multithreaded compression, hits 7,550 MB/s with 16 workers — jedisct1 · 2026-11-05
- Engineer praises turbopuffer as community 'sandbox tier list' makes rounds — davidcrawshaw · 2026-11-05
- Indie pretraining run hits 10T tokens of data, locks in Apple M5 and DGX Spark support — jon_durbin · 2026-11-05