Inference tricks distilled: prune experts + RL, session reuse, TP2×DP2 on one H200
JFPuget · x · 2026-11-04
A Japanese thread distills key points from an inference-engineering writeup:
- Prune experts and restore accuracy with RL — far fewer parameters to update, possibly fitting on a single H200
- Keep sessions alive to avoid recomputing prefill
- TP2×DP2 parallel inference
Part of the post-competition writeups circulating in the ARC contest community; details in the linked piece.
More from Infra
- Big tech has contracted over 10 GW of nuclear power, but contracts aren't generation — ingliguori · 2026-11-04
- Fully Local AI Short Film Rendered on a Mac, Workflow Open-Sourced as Phosphene — Opening-Ad5541 · 2026-11-04
- Anthropic's 15-person chip team uses Claude to design its own AI chips — kimmonismus · 2026-11-04
- Anthropic's 15-person chip team uses Claude to design its own AI chips ahead of $1.5T IPO — kimmonismus · 2026-11-04
- AAOI signs $194M InP wafer supply deal with Yunnan Germanium at ~$1,077 per wafer — zephyr_z9 · 2026-11-04
- Google Cloud taps Tetrate to extend Gemini platform AI routing to multi-cloud and on-prem — IanAndrewsDC · 2026-11-04