Energy Throttling LLMs via MLP Contextual Bandit
hongyangzh · x · 2026-09-02
The author explores "energy throttling LLMs" to solve the challenge of selecting parameters when running EAGLE-3 speculative decoding with SGLang.
Methodology:
- Track traces from real runs.
- Treat each config as an action.
- Train a tiny MLP contextual bandit model on top of that.
Result: Even a small model with some classic RL ideas is sufficient to pick EAGLE-3 params based on the energy the GPU can actually sustain, instead of pushing speculation blindly.
More from Infra
- Speculative Decoding: Accelerating LLM Inference via Rejection Sampling — jbhuang0604 · 2026-09-02
- Podcast: Why you can't just rent GPUs, why inference is outdated, and HF hack analysis — ziv_ravid · 2026-09-02
- Google Cloud Run integrates Gemini Agent Platform with Identity and Registry — steren · 2026-09-02
- 27.5% of AI CAPEX goes to power and cooling analysis — NaveenGRao · 2026-09-02
- AI Data Center Boom Sparks Demand for Rare Earths Erbium and Yttrium — 量子位 · 2026-09-02
- FORGE Framework Released: Top 10 AI Data Center Security Risks — AccBalanced · 2026-09-02