Heterogeneous prefill-decode split sparks debate after engineer reveals 2023 trial

Engineers debated combining speculative decoding with disaggregated prefill/decode using heterogeneous models; developer bingxu noted he already tried 70B prefill plus 7B decode in 2023 and it proved useless. He now jokes his role since 2024 has been 'prompt engineer and agent foreman,' leaving infrastructure work to agents.

2026-09-03 ~ 2026-09-03 · 3 related posts

• Heterogeneous prefill/decode isn't new: dev tried 70B prefill + 7B decode in 2023, 'entirely useless' — bingxu_ · 2026-09-03
• Engineer tried 70B prefill + 7B decode back in 2023: combining spec dec with disagg was useless — suchenzang · 2026-09-03
• 'Since 2024 I've been a prompt engineer & agent foreman' — bingxu_ · 2026-09-03