Deep dive: OpenAI's Jalapeno inference accelerator architecture from Hot Chips 2026
bookwormengr · x · 2026-09-12
Silicon Co-Design publishes an advanced case study of OpenAI's Jalapeno general-purpose AI inference accelerator, based on the Hot Chips 2026 presentation by R. Ho, R. Narayanaswami, and C. Leary.
- Positioning: designed primarily for OpenAI's own workloads but generalizable to frontier models; not a magic GPU replacement but a stack of system-level tradeoffs.
- Spec & constraints: architecture derived end-to-end from user experience — LLM request latency, energy per request, and system-level Pareto frontier tradeoffs.
- Architectural observations: theoretical memory roofline exceeds real user token throughput; unpredictable workload characteristics across prefill, speculate, and decode; balancing large KV states on a single chip.
- Latency sources: network interconnect breakdown (SOTA-level detail) and compute stalls from late-arriving operands.
- Methodology: how agentic coding contributed to the shortened chip design timeline.
The first half is free for a limited time; the latter half moves behind a paywall.
More from Infra
- Clover-35B-A3B Quantized to 29.4 GB Runs at ~42 tok/s on a 64 GB Mac — viveky259 · 2026-10-31
- Liquid Agent brings fully local LFM3 agent to Linux with one-command install — JosephJacks_ · 2026-10-31
- A fully local office AI stack: MiniCPM, Qwen3, LFM2.5 and Whisper — HowDevelop · 2026-10-31
- a16z: GPU rental prices up 106% in 3 months while token prices fall 58% — a16z · 2026-10-31
- vLLM nightly adds PLE disk offload, with a telling [Qwen4Exp] tag on the PR — rmhubbert · 2026-10-31
- Solo AI studio: per-minute GPU rentals rendered a 328-frame Blender shot for ~$1 — r0ck3t23 · 2026-10-31