Gufo, an open-source Strix Halo inference engine, explains its hybrid model caching design
fedeizzo · reddit · 2026-10-20
Gufo is an MIT-licensed open-source AI inference engine for AMD's Strix Halo chip, started as a pet project by two friends to exploit the chip's large unified memory and lower bandwidth while supporting a few best-in-class models. Released about a month ago, it has gathered 829 GitHub stars.
The team is publishing a blog series on engine design decisions; the first post covers caching and how they built a custom solution tuned to Strix Halo's memory-heavy, bandwidth-constrained profile. The authors stress the articles are hardware-agnostic and cover concepts common to many inference engines, making them useful for anyone working on local deployment and inference optimization.
More from Infra
- doubleAI, co-founded by Mobileye CEO, claims its model cuts compute costs by 67% — pstAsiatech · 2026-10-20
- US sanctions 5 Chinese entities tied to ex-Google engineer who stole TPU cluster secrets — rohanpaul_ai · 2026-10-20
- AI chip startup Tenstorrent raises $1.5B at $6.2B valuation after takeover interest — pstAsiatech · 2026-10-20
- Fresher Lemonade: CLI tips for pulling, configuring and benchmarking local models — admcpr · 2026-10-20
- BigMoeOnEdge streams 91B+ MoE models on 12GB phones and 16GB Macs, hitting 25-30 tok/s — dai_app · 2026-10-20
- NVIDIA Brev lets you add your old gaming PC to a unified GPU console alongside cloud instances — victorialslocum · 2026-10-20