Local Inference of 91GB Audio Model: 127GB RAM Needed for 1M Context
andimarafioti · x · 2026-08-01
A developer shared real-world hardware metrics for running a massive audio model locally. The model uses 91GB of weights and requires 127GiB of resident memory at its default 1M context length. Capping the context at 32k reduces memory usage to 93GiB.
In terms of performance, audio encoding takes 575ms, while generating an answer takes 103 seconds. The test was conducted using llama.cpp (Metal backend) with Unsloth quants.
More from Infra
- NVIDIA quietly delists the entry-level RTX 5050 from its marketplace, following the RTX 5090 — chemist_slime · 2026-10-16
- Grok claims 816B MoE model runs locally on dual M5 Ultra Mac Studios via MLX — grok · 2026-10-16
- Same 1,000 Cloudflare Durable Objects: $4,050/month cut to $22 with one trick — threepointone · 2026-10-16
- NVIDIA ships FP4-quantized Qwen3.8-Flash for local use, trending on Hugging Face — nvidia · 2026-10-16
- Baidu's Virtual Head Attention tames the KV cache, FLOPs, quality triangle in long-context LLMs — baidu · 2026-10-16
- DuoDiscover agents auto-build fast eval proxies, scoring 2.68x TPU kernel speedup — berkeley_ai · 2026-10-16