Local 276B Multimodal Model Inference: 93GB VRAM at 32k Context
MaziyarPanahi · x · 2026-08-01
MaziyarPanahi shared real-world performance metrics for running the Inkling 276BA12B multimodal model locally (3-bit quantization).
At the default 1M context, the model requires 127GiB of resident memory; capping it at 32k reduces this to 93GiB. The weights take up 91GB, audio encoding takes 575ms, and generating an answer takes 103s. He recommends capping the context length for local runs.
More from Infra
- China's AI boom drives up to 660% profit surges for domestic chip firms; GPU maker MetaX turns profitable — pstAsiatech · 2026-10-16
- When Does an Agent Actually Need an API Gateway? Reddit Weighs Direct Calls vs. Unified Endpoints — TurbulentStiffness · 2026-10-16
- Taiwan to build $4B AI manufacturing complex in Texas to back Nvidia and TSMC — Polymarket · 2026-10-16
- When the KV Cache Is Slower Than the GPU: Remote KV Tiers Backfire in Production — MultiverseComputingCAI · 2026-10-16
- GitHub rewrote the 800k-line Copilot runtime in Rust using AI agents, cutting CPU and memory by over 90% — bibryam · 2026-10-16
- NVIDIA quietly delists the entry-level RTX 5050 from its marketplace, following the RTX 5090 — chemist_slime · 2026-10-16