Local Test of 276B Multimodal Model: 3-bit Quantization Works, 12B Active Beats 41B

Developers @MaziyarPanahi and @andimarafioti tested the inference performance and practical application of the large multimodal model Inkling-Small (276B A12B architecture) locally. With 3-bit quantization, the model successfully ran audio-video processing on Metal devices and showed surprising medical diagnostic ability, with the 12B active-parameter version even outperforming the larger 41B version.

Confirmed

Model weights are 91GB (3-bit quantization), with system resident memory up to 127GiB at default 1M context length; limiting context to 32k reduces memory usage to 93GiB.
Inputting a 2-minute knee consultation audio for multimodal testing, the model not only accurately diagnosed decompensated heart failure but also clearly identified the cause as the patient stopping furosemide.
@MaziyarPanahi compared and found that the 12B active-parameter version performed better than the 41B version tested two weeks ago—the former not only identified the cause but also proactively requested BNP, ECG, and chest X-ray tests.

Why it matters

This test proves that hundred-gigabyte-scale large multimodal models can be usable on consumer hardware after quantization, and shows unexpected reasoning depth and proactivity in professional medical diagnosis scenarios, providing an important reference for local deployment of multimodal LLMs in vertical domains.

2026-08-01 ~ 2026-08-02 · 5 related posts

Primary sources

Inkling-276B Multimodal Model Local Test: 3-bit Quantization Hits 37 t/s Decode — MaziyarPanahi · 2026-08-02 05:59 (UTC+08:00)
Local Quantized Inkling 276B Model Accurately Diagnoses Heart Failure from Audio — MaziyarPanahi · 2026-08-01 20:32 (UTC+08:00)
Inkling 12B Active Params Outperforms 41B Version in Medical Diagnosis — MaziyarPanahi · 2026-08-01 20:34 (UTC+08:00)

[source] Local Quantized Inkling 276B Model Accurately Diagnoses Heart Failure from Audio — MaziyarPanahi · 2026-08-01
[source] Inkling 12B Active Params Outperforms 41B Version in Medical Diagnosis — MaziyarPanahi · 2026-08-01
Local 276B Multimodal Model Inference: 93GB VRAM at 32k Context — MaziyarPanahi · 2026-08-01
Local Inference of 91GB Audio Model: 127GB RAM Needed for 1M Context — andimarafioti · 2026-08-01
[source] Inkling-276B Multimodal Model Local Test: 3-bit Quantization Hits 37 t/s Decode — MaziyarPanahi · 2026-08-02