Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs
Luka Ribar, Jeevan Bhoot, Douglas Orr
cs.CV, cs.LG
2026-08-21
Llama 3.2 11B Vision is distilled into 2.7-bit S3D8 at 3.7 GB. Four VQA tasks average 0.661 vs 0.744 bf16; Pixel 8a decodes at 3.8 tokens/s.
A vision-language model on a phone is mostly a memory problem. Llama 3.2 11B Vision Instruct occupies about 21.3 GB in bfloat16. Quantization shrinks that, but below 3 bits, casting or GPTQ collapses quality. Quantization-aware training (QAT) recovers more, and it usually wants the original training mix, which release checkpoints do not include.
Graphcore Research and Arm take both constraints at once: compress below 3 bits in a layout that Arm CPUs can decode quickly, and do it without the original data.
S3D8 stores three signed weights in one byte: a shared 5-bit centroid index plus three sign bits, about 2.67 bits per parameter, with bfloat16 per-channel scales. The codebook is 32 absolute 3-vectors; the eight orthants come from reflecting those centroids by sign. Decode is built around Arm TBL lookups. Five logical SIMD ops build three table indices and read signed INT8 values, then channel scales are applied. Weights that share a byte belong to consecutive output channels, which fits a fused dequant-dot kernel.
Centroids are fit once with Lloyd-Max before QAT and then frozen. During QAT the forward pass snaps each scaled 3-vector to the nearest centroid; gradients use a straight-through estimator. The student runs quantized weights. The teacher is the original bfloat16 model. The loss is KL on response tokens only.
Images come from the ImageNet training set, about 1.28 million photos. The teacher answers prompts sampled from a generic pool: the instruct template is applied with probability 0.75, an extra instruction block with probability 0.7, and length probes biased toward short and medium answers. No downstream benchmark images or questions are used. A single fixed prompt, "Describe the image:", is markedly worse under the same QAT budget.
Evaluation uses fixed 1,024-example slices of VQAv2, ChartQA, DocVQA, and AI2D. The first three plus AI2D report accuracy; DocVQA reports ANLS. Prompts and splits do not match the official model card, so numbers are for relative comparison. All language and vision linear weights are quantized except 1-D parameters. Direct-cast and QAT use channel-scaled INT8 activations; GPTQ keeps bfloat16 activations.
| Format | Size | bpp | Avg |
| bfloat16 | 21,340 MB | 16.00 | 0.744 |
| S3D8 + QAT | 3,569 MB | 2.68 | 0.661 |
| INT + QAT | 3,620 MB | 2.71 | 0.347 |
| student-t + QAT | 3,620 MB | 2.71 | 0.565 |
| lloyd-max + QAT | 3,459 MB | 2.59 | 0.436 |
S3D8 drops 0.083 from the dense average. Rate-matched INT QAT only reaches 0.347. Per task, S3D8 scores 0.702 / 0.648 / 0.740 / 0.554 against 0.754 / 0.747 / 0.844 / 0.631; document ANLS is the steepest drop. At about 2.7 bits, GPTQ+S3D8 averages 0.340 versus 0.018 for GPTQ+INT. QAT then lifts S3D8 to 0.661. The paper reports about 22% extra compression versus INT at matched task score.
The packed file is 3.73 GB including the vocabulary. On a Pixel 8a, one image tile, a short prompt, and 100 decode tokens run at a median 3.8 tokens/s. INT8 does not fit in that phone's memory. On Graviton4, S3D8 reaches 36.8 tokens/s versus 26.4 for INT8. Generation (m=1) is bandwidth-bound and favors S3D8; prefill and vision encoding are compute-bound and match or trail INT8. Default QAT is 2,048 steps at batch 128. Keeping the LM head or vision encoder in INT8 grows the model to 4.5 GB and adds only 0.006 average.
This is a reproducible on-device recipe: multimodal QAT without original data, and a numeric format designed around the Arm decode path rather than a grid that later needs a kernel. 3.7 GB and 3.8 tokens/s on a mid-range Android CPU is enough to run an 11B VLM at all. The cost is explicit: 0.083 off the average, and document ANLS from 0.844 to 0.740, a larger hit than VQAv2.
Only Llama 3.2 11B Vision Instruct is quantized, and only on CPU. S3D8 assumes Arm's 64-entry table lookup; other ISAs need new kernels. The eval is four VQA slices of 1,024 items from the original suite, with no long-form, tool use, or multi-image tests. Prompt and image choice can still be tuned. Mixed-precision layers did not improve the accuracy-size curve in their trials. Teacher sampling uses temperature 0.6, so the student copies this teacher's answer style; a new backbone needs a new synthetic set.