GCMP update: LoRA-quality at 2.9M params, serves 256 adapters on one A40 with zero throughput drop
textclf · reddit · 2026-11-21
Author shares full benchmarks for GCMP, a LoRA replacement training just 2.9M params (11 MB) on Qwen3.8-27B.
Quality: statistically tied with LoRA r=8, r=32 and VeRA across 9 benchmarks (paired McNemar tests, all p ≥ 0.15). Since r=8 matches r=32, GCMP is 14x smaller than LoRA r=8 (not 55x as earlier claimed). VeRA is the same size and quality and trains 2x faster, but vLLM can't serve it.
Serving many adapters (one A40 48GB, vLLM 0.27.1, CQ 4-bit base, 32 concurrent requests, random adapter each):
- GCMP stays flat at 83.7 tok/s from 1 to 256 adapters, p95 TTFT 2.2 s, all 256 loaded in 6 min.
- Default LoRA drops 35-40% throughput by 64 adapters with p95 TTFT 50 s; loading 256 takes 30-77 min.
- LoRA r=32 can't even start with 32 slots on 48 GB (negative KV cache memory).
CQ 4-bit base matches BF16 on all 9 benchmarks while NF4 loses 12 points on GSM8K. Main limitation: GCMP trains 2x slower per step; next up is faster CQ decode kernels.
More from Infra
- Hands-on: AMD ROCm 10.2 delivers 2.2x token throughput on Qwen3.8-27B — AnushElangovan · 2026-11-21
- OpenVINO: the open-source toolkit for running AI on Intel CPUs, GPUs and NPUs — abhishek__AI · 2026-11-21
- AWS Lambda now supports GPUs via EC2 capacity providers — samgoodwin89 · 2026-11-21
- 27,262 thinking tokens, zero answer: Google auto-billed 23 empty Vertex AI crashes via HTTP 200 — Plenty-Jackfruit-218 · 2026-11-21
- Hy4 Flash reads 1M tokens in 44 min on Mac Studio M3 Ultra, 81 min on DGX Spark — AIFlow_ML · 2026-11-21
- local-ai-registry: one battle-tested local LLM recipe per GPU, with evidence attached — LiTianleli · 2026-11-21