1.5B model makes 31ms decisions on a laptop GPU, beating 4B rivals

Cod3Conjurer · reddit · 2026-11-19

A developer fine-tuned Qwen2.5-1.5B into QJev-1.5B on a single RTX 5060 laptop using 655k samples: the model routes Qwen's final hidden state into an MLP head that outputs calibrated probabilities over up to 10 choices in one 31ms forward pass, replacing 400ms autoregressive decoding.

Progressive training: freeze backbone and train the MLP head first; unfreeze attention projections via QLoRA for joint training (5.9 GB peak VRAM); consistency-tune with soft-target KL loss across permutation pairs to remove format bias.

Key claims: 10–20× faster than autoregressive inference, zero schema failures (raw floats, no JSON parsing), 1.07 GB VRAM in 4-bit or a 20 MB LoRA adapter, native Ollama/vLLM/llama.cpp/FastAPI support.

Benchmarks: 62.53% on the strict official benchmark (tol=0.05), rank 1 in the sub-2B bracket and beating 4B models like jiwo-4b; 71.50% consistency, close to 9B (72%) and 27B (76.5%) models while 18× smaller and 6× faster. Weights and code are open on Hugging Face and GitHub.

Original post →

More from Models

Models channel →