1.5B model makes 31ms decisions on a laptop GPU, beating 4B rivals
Cod3Conjurer · reddit · 2026-11-19
A developer fine-tuned Qwen2.5-1.5B into QJev-1.5B on a single RTX 5060 laptop using 655k samples: the model routes Qwen's final hidden state into an MLP head that outputs calibrated probabilities over up to 10 choices in one 31ms forward pass, replacing 400ms autoregressive decoding.
Progressive training: freeze backbone and train the MLP head first; unfreeze attention projections via QLoRA for joint training (5.9 GB peak VRAM); consistency-tune with soft-target KL loss across permutation pairs to remove format bias.
Key claims: 10–20× faster than autoregressive inference, zero schema failures (raw floats, no JSON parsing), 1.07 GB VRAM in 4-bit or a 20 MB LoRA adapter, native Ollama/vLLM/llama.cpp/FastAPI support.
Benchmarks: 62.53% on the strict official benchmark (tol=0.05), rank 1 in the sub-2B bracket and beating 4B models like jiwo-4b; 71.50% consistency, close to 9B (72%) and 27B (76.5%) models while 18× smaller and 6× faster. Weights and code are open on Hugging Face and GitHub.
More from Models
- FoodTruck Bench, an LLM agent business-sim benchmark, lands on Papers with Code — NielsRogge · 2026-11-19
- GPT's Ultrafast mode is 'really good' but burns through tokens, says dev — lxfater · 2026-11-19
- Hermes adapter solves a SWE-Bench Pro task in 35 tool calls where the base Qwen3.8 model failed — yb2698 · 2026-11-19
- NousResearch's Hermes adapter lifts Qwen3.8-27B agent scores: SWE-Bench Pro 29.4%→43.1% at 4× fewer tokens — yb2698 · 2026-11-19
- Gemini 4 Neon hailed as 'most tasteful model ever' for its essay on thriving in the AI age — daniel_mac8 · 2026-11-19
- GPT-6.2 Astra's Backlash vs Its Efficiency Numbers: What OpenAI's Enterprise Push Signals — The AI Daily Brief · 2026-11-19