Baseten's joey-1 hits 20ms P50 latency at $0.01 per million tokens
baseten · x · 2026-10-30
Baseten's blog explains why decision models exist: many LLM workloads need decisions (route this ticket, flag this message, approve this request), not generated text — and generating text adds cost and latency that compound across chains.
Key numbers for joey-1 (2B):
- 20ms P50 end-to-end latency on 2,000-token requests, 2.5x faster than elephant-1 (49ms) and 5x faster than comparable models
- $0.01 per million tokens, one-tenth of mammoth-1
- Strongest decision model at ≤3B params in internal testing; #1 in its class on Jev Decision Index 0.4
The family now spans mammoth-1 (27B), elephant-1 (9B), and joey-1 (2B), all accepting text and images.
More from Infra
- zml/llmd ships Day 0 support for Mistral Large 4, GLM 5.2/5.3 and LFM 3 — qtnx_ · 2026-10-30
- Google's Ed Chi asks: will future phones be thin clients or run more on-device intelligence? — edchi · 2026-10-30
- UkisAI cuts thinking tokens 61.4% on GLM-5.3-Flash with under 0.5% accuracy loss — Secure_Recording_472 · 2026-10-30
- tiny-llm: an open-source course where you build a vLLM-style inference engine for Qwen3 on a Mac — techNmak · 2026-10-30
- WanGP one-click launcher hits Pinokio: AI video generation on 6GB+ VRAM GPUs — cocktailpeanut · 2026-10-30
- DeepSpeed fixes ZeRO-2 fp32 precision bug, users urged to upgrade to 0.19.8+ — StasBekman · 2026-10-30