Dev strips decoding from Qwen 3.5 9B to serve token probabilities locally at 100ms
telltaleatheist · reddit · 2026-09-23
A Reddit user shared "jev", a local tool that cuts off the decoding stage of qwen 3.5:9b, retrieves the raw softmax output, and returns token probabilities as a structured JSON payload. The author reports 100ms responses running fully locally for free, and notes the vision tower remains usable for image analysis. It targets use cases that need probability distributions rather than generated text.
More from Models
- MiMo paper reveals 1.27M RL trajectories, but V4.1 Flash still looks better-baked — teortaxesTex · 2026-09-23
- GPT-6 Sol reportedly worse than 5.6 Sol on DeepSWE and computer use; mocked as rebranded Terra — kristoph · 2026-09-23
- Mini benchmark: Opus 5.5 draws better but burns 10x more tokens than Astra — OfirPress · 2026-09-23
- Author says $20 Claude plan can't finish a single work session without hitting limits — burkov · 2026-09-23
- Opus 5 scored 53.4% on FrontierCode at launch, but its new card shows just 48% — andrew_n_carr · 2026-09-23
- 10 Wild Demos Show Claude Opus 5.5 Building 3D Games and Animations Nonstop — minchoi · 2026-09-23