TensorSharp's logit-reading approach beats LocalJev at structured decisions, 3.3x faster
fuzhongkai · reddit · 2026-09-23
The developer of open-source TensorSharp, inspired by vLLM PR #57250, added a native /v1/systemone endpoint for Jev-style structured decisions using DiffusionGemma GGUF weights, and benchmarked it against the original LocalJev engine.
The key difference: TensorSharp reads requested label logits directly, while LocalJev builds a larger chat prompt and generates probabilities as JSON — average input 192.7 vs 589.6 tokens.
Benchmark (12 cases × 3 reps, 108 expected decisions per implementation):
- TensorSharp: 36/36 valid requests, 108/108 correct decisions, p50 2.877s, p95 3.165s
- LocalJev: 27/36 valid (9 HTTP 422 schema-validation failures), 81/81 correct on valid responses, p50 10.479s, p95 26.167s
- Median latency ratio across 27 matched pairs: 3.345x
The author notes this is an end-to-end comparison, not a kernel-only one, and the small set isn't a general accuracy benchmark. Local repro steps and a Python example are included.
More from Infra
- tinygrad hits ~200 tok/s MiMo-V2.6-Pro on MI300X, brought up via GLM-5.3 — AIFlow_ML · 2026-09-23
- Rumors: 64GB+ VRAM RTX 5090 in R&D but not coming anytime soon — AIFlow_ML · 2026-09-23
- Swarm-built inference engine runs Qwen Image-2.1: 1K images in under 0.5s — bingxu_ · 2026-09-23
- MLX MTP head silently ignored: a 3-line fix boosts Mac local decode speed by up to 79% — Micha0827 · 2026-09-23
- Pluton: open-source self-hosted backup platform wrapping Restic and Rclone for encrypted cloud replication — tom_doerr · 2026-09-23
- AWS Open-Sources Strands Harness, Claims 28% Fewer Agent Tokens — shashib · 2026-09-23