jinfer brings native AI inference to the JVM, matching llama.cpp on CPU with zero Python
mukel90 · reddit · 2026-09-15
Developer mukel90 (author of llama3.java and gemma4.java) released jinfer, an Apache 2.0 open-source inference engine built for the JVM from first principles: chat, vision, audio transcription, embeddings, reranking, and TTS—with no Python runtime, ONNX, or sidecar process. The stack includes toknroll (pure-Java zero-dependency tokenizers), native gguf/safetensors read/write, jam (quantized matmul routines with Vector API + optional native backend, competitive with llama.cpp on CPU), and jota (a tensor API targeting Java, C, CUDA, HIP, Metal, OpenCL, and Mojo). It integrates with Spring AI and LangChain4j and supports GraalVM Native Image. It's an early release: CPU is the main target today, GPU support is in progress. Benchmarks and runnable examples at qxotic.ai.
More from Infra
- Tuning SGLang on a single 5090 for Qwen3.8-27B: 100 tok/s but only 82k context — ni1by2thetrue · 2026-09-15
- Four dev boards hooked to the internet: test AI-written firmware on real silicon via HTTPS — SelfishlyWandering · 2026-09-15
- Grouped Value Attention shrinks KV cache by reconstructing keys on demand — Vishesh Tripathi · 2026-09-15
- Wan 2.2 on one RTX 5090: frame count doesn't touch VRAM, but resolution drops it by 10GB — Realistic-Fennel-190 · 2026-09-15
- Stateless LLM failover preserves ~0% context; ContinuityBench proxy hits 99.20% CPR — its_vayishu · 2026-09-15
- 500 trillion tokens in a week: chart points to surging LLM usage — gajesh · 2026-09-15