Running Local LLMs Natively in Java 22
blintz · hn · 2026-07-15
The author developed a solution to run local LLMs within the JVM, using OpenJDK Panama FFM to interface directly with llama.cpp instead of relying on a traditional REST sidecar.
Key implementation details:
- Shifted from a sidecar to direct native library calls
- Wrapped libargus.cc to provide a cleaner structured ABI on the JVM side
- Continues to interface with llama.cpp, whisper.cpp, and ggml via Panama
- Achieves zero allocations on the hot path; memory for prompts and tokens is allocated only once
- Ensures safe access by mapping native structures and matching compiler padding
- Pre-compiled native binaries are packaged into the jar for easy deployment
The author notes that this engine is the foundation for an upcoming spatiotemporal memory layer (L-TABB) to replace RAG, and welcomes technical feedback on Panama or low-latency systems.
More from Infra
- NVIDIA publishes Vera CPU architecture details before AMD’s AI event — ryanshrout · 2026-07-22
- oMLX 0.5.2 adds Mac menu-bar stats, low-bit decode kernels, and faster downloads — awnihannun · 2026-07-22
- Strangeworks launches Aura to turn enterprise ops into production optimization systems — whurley · 2026-07-22
- Graph workload 854.graph500 enters SPEC CPU 2026 as a new CPU benchmark — Prof_DavidBader · 2026-07-22
- HilbertRaum open-sources a fully local AI chat and document analysis app for private use — Vladowski · 2026-07-22
- Hybrid and local inference are emerging as a response to AI energy and token costs — dmitry140 · 2026-07-22