Run Local LLMs Directly Inside Java 22
KingJoker · hn · 2026-07-14
The author introduces a solution for running local LLMs directly within the JVM, leveraging OpenJDK Panama FFM to interface with llama.cpp natively instead of relying on a traditional REST sidecar.
Key implementation details:
- Shifted from a sidecar to direct native library calls
- Wrapped libargus.cc to provide a cleaner structured ABI on the JVM side
- Continues interfacing with llama.cpp, whisper.cpp, and ggml via Panama
- Achieves zero allocation on the hot path, allocating memory for prompts and tokens only once
- Ensures safe access by mapping native structures and matching compiler padding
- Pre-compiled native binaries are packaged into the jar for easy deployment
The author notes that this engine serves as the foundation for an upcoming spatiotemporal memory layer (L-TABB) to replace RAG, and welcomes technical feedback on Panama or low-latency systems.
More from coding & agent
- AI agents are starting to strain code hosting platforms — craigsdennis · 2026-07-21
- Async OPD distillation doubles throughput while matching synchronous math accuracy — _lewtun · 2026-07-21
- Omnigent 0.6.0 adds Claude Code imports, Slack approvals and desktop apps — matei_zaharia · 2026-07-21
- Google appears to have quietly shipped Gemini 3.6 Flash, with lower pricing and better agentic scores — xiaohu · 2026-07-21
- Open-source CLI audits AI tools, MCP configs, and agent skills on local machines — Initial-Copy332 · 2026-07-21
- Coding agents feel less stressful when the 5-hour limits are temporarily removed — iamrobotbear · 2026-07-21