Gewell v0.2 runs full-precision EmbeddingGemma2 in 1.13 GiB, boosts 26B prefill up to 74%
stoppableDissolution · reddit · 2026-10-12
Independent developer releases Gewell v0.2, a custom inference engine squeezing full-precision EmbeddingGemma2 (text+image) into 1.13 GiB VRAM / 0.93 GiB host RAM via workspace overlapping, host-memory embedding eviction, and a single cuBLAS handle — with image embedding fidelity (cosine 0.9997 vs reference) far above llama.cpp and vLLM. Also adds 26B-A4B support with 10–74% faster prefill than vLLM and 0.625x context VRAM, plus 20–24% tg gains for 31B at short contexts. Fun finding: FlashAttention made llama.cpp/vLLM slower on the author's machine.
More from Infra
- Liquid AI Launches RIPL: Ultra-Low-Latency Inference Engine With Sub-50ms End-to-End Responses — JosephJacks_ · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Jensen Huang lays out NVIDIA's full Vera Rubin agentic system: NVL72, MGX, CMX and RTX — rohanpaul_ai · 2026-10-23
- ONNX to Transformers.js: running models directly in your browser — JosephJacks_ · 2026-10-23
- MiniMax M3.1 Flash free for 3 days on 0G, with 1M context and 512K output — MiniMax_AI · 2026-10-23
- "We bet on Cloudflare and it paid off": R2 and D1 at billion scale — dinasaur_404 · 2026-10-23