Show Reddit: ge, a Rust GGUF engine, benchmarks against llama.cpp on a 4GB RTX 3050
Raptor2496 · reddit · 2026-11-12
A Redditor built ge, a pure-Rust GGUF inference engine (4 MB single binary, own int8/AVX2 CPU kernels, Vulkan GPU kernels, automatic CPU/GPU split, built-in OpenAI/Anthropic-compatible server) and benchmarked it back-to-back against llama.cpp b9102 on a 4GB RTX 3050 laptop:
ge wins: CPU prompt processing (84 vs 59 tok/s on Qwen3-4B at 512 tokens), RAM peak (2.4 vs 4.7 GB for 8B hybrid), lossless default 8-bit KV cache (6.4 vs 3.6 tok/s at 8K on CPU), and 37 vs 22 tok/s generation when 16K context overflows VRAM.
Ties: CPU generation (RAM-bandwidth-bound), 8B hybrid generation.
llama.cpp wins: GPU prompt processing (1282 vs 968 tok/s via cooperativematrix2) and ecosystem breadth.
Caveats: one machine, source not yet public, license undecided. Author asks others to run ge bench-all on AMD/Intel GPUs and AVX-512 CPUs.
More from Infra
- AMD runs MolmoAct2 robot policy locally on a Ryzen AI Max+ 395 mini PC — DJiafei · 2026-11-12
- Opinion: AI infra is becoming a coding runtime as intelligence units outgrow tokens — prateekj · 2026-11-12
- PrefixRelay: open-source proxy gives Cursor 97%+ KV cache hits on local models — jjusko20 · 2026-11-12
- Kinetic, a Rust inference engine, answers questions on a 22.6k-token doc in 0.13s vs llama.cpp's 10.95s — jasonkneen · 2026-11-12
- HP starts selling $132,000 GB300 desktop with 748GB memory for local AI — gnukeith · 2026-11-12
- Dev runs out of GitHub Actions minutes, shows off Mac Studio with unreleased M5 Ultra chip — ow · 2026-11-12