Project Genesis: CUDA co-processor claims 16-30% LLM latency cut without touching weights
ReportProfessional57 · reddit · 2026-11-15
A Reddit user released Project Genesis, a cache-resident C++/CUDA geometric co-processor (0.51MB runtime) that attaches inline to existing causal LLMs during inference, reducing latency and boosting throughput without modifying host model weights.
- Reported 16-30% end-to-end latency reduction with matching throughput gains and O(1) memory overhead, tested on consumer NVIDIA Blackwell hardware across Qwen, Gemma, Llama 3.2, Nemotron, and Mistral checkpoints
- Ships as a sealed runtime bundle supporting 20 common hidden dimensions (dmodel 576-8192) on 64-bit Linux / WSL2 with NVIDIA GPUs of Compute Capability 8.0+ (RTX 30/40/50, A100, L40S, H100)
- Available on Hugging Face for non-commercial/research use; the author invites users to run with enabletelemetry=True and share their own latency/throughput/ring-buffer results
The claims are self-reported and not yet independently verified, so treat with healthy skepticism until third-party benchmarks land.
More from Infra
- Why No One Can Replicate Jensen Huang's Asian GPU Supply Chain — _onionesque · 2026-11-15
- Running 3 AI agents in parallel: even 32GB machines hit memory limits — PawelHuryn · 2026-11-15
- Curated 10-resource path to learn LLM distributed training from scratch — East-Muffin-6472 · 2026-11-15
- Report: Nvidia's interim CEO reviews AI customer financing, $300B OpenAI Ohio chip deal in doubt — SumitGup · 2026-11-15
- Skip the $3,000 Mac mini: a $1,000 renewed Dell micro PC works fine for agents — PawelHuryn · 2026-11-15
- Pirate Face mirrors open models as torrents, already indexing 28,890 magnet links — zainhas · 2026-11-15