Building a Local Inference Server with $100 GPUs
Boricua-vet · reddit · 2026-07-12
The post shows a configuration to piece together local inference capability with $100-level GPUs, targeting 20GB VRAM, 448GB/s bandwidth, and support for 3 concurrent users.
Logs show the author loading Qwen3.6-35B-A3B-UD-IQ4XS.gguf on llama.cpp / llamaserver, encountering:
- ngpulayers too high causing parameters to not fit VRAM
- Context length nctxseq 32768 < nctxtrain 262144, cannot fully utilize training context
- Prompt cache, context checkpoints, speculative decoding and other server capabilities enabled
Overall, it discusses how to make a usable local model service with low-cost hardware, exposing real-world deployment issues with memory, context, and cache configuration.
More from Infra
- SpaceX strikes $40B deal with Apollo Global to lease AI chips and data center hardware — Polymarket · 2026-10-26
- India Needs at Least 5GW of Compute by 2028 to Stay Relevant, User Argues — EAccelerate_42 · 2026-10-26
- SpaceX AI revenue up 830%, Claude designs 318K antivenom proteins in 64 hours — PeterDiamandis · 2026-10-26
- Meta plans up to $95 billion stock sale to fund AI buildout — The Decoder · 2026-10-26
- Pandora M1: first prototype of a physical device for distributed GPU monitoring — tarantulae · 2026-10-26
- INT21's AI agent swarm rewrote Drama 3's inference runtime in Rust within one hour, beating SGLang-Omni — bingxu_ · 2026-10-26