Remote Prompt Processing for Low VRAM
segmond · reddit · 2026-07-17
The author proposes a remote prompt processing approach for users with low VRAM: - The real bottleneck for local LLMs is often not generation speed, but slow **prompt processing (pre-processing)**, especially under partial/full CPU offloading scenarios. - The idea is to offload long-context pre-processing to a machine with sufficient VRAM, save the intermediate state, and transfer it back to the host for token generation. Even if the host only generates at 5-10 tok/s, wait times for long prompts drop significantly. - Example: 50k tokens at 10 tok/s takes about 1.5 hours; with a stronger GPU handling prompt processing first, it could finish in under a minute and start generating immediately. - The author envisions a shared service where multiple users share one server just for PP, rigged together via `curl/scp/bash`, ideally packaged eventually as a `--pp-server`.
More from Infra
- Larry Fink says China is ahead in the AI energy race, citing 100 GW nuclear buildout — rohanpaul_ai · 2026-07-21
- Local AI may pay back in 6–7 years and cut long-term costs by 30–40% — DavidLinthicum · 2026-07-21
- TSMC reportedly plans up to 10% chipmaking price hikes in 2027 — kimmonismus · 2026-07-21
- More open models and llama.cpp updates are coming, says Merve Noyan — mervenoyann · 2026-07-21
- Why adding a second LLM provider breaks more than the API surface — Ok_Extension6373 · 2026-07-21
- UK AI datacentres face backlash over heat, noise and land use — nordicinst · 2026-07-21