GP builds local clinical assistant on one RTX 4090: Gemma 4 26B QAT cuts response to ~3 seconds
looneycheetah · reddit · 2026-10-13
A London GP open-sourced OpenConsult (AGPL-3.0), a fully local clinical assistant that listens to consultations, suggests questions and diagnoses in real time, and writes SOAP notes with diarised transcripts — all scripted cases, no real patients. Target: speak the next question within 3 seconds of the patient finishing.
On a single RTX 4090 (24GB) + 96GB RAM + i9-13900K, sharing VRAM with speech and embedding models:
- MedGemma 27B (Q4KM): 18s per pass; Gemma 4 26B-A4B QAT: 5s with equal or better clinical scores
- vLLM hits 3.5s but holds 94% of the card vs 70%; batching the pass's two calls drops Ollama to 3.8s and vLLM to 2.8s
- Fresh context per call works at least as well as a shared consultation context; Gemma 4 beats Qwen3.6-35B-A3B, Nemotron 3.5 Lightning, and gpt-oss-20b
- A pass is two calls (1000 in/160 out, then 800 in/570 out); bottleneck is generation at 170 tok/s
Requirements: never drop serious diagnoses, always emit clean JSON. Author seeks further speedups (speculative decoding, CPU offload), engines to benchmark, and model alternatives.
More from coding & agent
- LangChain Interrupt London: Harrison Chase keynotes on owning your intelligence — marlene_zw · 2026-10-13
- Why MMR keeps returning duplicate chunks to your agent: check candidates_limit first — izgorodin · 2026-10-13
- Survey: Evaluating LLM Outputs Is the Top Bottleneck for 49% of NLP Builders — _inesmontani · 2026-10-13
- UX Without a UI: Designing Terminal Experiences for an Agent-First Product — dl_weekly · 2026-10-13
- Stalkr adds AI translation to flag and translate foreign-language brand mentions — marclou · 2026-10-13
- LangChain Interrupt: visual agents at DB and mission-critical agent deployments — LangChain · 2026-10-13