Two days tuning one Tesla P40: 35B MoE with full 262k context at ~60 t/s
sargetun123 · reddit · 2026-10-28
The author spent two days squeezing a single Tesla P40 (24GB, PCIe x1, Windows, custom llama.cpp fork) to run Ornith 1.5 35B-A3B (Qwen3.5 MoE arch) at full 262k context without going below Q4 weights or q8 KV: 60 t/s short decode, 30 t/s at 100k depth.
What worked:
- ECC off: +1.5GB VRAM, 9% decode
- q80 KV instead of q40: q4 KV degrades with depth (30.6 vs 42.2 t/s at 32k)
- Checking the MTP head: the model's first upload had an untrained MTP head (blk.40.nextn.sharedheadnorm 1.02 = untrained, 2.93 = trained); after fixing, draft acceptance went 45% → 60-85%
- int8 dp4a for the KQ half of the flash-attention tile kernel: prefill +26% at 32k, +41% at 100k on tensor-core-less Pascal
- Limiting the MTP prefill window to the last 8k of the prompt: 121k prompt prefill 327 → 373 t/s, output unchanged
What didn't: f16 KV, in-kernel V dequantization (-5%), official Q4KM (same quality, slower, less context). Full llama-bench table and depth-wise throughput curve included; patches available on request.
More from coding & agent
- Opus 5.5 as orchestrator plus DeepSeek Flash as executor: a two-model agent combo gaining fans — samsja19 · 2026-10-28
- LangChain ships langchain-anthropic 1.7.7 with tracing fix and dependency updates — caolonghua · 2026-10-28
- MPO: turn-scale credit assignment for long-horizon LLM agents — IQuestLab · 2026-10-28
- Meta FAIR's h1: simple agentic loop hits SOTA in formal theorem proving — meta-llama · 2026-10-28
- Codexmaxxing on the Duo is real, confirms well-known iOS developer — Dimillian · 2026-10-28
- Two free deep-dives: multimodal RAG bottlenecks in order and shipping agents in a real product — ubiquitous_tech · 2026-10-28