AI infra engineer project list: vLLM tuning, KV-cache routing, Triton kernels
snowclipsed · x · 2026-10-12
The retweet itself is a joke — an "AI Native gateway" that filters slop on Twitter, supposedly cutting TTFBT ("time to first brain tumor") by 98%.
The quoted thread is a serious project list for AI infrastructure engineers:
- High-throughput serving tuning: deploy vLLM or SGLang and tune chunked prefill, continuous batching and max-num-seqs for a specific GPU architecture — default configs waste 60% of VRAM.
- KV-cache prefix routing proxy: hash incoming system prompts and route requests to the replica holding that KV-cache block in memory; reusing context can slash TTFT by 80%.
- Custom fused Triton kernel: write a fused RMSNorm or RoPE kernel (thread truncated).
More from Infra
- Meituan LongCat's QuantMLA: joint INT4 MLA KV caching with 3.59x compression at 128K — meituan-longcat · 2026-10-12
- Engy dev: token counts are fun, we also rent 221 GPUs above cost — jon_durbin · 2026-10-12
- Microsoft unveils Maia 300 chip, to make in-house silicon its primary inference system in multi-gigawatt 2027 scale-up — satyanadella · 2026-10-12
- How One Dev Rents a 64-Core, 256GB Box for $800/Month to Run 30 Parallel AI Subagents — TheZachMueller · 2026-10-12
- Microsoft unveils Maia 300 AI chip, its primary inference system rolling out 2027 — The Verge AI · 2026-10-12
- NuMind open-sources vLLM plugin: free multimodal embeddings from your already-loaded LLM — Gailenstorm · 2026-10-12