Open-source Profile v2.2 tunes vLLM from 81 to 421 tok/s on a single RTX 5090
Inevitable-Diet-1870 · reddit · 2026-08-19
Profile, an open-source optimizer for inference servers, shipped v2.2: it computes your GPU's roofline ceiling, measures your live vLLM server, names the bottleneck and hands you the flag — then re-measures, turning tuning into deterministic engineering.
This release rewrites the core rule engine as a priority DAG of 8 mutually exclusive rules: five alarms fire, echoes are silenced, one true cause survives — deterministically. AMD GPUs are now supported. The author's benchmark (RTX 5090, muse-glimmer 30B, SWE-Bench agent load, no DFlash speculative decoding): 81 → 421 tok/s at 25k context, $3.41 → $0.65 per 1M output tokens, TTFT 224ms (p95 500ms), and 4.72 → 1.08 J/tok — in 4 iterations, 30 minutes. One regression (KV thrashing, TTFT 32.8s) was labeled worse and recovered; regressions stay in the record.
Usage:
bash
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/jungledesh/profile/releases/latest/download/profile-installer.sh | sh
profile diagnose --url http://localhost:8000/metrics --duration 2m
vLLM only today; multi-GPU/TP, more engines and k8s are on the roadmap.
More from Infra
- Musk says he wants 'maybe more than 10GW' of compute; Nvidia now counts SpaceX as a CSP and co-designs a space server — elonmusk · 2026-10-16
- Jensen Huang: Musk's AI buildout moved from K's to M's, now a CSP-tier NVIDIA customer — elonmusk · 2026-10-16
- EpochAI pegs GPT-6 training at up to 1.8e27 FLOP; scaling01 calls param estimates way too high — scaling01 · 2026-10-16
- Meta's Muse Spark Open scores 39 vs 48 for closed Spark 1.3 on AA Intelligence Index — EAccelerate_42 · 2026-10-16
- LLM spots useless 95GB-per-frame memcpy in ffmpeg dblur, cutting runtime from 2452ms to 49ms — IgorCarron · 2026-10-16
- DeepSeek-V4-Flash on 4x Radeon AI PRO R9700: the 'refused' 4-card config hits ~79 tok/s — brainchillzZ · 2026-10-16