SGLang boosts multimodal inference >10% by replacing GPU memory bookkeeping with a dict lookup
charles_irl · x · 2026-11-02
A Modal engineering post (amplified by charlesirl to mock "performance slop" on the timeline) shows what real perf engineering looks like:
- "Boosting multimodal inference performance by >10% with a single Python dictionary": profiling SGLang's scheduler on a multimodal workload revealed expensive bookkeeping around shared GPU memory that could be replaced with a simple cache lookup—throughput and latency both improved >10%, merged in SGLang v0.5.10
- The tool was literally py-spy: a flamegraph showed newsharedcuda at 3.43%
- The quoted methodology: perf eng = get CPU flamegraph plus heap/disk/network profiles, make the biggest bar smaller, else move to the next biggest bar
A pragmatic lesson: skip the design-principle posturing—profile first, attack the top of the flamegraph.
More from coding & agent
- 10 open-source agent repos small enough to read in an hour, from PocketFlow to mini-swe-agent — victor_explore · 2026-11-02
- spaCy makers launch free Solo plan for Ellf, an AI agent for NLP engineering — _inesmontani · 2026-11-02
- Ellf launches free Solo plan for solo developers working locally — _inesmontani · 2026-11-02
- DeepSeek Harness launches Creator Mode to turn ideas into plugins — CurieuxExplorer · 2026-11-02
- Altman: model that cracked Navier-Stokes is the next GPT, run as 10,000 agents for 4 days — Hesamation · 2026-11-02
- HF engineer's agent hits 99.7% storage savings on Parquet edits in DuckDB via content-defined chunking — lhoestq · 2026-11-02