Cloudy: autonomous cloud research agent hand-writes Triton kernels that beat tuned vLLM by up to 15%

saurabhtwq · x · 2026-10-24

What it is: Cloudy is a tool for long-running automated ML research on cloud GPUs — experiments move between CPUs and GPUs with TB-scale storage, pause/resume anytime, while an agent designs experiments, learns from results, and iterates for hours before handing over code and numbers.
Demo run: The agent profiled gpt-oss-20b/120b decode on 1× H100, found MoE and attention kernels dominate, then rewrote kernels.py in 297 lines using only torch + triton (RMSNorm, RoPE, split-KV decode attention, grouped MXFP4 MoE with load-time weight rearrangement).
Results vs tuned vLLM 0.31.0 (baseline 330.9 tok/s at 20b batch 1): +15.2% (20b, batch 1), +10.8% (20b, batch 64), +14.9% (120b, batch 1), +12.0% (120b, batch 64).
Correctness: greedy outputs match an fp32 reference over 20 prompts × 1000 steps; GPQA-D and AIME25 scores fall within published 95% intervals.
Known gap: 120b serving under saturation runs at 0.946× vLLM because prefill pauses decoding.

More from coding & agent