Hill climbing isn't RL: dev untangles benchmarking vs benchmaxxing vs metric optimization
dejavucoder · x · 2026-11-11
Replying to a thread by @llmwizard, dejavucoder clarifies three commonly conflated concepts:
- Hill climbing: often mistaken for RL, it's really about optimizing a domain metric by iterating on inputs — SFT data, RL environments, scaffolds, harnesses, prompts, etc.
- Benchmarking: measuring a model's performance in a domain via task sets.
- Benchmaxxing: training literally on eval tasks or on very similar likely-synthetic tasks, or over-tuning scaffolds — scores look great but capability isn't there.
The takeaway: hill climbing is a training-side optimization method, benchmarking is measurement, and benchmaxxing is what decouples scores from real capability.
More from Research
- When AI becomes an oracle, what happens to mathematics? — stevenstrogatz · 2026-11-11
- 200 Years of Linear Algebra in One Chart: From Leibniz's Determinants to Peano's Vector Spaces — burny_tech · 2026-11-11
- MessyMem on Compact Long-Term Memory for Agentic Robots Hits CoRL 2026 Main Poster — leto__jean · 2026-11-11
- Tree-Ising repairs looping LLMs via bit-level Ising edits at $0.90 per fix — pmttyji · 2026-11-11
- A 4,000-Word Visual Guide to Graph Theory, From City Traffic to Shortest Paths — TivadarDanka · 2026-11-11
- Even after a 6-month recursive leap, AI progress stays 3x the 2024 rate — ben_j_todd · 2026-11-11