Same Weights, 5x Cost Gap: HF Engineer's Agent Harness Workshops at AI Dev NYC
DeepLearningAI · x · 2026-11-17
Hugging Face ML engineer Joel Niklaus will teach two hands-on agent harness workshops at AI Dev NYC (Nov 30–Dec 1), hosted by Andrew Ng's DeepLearning.AI.
Part 1: Your benchmark score is a harness score
- A harness is the code around the model: prompts, tools, context management, stopping logic
- Identical Muse-Glimmer-30B weights cost $1.73 per solved Terminal-Bench 2.1 task in OpenCode vs $8.80 in OpenClaw — a 5x gap
- Swapping harnesses moved GLM-5.2's pass@1 from 23% to 52% on 250 held-out SWE-bench Pro tasks; harness rankings don't transfer across models
- Attendees run two harnesses on a 10-task Terminal-Bench slice and build a shared pass-rate/cost leaderboard
Part 2: Don't train the model, evolve the harness
- A meta-harness agent reads failed runs and patches the harness around a fixed model
- The loop took GPT-OSS-20B from 4.8% to 14.8% on held-out Terminal-Bench 2.1 for roughly $50 of harness search
Niklaus previously built legal LLM systems at Harvey, pretrained LLMs at X and Google's moonshot lab, and won Outstanding Paper Awards at ACL and COLM.
More from coding & agent
- AI reviewer handled 110 PRs with 22-min median first comment at healthcare data firm — alex_verem · 2026-11-17
- Opus 5.5 ran an AI company for 7 weeks, earning 50k downloads and 100+ GitHub users — maksym_andr · 2026-11-17
- Opus 5.5 Ran an AI Company for Weeks and Kept Climbing Hugging Face Downloads — maksym_andr · 2026-11-17
- Developer uses his GitHub Radar project as an agent entry point to decide what to work on next — dSebastien · 2026-11-17
- Lectern CLI turns agent-written Markdown into narrated explainer videos in under 30s — gabe_ragland · 2026-11-17
- 6,167 x402 agent-payable endpoints audited: 90% accept payment, 51% CORS-blind — JonahDiedrick · 2026-11-17