Qwen 27B + Python REPL harness hits 78.9% on Harvey benchmark, beating larger flagships
bansalg_ · x · 2026-11-06
Using a Qwen 27B model with a Python REPL harness and no model training, ekzhu's team scored 78.9% criteria pass and 35.2% all pass on Harvey LAB Firm Knowledge — while much larger models like GLM-5, Gemini 3.5 and Opus 4.8 mostly land at 60-75% criteria pass and around or below 30% all pass in other work. Takeaways: (1) harnesses matter enormously but are still poorly understood; (2) Qwen 27B is a strong baseline for validating new models. The quoted thread also frames Recursive Self-Improvement: model, skills, harness and weights are all learnable end-to-end, with skills/memory/prompts as efficient, inspectable inference-time training that gets learned first.
More from coding & agent
- Conductor shuts down after Meta acquisition; cherry π ships a one-command migration path — sull · 2026-11-06
- LangChain says most of its internal agents rely on async subagents, now in deep agents — Hacubu · 2026-11-06
- SaaStr runs Replit Agent as its 'CMO': 3 marketing ideas a day — amasad · 2026-11-06
- Visa's Trusted Agent Protocol uses passkeys to verify AI shopping agents at checkout — NVIDIAAI · 2026-11-06
- Cohere's North platform secures AI agents across cloud, private and air-gapped deployments — NVIDIAAI · 2026-11-06
- Choosing browsers by how easily Codex can control them — TheZachMueller · 2026-11-06