Hardcore eval: Fixing gpt-oss harness and testing 320k cases
skeole · reddit · 2026-09-01
The author built a custom inference harness to fix tool calling issues in gpt-oss-20b and ran 320,192 evaluations over 1,062 GPU hours on a single RTX 3090, processing 3.49B tokens. Key findings include:
- Reasoning quality varies with effort: Same token counts yield different reasoning quality; 'Medium' effort achieved 97.1% accuracy vs 38.3% for 'Low'.
- Preserving reasoning isn't a silver bullet: It stabilizes variance but only slightly boosts accuracy and can hurt performance on tasks needing specific formatting.
Code, data, and the fixed template are open-sourced.
More from Research
- Opus-assisted open design: a $31k low-field head MRI sketched in hours — DreamFly_13 · 2026-10-18
- semiconductor.diy: open-source guides to make real chips in a garden shed — jwt0625 · 2026-10-18
- Work in progress: geometric structure of GPG projections as Markov chain coarsening — michaelchchoi · 2026-10-18
- Backprop-free pretraining gets 2.3x speedup as Noise method crushes Dust record on 8 GPUs — inductionheads · 2026-10-18
- q3dviewer: open-source Qt+OpenGL tool for painter-like point cloud editing — 4310sy · 2026-10-18
- Smarter Every Day's Destin to defend PhD dissertation on impact fusion — CatAstro_Piyush · 2026-10-18