Fine-tuned 2B models beat OpenAI o3 and Gemini 2.5 few-shot on molecular classification
madeofAjala · x · 2026-11-22
Attached to a tweet asking Logan K and Greg Brockman for free tokens, the author shared a hands-on molecular classification benchmark with a solid conclusion:
Fine-tuning always wins — even the weakest fine-tuned EF@1% (Gemma-2 2B at 3.660) beats the best closed-source few-shot result (Gemini 2.5 at 3.340).
Key numbers:
- Classical baselines: Random Forest ROC-AUC 0.683, XGBoost 0.665.
- After 1 epoch of QLoRA fine-tuning, Gemma-2 2B hits the top ROC-AUC of 0.706; LlaSMol-Mistral 7B leads on MCC (0.563) and EF@1% (4.982) while staying fast at 0.0816 s/mol.
- Closed-source few-shot: OpenAI o3 only 0.591 ROC-AUC, Gemini 2.5 at 0.528.
- 95% bootstrap CIs confirm the LlaSMol vs Gemma-2-2B EF@1% ranking holds; the ROC-AUC gap remains provisional.
More from Models
- Dev rants against GPU-stacking: DeepSeek's <1GB RAM per 1M context shows efficiency wins — gnukeith · 2026-11-22
- Ant open-sources LingSante-3.1-flash, a 560B healthcare model with FP8/MXFP4 weights — BerryBlush_Rogers691 · 2026-11-22
- Gemini 4 Neon takes #1 on LiveBench 2.0 with 87.4, beating Claude and GPT by 2.5 points — demishassabis · 2026-11-22
- User Reports Noticeable Quality Drop in Claude Opus 5.5 Recently — jiayuan_jy · 2026-11-22
- Speculation: OpenAI quietly throwing massive compute at remaining Millennium Problems — haider1 · 2026-11-22
- Sakana AI announces its own pretrained LLM series Sakana Sawara, first model due spring 2027 — SakanaAILabs · 2026-11-22