ModernBERT Contributor Confirms: Multi-Head Training With 1K Examples Works Now
alexisgallagher · x · 2026-09-17
A ModernBERT pretraining contributor (disclosing potential bias) reiterates that you can train multiple heads on a ModernBERT model right now with about a thousand examples, building a similar system in an afternoon.
Related event: ModernBERT Author: ~1,000 Samples Replicate Similar System in an Afternoon(3 posts)→
More from Research
- Anthropic interpretability researcher tells Tim O'Reilly: an LLM's world model is readable — mlpowered · 2026-09-17
- Softmax probabilities don't match real-world frequencies, breaking cross-company calibration — HanchungLee · 2026-09-17
- PiSSA-style SVD init for LoRA adapters also improves downstream RL training, Trajectory Labs reports — simonguozirui · 2026-09-17
- New report examines how AI is reshaping science and innovation today — soumitrashukla9 · 2026-09-17
- AI Slop Papers Make Reviewing Easier: Only 25-50% Deserve Careful Reads — tallinzen · 2026-09-17
- TMLR's unusual Zoom tests: 7 of 7 invited authors fail to explain their own papers — deliprao · 2026-09-17