ModernBERT Contributor: Train Multiple Heads With ~1K Examples in an Afternoon
alexisgallagher · x · 2026-09-17
In a discussion about how a product's value lies in pure in-context learning and cognitive ease, ModernBERT pretraining contributor alexisgallagher points out that many people don't realize you can already train multiple heads on a ModernBERT model to build a similar system — with about a thousand examples, doable in an afternoon.
Related event: ModernBERT Author: ~1,000 Samples Replicate Similar System in an Afternoon(3 posts)→
More from Research
- Softmax probabilities don't match real-world frequencies, breaking cross-company calibration — HanchungLee · 2026-09-17
- PiSSA-style SVD init for LoRA adapters also improves downstream RL training, Trajectory Labs reports — simonguozirui · 2026-09-17
- New report examines how AI is reshaping science and innovation today — soumitrashukla9 · 2026-09-17
- AI Slop Papers Make Reviewing Easier: Only 25-50% Deserve Careful Reads — tallinzen · 2026-09-17
- TMLR's unusual Zoom tests: 7 of 7 invited authors fail to explain their own papers — deliprao · 2026-09-17
- Human organoids grow to occupy most of mouse cortex in Nature xenocortical study — Dr_Alex_Crimi · 2026-09-17