Thinky Model Scaling and Multimodal Insights

burny_tech · x · 2026-07-16

The shared content summarizes several technical assessments of the Thinky model:

muP becomes meaningful again: The author believes muP helps ensure returns on the scaling ladder for 1T+ parameter models.
Low sparsity: Speculated to be caused by hardware constraints or lower MFU.
45T tokens: One of the highest training token counts for an open model to date. The author feels this aligns with scaling law expectations, but the model quality still appears somewhat weak, indicating insufficient gains from effective parameters.
Unconventional architectural choices: Including convolutions in attention and the coupling of weight decay with learning rates, which the author thinks is likely tied to specific training setups.
Native multimodal training: The author speculates the team may have learned methods from OpenAI, will prioritize this as a differentiator for open models, and guesses that multi-task training might aid coding and reasoning capabilities.

More from Models