Dev tests four tiny OCR architectures inspired by Meta's in-fusion paper
wightmanr · x · 2026-11-21
Developer wightmanr, retraining his old OCR project, found Meta's in-fusion paper a natural fit and ran a series of experiments:
- Trained a decoder-only variant as an alternative to NaFlexViT encoder + text decoder, adding in-fusion by rebalancing text vs. image param widths. His short training budget couldn't beat the pretrained-weight baseline, but it converged well and notched clear improvements.
- Also fused vision params into a pretrained SmolLM2 135M to match his 225M in-fusion run. It worked but converged slower—likely the >2x depth hurt the short training cycle; differential learning rate ablations are needed.
He catalogs four NaFlex-friendly configs, dessert-named: Cruller (cross-attention encoder+decoder, Donut style), Fritter (image tokens as prefix to decoder-only LM), Cronut (single native decoder-only trunk over raw patches + text, optional in-fusion vision weights), and Berliner (Cronut-style with a pretrained LLM trunk like SmolLM/Qwen3 plus in-fusion vision weights).
Related event: Developer Retrains Small OCR Models Inspired by Meta's In-Fusion Paper(2 posts)→
More from Multimodal
- Ex-TensorFlow early member founds TensorScale, betting world models will beat LLMs — TerryTangYuan · 2026-11-21
- Cinematographer Tests Minimax's Cinestill 800T Emulation for a Blade Runner Look in Neon Rain — Purple-Let4907 · 2026-11-21
- 15-year electronic producer built an agentic DAW with Claude and made a full album — yawnxyz · 2026-11-21
- Self-hosted cloud GPUs beat Higgsfield and Runway APIs on cost and flexibility — Al_Grigor · 2026-11-21
- Gaussian splat top-down scan shows how Fable plans object placement in pixel apartment — XRarchitect · 2026-11-21
- Pixel replica of a real apartment built from Atlas gaussian splats with Fable 5.2 — XRarchitect · 2026-11-21