Prof. Tom Yeh Explains Two-Tower MLP by Hand, From Recommenders to CLIP
ProfTomYeh · x · 2026-09-14
In the 11th installment of his "AI by Hand" series, Prof. Tom Yeh breaks down the Two-Tower MLP with an interactive hand-drawn diagram: two inputs pass through independent MLPs whose outputs are concatenated and linearly projected. The pattern powers recommendation systems (user/item towers) and contrastive learning, and its most far-reaching use is multimodal—one tower reads images, the other text, and training pulls matching pairs into one shared space. That's exactly how CLIP works, reducing image-text matching to simple vector arithmetic. Interactive diagram available on byhand.ai.
More from Research
- Ted Nelson's 1965 Thoughts on Hypertext Resurface — dreamwieber · 2026-09-14
- Meme nails AI research's oldest gripe: papers without reproducible code — CSProfKGD · 2026-09-14
- Csaba Szepesvari: Math's Love of Infinity May Be a Fiction — CsabaSzepesvari · 2026-09-14
- TCS researcher reflects on problem-solving vs theory-building in the AI era — jasondeanlee · 2026-09-14
- Study: dropping 0.003% of LMArena votes can flip the top model — rishabh16_ · 2026-09-14
- Just 21 annotators (6.5%) provide half the votes in Anthropic-HH-RLHF — rishabh16_ · 2026-09-14