GAP: experimental gated block merges attention and MLP, cutting ~20% params
WAMFT · reddit · 2026-11-09
Reddit user rickey1990 open-sourced GAP (Gated Attention Projection), an experimental Transformer architecture that merges attention and MLP-style processing into a single gated block instead of separate sublayers.
- Trained each model on 100M WikiText-103 tokens against a standard Transformer baseline.
- GAP20 uses 19.91% fewer parameters; an aggressive variant saved 34.31% but performed worse, so the author reinvested 14% of savings into width.
- The standard Transformer still won on final LM accuracy; torch.compile boosted GAP20 CPU throughput by 19.9%.
- Code, configurable sizes, speed benchmarks, and a paper are on GitHub (rickey1990/Gated-Attention-Projection).
The author frames it as an experiment, not a claim of beating Transformers, and invites feedback on the math and design.
More from Research
- NVIDIA's SIGGRAPH 2026 paper DAT delivers penetration-free contact for cloth, soft bodies and robots — zhengyiluo · 2026-11-09
- USC paper shows AI agents leak secrets via the "mosaic effect" without reading classified files — 2C_ornot2C · 2026-11-09
- Tsinghua & Moonshot's TCSExam: GPT-6 Astra proves 70% of theorems, solves 0 open conjectures — rohanpaul_ai · 2026-11-09
- TCSExam benchmark: GPT-6 Astra tops Full at 71.9, Opus 5.5 leads Lite at 72.3 — rohanpaul_ai · 2026-11-09
- Meta teases entry into AI-for-science race, joining frontier labs push — kimmonismus · 2026-11-09
- Study: 3g Daily Creatine for 4 Weeks Improved Cognition in Women, Most in Older Adults — 2C_ornot2C · 2026-11-09