DAGFormer: Predicted Cross-Layer Topologies Cut Perplexity Across Model Scales
AI-Interpretability-illinois · hf · 2026-10-19
An Illinois paper introduces DAGFormer, improving Transformer efficiency: instead of every layer reading the previous one, a jointly trained 2-layer causal Transformer topology predictor plans Q/K/V/residual cross-layer connections per token, forming a global DAG over read sources. At 75M, 300M and 1B scales, the planned-topology variants with refinement reduce language modeling perplexity and boost downstream performance, with an interactive visualization of the trained graphs.
More from Research
- 1.9 MB extracted (not trained) LoRA fixes vertical color stripes in MOSS-Image SFT — Nynxz · 2026-10-19
- Three ways to execute a AND b AND c: DataFusion filter PR explained — Abhishekcur · 2026-10-19
- ACT3D trains VLMs to actively explore 3D worlds with RL, hitting 53.2% accuracy — UCSanDiego · 2026-10-19
- OPMD: ShanghaiTech distills teacher intermediate computations beyond output supervision — ShanghaiTech · 2026-10-19
- LexFrame: interpreting LM residual streams in the model's own semantics — Jian Gu · 2026-10-19
- Amazon Paper: QA Scores Miss Agent Memory Failures by 16.8 Points, New FEB-30 Benchmark Fixes It — rohanpaul_ai · 2026-10-19