Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference
Burc Gokden
cs.LG, cs.CL
2026-08-11
PLDR-LLM replaces dot-product attention with a learned input-generated operator that goes near-invariant after training (fluctuations ~1e-6), so the deep generator can be cached.
Standard scaled dot-product attention (SDPA) scores queries and keys with a fixed bilinear form, QK^T/√d. Since 2019's CoulGAT, Burc Gokden has been replacing that fixed form with a learned, input-generated bilinear operator, culminating in PLDR-LLM (2024) and its attention, Power Law Graph Attention (PLGA). This paper is the sixth in the line, and it introduces no new model and chases no benchmark. It writes every component of PLDR-LLM as a verifiable definition, then pins the program's three core findings (PLGA contains SDPA, the operator nearly freezes after training, training sits in a critical-like regime) to their actual epistemic strength, labeling each claim a theorem, a conditional theorem, a measurement, or a conjecture, never blurring the categories.
The question is real: input-conditioned attention operators have been studied for years along the hypernetwork line, but whether the learned operator can be characterized and equivalently replaced after training had no clean analytical answer. This paper fills that gap.
PLGA turns the bilinear form inside the score from a constant into a learned operator:
The intermediate tensors A, ALM, AP, GLM, ELM are exposed explicitly as "deductive" outputs, separate from the "inductive" output V that predicts the next token. A NOTEARS-form DAG regularizer pushes GLM toward acyclicity in head space, though strict positivity obstructs exact acyclicity.
Three structural results hold: PLGA contains SDPA exactly at GLM = I; under RoPE nonresonance, a commutant criterion identifies exactly which inserted operators preserve relative-position dependence; and an inference-collapse theorem says that if the deductive outputs are exactly input-invariant, inference reduces to generalized SDPA with a constant operator.
The results here are structural measurements on a released checkpoint, not benchmark wins:
| Metric | Value |
| Operator relative fluctuation | 1e-6 to 1e-11 |
| Generator A numerical rank | 1 (rank collapse) |
| ALM median numerical rank | 62.5 / 64 |
| TruthfulQA mc2 (block = sequential) | 0.3977 |
| WikiText-2 blockwise CE = sequential NLL | 3.4794 nats/token |
After training, the generator A has numerical rank one under an explicit singular-value tolerance, with identical rows across heads in a layer; ALM has a float-zero determinant but near-full numerical rank (median 62.5/64), so the zero determinant is underflow, not real rank loss. The operator GLM fluctuates relative to the input by 1e-6 to 1e-11, and zero at floating-point resolution for the best models, so the whole deep generator can be replaced by one cached constant operator at inference with no benchmark change.
Across eight zero-shot tasks (ARC-Easy/Challenge, HellaSwag, PIQA, Social-IQA, WinoGrande, TruthfulQA, OpenBookQA), 100 items each, block scoring and sequential scoring show zero argmax flips and zero discordant pairs; TruthfulQA mc2 agrees between the two protocols to 8.5e-10 in the 100-item mean; on WikiText-2 held-out windows, blockwise CE and sequential NLL differ by 1.8e-9.
Be honest about this: a TruthfulQA mc2 around 0.40 is low (mainstream models sit around 0.5 or higher), so the audited checkpoint is small and experimental. The paper claims no benchmark superiority over SDPA.
For anyone designing attention mechanisms, this is a clean case study: a hypernetwork-style, input-conditioned operator that empirically collapses after training into a near-constant operator, so the architecture is expensive to train but cacheable at inference. The collapse itself is the interesting phenomenon, because it surfaces the tension between a learned dynamic operator and the fixed form it settles into: you spend training compute learning a dynamic structure that ends up close to constant.
The epistemic discipline is unusual for this flavor of paper: every claim carries a strength label, selected proofs are machine-checked in Lean 4, and an appendix documents floating-point sensitivity. For researchers tracking cacheable or interpretable attention, the formalization and audit scripts are a useful reference base.
It is not a drop-in for practitioners: single author, small checkpoints, no frontier-scale validation.
The author states the limits plainly, which is more than most architecture papers manage. Containment of SDPA is not superiority; "strict function-class separation at matched resources is explicitly not proved." Operator invariance is a measured hypothesis, not a theorem. The perturbation bounds "do not certify cached inference; the assembled proxy misses the decoding margin." Self-organized criticality is a phenomenological framework, not an established result. Several theorems are conditional on Lipschitz or stationarity assumptions never verified for trained models. The audit covers one checkpoint, eight prompts, and one software stack, so extrapolation across seeds, scales, or datasets does not hold. Downstream of the power-law stage, local amplification reaches about 1e10, and values drift with the hardware and library stack.
The sharpest concern: the practical payoff (cache the operator for inference speedup) depends on invariance holding at larger scale, which is exactly what was never tested. With TruthfulQA this low, you also cannot tell whether collapse helps or hurts quality at scale. The whole program is single-author, and the theorems that matter most still rest on hypotheses that have not been verified for real trained models.