Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models
Nischay Dhankhar, Dos Baha, Abulhair Saparov
cs.CL, cs.LG
2026-07-22
A hypernetwork reads a corpus of facts and generates a fixed LoRA adapter for a frozen LLM. The paper gives the first scaling laws for this setup: injection quality follows power laws along width, depth and target size, and OOD generalization scales steeper than LoRA or full fine-tuning, with the gap widening as the target model grows.
Injecting factual knowledge into large models reliably and at scale is an unsolved problem. Full fine-tuning is expensive and prone to catastrophic forgetting; LoRA and other PEFT methods cut cost but do not eliminate forgetting and generalize poorly to unseen entity and relation combinations. More fundamentally, SFT learns new facts slowly: facts consistent with pretraining are picked up fast, genuinely new facts are not. This work targets train-time injection, not test-time adaptation or knowledge editing: both the hypernetwork and all baselines are trained to compress a fixed fact corpus into target-model weights, with no access to that corpus at evaluation.
A hypernetwork, a second network, reads the set of facts to inject and generates a set of LoRA-style weight changes for a frozen target model. Only the hypernetwork is trained; target-model weights stay frozen through training and inference. The hypernetwork is Transformer-based, randomly initialized (not from pretrained weights), generating LoRA with rank 4 and scaling factor 8, applied to roughly the last half of the target model's layers.
The pivotal design move decouples the hypernetwork's injection capacity from the target model's general capability: injection capacity belongs to the hypernetwork, general capability to the frozen target. This enables the first rigorous, controlled study of scaling laws for the hypernetwork architecture itself, unconfounded by changes in target-model capability. The accompanying MegaWikiQA dataset is built from Wikidata5M (about 4.6M entities, 822 relations, 22M+ triplets): roughly 10M samples per hop count up to 4 hops, stratified into 1.25M training samples across 39 semantic domains, with philosophy, linguistics and civil engineering held out as explicit OOD splits. Questions are generated by deterministic grammar, not neural generation.
Two main findings.
First, injection quality follows power laws along all architectural axes. Target-model size is the steepest: ID validation loss exponent -0.226, OOD non-rephrased -0.184. Hypernetwork width and depth also yield predictable power laws.
| Axis | OOD rephrased exponent |
| Hypernetwork | -0.107 |
| LoRA fine-tuning | -0.083 |
| Full fine-tuning | -0.069 |
| Axis | OOD MCQ exponent |
| Hypernetwork | -0.171 |
| LoRA fine-tuning | -0.119 |
| Full fine-tuning | -0.101 |
Second, the hypernetwork's OOD scaling is steeper than LoRA and full fine-tuning, and the gap widens monotonically with target-model size. The advantage is not a fixed offset; it grows as the model grows.
For teams weighing knowledge injection against RAG, this elevates the hypernetwork from "a test-time adaptation gadget" to "a scalable substrate for train-time injection," and gives the first scaling laws to guide design. The steepening OOD scaling with scale is especially valuable: it means that injecting new facts into ever larger models gives the hypernetwork a growing edge over fine-tuning, exactly the regime real deployments care about.
Parameter overhead is large: in the highest-capacity experiment the hypernetwork reaches about 2.5B parameters against a 1.5B target model, nearly as large as the target. Scale covers only up to Qwen2.5-14B; 70B and beyond remains open. Evaluation covers single-hop and shallow multi-hop only; deep multi-hop compositional reasoning is unverified. Across all axes, OOD rephrased (robustness to linguistic variation) scales flattest, so this problem does not yield to scaling any single dimension. The authors are from Nace AI, and the work was promoted as a scaling-law result for hypernetwork knowledge injection; the conclusions skew positive, so read with that framing in mind.