Hypernetworks show power-law scaling for factual knowledge injection in LLMs
Nischay Dhankhar · hf · 2026-07-23
This paper studies hypernetworks as a train-time method for injecting factual knowledge into LLMs at scale.
The authors generate a fixed LoRA adapter from a hypernetwork trained on large corpora of facts, then insert that adapter into the target model so it can answer questions about those facts. They introduce MegaWikiQA, a dataset with tens of millions of multi-hop QA examples across 39 domains built from Wikidata5M, and examine how performance changes with hypernetwork depth, width, and target model size.
Main findings:
- hypernetwork-based injection shows broadly predictive power-law scaling across architecture axes;
- it generalizes OOD reliably as scale increases;
- in OOD evaluations, it exhibits steeper scaling exponents than LoRA finetuning and full finetuning.
The paper positions hypernetworks as a principled substrate for scalable train-time adaptation and claims this is the first empirically grounded scaling-law study for such systems.
More from Research
- Exponential launches RepoInject and BrowserInject benchmarks for prompt injection robustness — maksym_andr · 2026-10-27
- Redwood: CoT controllability evals are under-elicited; better prompts boost scores 2-3x — dfrsrchtwts · 2026-10-27
- Inverse-designed optical system solves MNIST using BeamZ — S_Conradi · 2026-10-27
- DeCert supervises generative reward models' rationales without reference explanations — Guiquan Sun · 2026-10-27
- VLM harnesses act as compositional generalizers, new blog post shows — a1zhang · 2026-10-27
- Article proposes a Human Genome Project for AI interpretability — ninamiolane · 2026-10-27