DynaTokens: Controlling Token Dynamics for Continual Video-Language Understanding
Toan Nguyen, Yang Liu, Celso De Melo, Flora D. Salim
EMNLP 2026 Main Conference
cs.CV, cs.LG
2026-03-02
DynaTokens generates adapter tokens from compact task codes via a shared transformer. On NExT-QA it hits 64.11 Acc and 4.76 forgetting, beating Bisecle at 62.37 / 5.34.
Continual VideoQA on a frozen multimodal LLM is stuck on a storage-versus-interference bind. Fine-tune on each new task and earlier answers collapse. That is catastrophic forgetting. Keep a separate prompt or adapter per task and both storage and routing grow with the stream. ProgPrompt's peak GPU memory on NExT-QA climbs from 21.16 GB to 25.43 GB. Share one prompt across tasks and later gradients fight earlier ones.
VideoQA is a steeper shift than static VQA. The same clip can ask causal, temporal, and descriptive questions. Most parameter-efficient adaptation (PEA) recipes were tuned on image-text streams. On a frozen LLaMA-2-7B plus CLIP ViT-L/14, there is still no recipe that keeps per-task memory almost flat while holding old-task accuracy.
DynaTokens generates prompts instead of storing them. Each task keeps two small objects: a generative code z^t and a retrieval key k^t. A shared transformer hypernetwork Hφ synthesizes the adapter prompts that actually enter the LLM. Generator weights are reused, so model size does not grow with the task sequence.
Generation is layer-conditioned. The code splits into a shared invariant part zinv and a task-specific part zsp, plus a learnable layer embedding so shallow and deep layers get different prompts. On LLaMA-2-7B with 10 prompt tokens per layer and 32 layers, the materialised prompt tensor is about 1.3×10^6 values. Each task stores about 6.7×10^3 values, roughly 26 KB, a 200× cut versus storing the prompts.
Routing is gradient-free by design. Frozen pretrained embeddings of the video and question are pooled, then passed through a fixed Rademacher random projection to form the key. At test time the query is processed the same way and the nearest code is retrieved by cosine similarity. No task id is required. The router has no trainable parameters, so it cannot drift.
Forgetting is held down with look-ahead regularisation (LA-Reg). The generator takes a few inner-loop steps on the current task, then a penalty measures how much the prompts emitted for old task codes moved relative to a snapshot taken before the update. Only codes are stored; old videos are not replayed. The paper ties this term to sharpness-aware minimisation: it penalises sharp directions along the current-task gradient that would warp old prompts. A quadratic anchor sits on the shared code. Two auxiliary losses sit on top: predict the question from video and answer (weight 1.0), and predict the video from question and answer (weight 0.1). The second is anti-causal and is kept only as a weak visual-grounding signal.
Every method freezes LLaMA-2-7B and CLIP ViT-L/14 and uses 32 LLaMA-Adapter layers. NExT-QA is split into eight tasks by question type. DramaQA is split What → Who → Where → How → Why. Metrics are average accuracy Acc (mean top-1 across tasks after the stream) and average forgetting Fog (mean drop from each task's peak to its final score).
| Method | NExT-QA Acc / Fog | DramaQA Acc / Fog |
| LLaMA-Adapter | 46.58 / 13.83 | 60.99 / 24.39 |
| ProgPrompt | 53.95 / 10.69 | 67.92 / 14.95 |
| ColPro | 55.14 / 7.43 | 71.24 / 12.64 |
| Bisecle | 62.37 / 5.34 | 71.49 / 10.37 |
| DynaTokens | 64.11 / 4.76 | 72.52 / 9.76 |
Against the previous best, Bisecle, that is +1.74 Acc and −0.58 Fog on NExT-QA, and +1.03 Acc / −0.61 Fog on DramaQA. Chaining the two benchmarks into a 13-task domain-incremental stream (real-world video then TV drama) gives DynaTokens 61.96 Acc and 11.72 Fog, versus Bisecle at 55.31 / 13.30. Zero-shot transfer: train on NExT-QA, test on DramaQA, 52.83 versus Bisecle's 49.07; the reverse split is 36.29 versus 27.86.
Ablations make LA-Reg the main driver. With every extra term off, accuracy is 21.28 and forgetting is 16.46. LA-Reg alone reaches 62.12 / 5.46. Auxiliary losses add about 1.67 Acc. Adding look-ahead steps moves Acc 54.54 → 62.30 → 63.92, and the default two-step run lands at 64.11. Joint training on all eight NExT-QA tasks sits at 68.42; DynaTokens reaches 93.7% of that ceiling, Bisecle 91.2%.
A new ImageQA→VideoQA protocol pretrains on Visual7W, then continues on NExT-QA. Bisecle's video accuracy falls from 62.37 in the video-only setting to 58.24 (−4.13). DynaTokens falls from 64.11 to 62.31 (−1.80). Negative transfer is still there, at less than half the size.
For anyone running lifelong VideoQA on a frozen multimodal LLM, this is a deployable trade. Inference latency is almost identical to Bisecle (49.3 vs 47.7 ms/query). Each task adds about 26 KB, so the prompt warehouse does not grow with the stream. Test-time task ids are unnecessary.
It is not a free lunch. Trainable parameters are 9.6M, about 3× Bisecle. Training NExT-QA takes 5.55 hours and 103.8 GB peak memory; Bisecle takes 1.48 hours and 18.9 GB. A one-step look-ahead variant cuts that to 2.46 hours and 82 GB, still at 63.92 Acc versus Bisecle's 62.37. Training is expensive. Inference is not. The gains are a solid incremental step, not a new paradigm.
The paper lists several. Image pretraining does not produce a net gain on video, only milder negative transfer. The standard splits are question-type incremental; streams with broader visual and temporal drift are untested. DramaQA is visually homogeneous, so nearest-key routing hits only 41.7% accuracy. Downstream that costs 0.50 Acc versus oracle routing, but the router itself is shaky. Experiments stop at 13 tasks. LA-Reg grows linearly with stored codes, and longer horizons are open.
Two further caveats sit outside that list. The backbone is LLaMA-2-7B, dated for current video LLMs. During ImageQA→VideoQA the authors relax LA-Reg on the image task; that selective relaxation helps the transfer, and it also shows the regulariser is not drop-in. Across four task orders, Bisecle wins one split on Acc (63.09 vs 62.96). The ranking is not locked under every permutation.