VisionHOPE: Visual Backbones as Self-Modifying Learning Systems
Siran Peng, Tianshuo Zhang, Tianyu Fu, Weisong Zhao, Haoyuan Zhang, Jiankuo Zhao, Minghui Wu, Ping Jiang, Xiangyu Zhu, Chenxu Zhao, Zhen Lei
cs.CV
2026-09-27
VisionHOPE builds a visual backbone from Nested Learning's five coupled memories; ImageNet-T hits 84.1 Top-1, COCO-T 47.9 box AP, ADE20K-T 49.4 mIoU.
Visual backbones have widened how much they adapt inside a single image: shared local kernels, then content-based attention, then input-dependent SSM transitions, then TTT layers that update an inner learner while scanning. The rule that produces that adaptation still sits in the trained outer weights. In Mamba-style models a token picks a transition, but the map from token to transition does not change within the image. ViT3 and Vision-TTT update the inner learner; the maps that build keys and values, and the update rule itself, stay largely outer-parameterized.
Nested Learning (NL) treats a network as coupled online learners whose content, key, value, step-size, and retention memories co-evolve. Dropping that unconstrained self-referential update into a visual backbone is unstable: expansive memory transitions feed back along the scan and training diverges. VisionHOPE asks whether a backbone can change both what it stores and how it learns while reading one image, without the forward pass diverging.
VisionHOPE wraps NL as a stabilized Self-Referential Nested Learning (SRNL) module. Five memories move on every token.
The write rule is retained Delta Gradient Descent: a rank-one gradient write plus a residual correction against whatever the memory already stores at the current key. That correction is there because neighboring visual tokens are highly correlated. Each memory maps the shared value through its own state to make a private target, then forms an L2 learning signal at the current key. What is stored, where it is written, how fast, and how much is forgotten therefore move together.
Without extra control, the spectral norm of one transition can exceed 1. VisionHOPE first applies stabilized L2 normalization to keys, then a soft cap that keeps self-referential injection inside the contraction margin 1-α, then a spectral clamp that keeps the retained transition at most α. The proof is that memory Frobenius norms are non-expansive along each scan. On P-VisionHOPE-S, removing the soft cap yields NaN early in training. Removing the spectral clamp leaves Top-1 at 82.3.
For 2D maps, four independent scans run with chunks aligned to full rows or columns (chunk length equals width on row-major routes, height on column-major). Boundary states generate token-wise quantities in parallel inside a chunk; working memories still update in token order. Outputs are restored and fused with learned channel scales. Cost is linear in token count. The block is a standard two-branch residual: depthwise conv for local context, SRNL for long range, then an FFN. Hierarchical models use four downsampled stages; plain models stay at one resolution.
ImageNet-1K, 224×224, trained from scratch. Hierarchical Tiny: VisionHOPE-T has 27M parameters and 4.9G FLOPs, 84.1 Top-1, tied with RMT-S, above H-ViT3-T at 84.0 and VMamba-T at 82.6. Small is 85.2 (H-ViT3-S 84.9). Base is 85.6 (H-ViT3-B and RMT-L both 85.5). The plain gap is larger: P-VisionHOPE-T (6M, 1.2G) reaches 78.4 versus Vision-TTT-T 77.7 and DeiT-T 72.2; Small/Base are 82.3/83.4.
Hierarchical runs add MESA. Without it, T/S/B still score 83.7/84.7/85.2 against H-ViT3 at 83.5/84.4/84.9 under the same setting.
| Setting | VisionHOPE | Neighbor |
| Tiny ImageNet Top-1 | 84.1 | RMT-S 84.1 |
| Small ImageNet Top-1 | 85.2 | RMT-B 85.0 |
| COCO-T box AP (1×) | 47.9 | H-ViT3-T 47.3 |
| ADE20K-T mIoU | 49.4 | H-ViT3-T 48.0 |
On COCO with Mask R-CNN 1×, T/S/B box AP is 47.9/49.5/50.5 and mask AP is 43.1/44.2/45.0. Base ties MILA-B. On ADE20K with UPerNet, mIoU is 49.4/50.3/51.8: Tiny is 1.4 above H-ViT3-T, Base only 0.1 above.
P-VisionHOPE-S ablations: full model 82.3; freeze adaptive K/V and dynamics, 81.7; one-way scan, 81.6; hard injection clamp, 81.8, with the clamp active on 49.68% of token updates at inference. Against DeiT, FLOPs and memory grow linearly in tokens at high resolution. Figure 3, Base models, one A100, BF16, batch 8, marks about -94.6% FLOPs and -52.8% memory versus DeiT-B. Versus Vim the FLOP curve is similar, throughput is higher at every tested resolution, and memory is lower at large inputs.
Self-referential memory can run as a generic visual backbone on classification and dense prediction, and the code is public. High-resolution inference has a real linear-complexity edge over ViT. Against Vim and VMamba, the extra claim is that the learning rule itself changes with the image.
The accuracy ledger is modest. Hierarchical ImageNet and COCO Base are 0.1-point ties or nudges, with slightly higher FLOPs: VisionHOPE-S 9.8G versus MILA-S 7.3G and H-ViT3-S 8.8G.
This is closer to an engineering proof that NL can be a vision backbone than a new accuracy ceiling. The piece worth copying is the instability diagnosis and the soft injection cap, not a drop-in backbone swap for free points.
The original HOPE architecture also has a Continuum Memory System that updates at several timescales. Discriminative vision has no canonical token-wise target along a scan, so CMS is dropped and only the self-referential part remains. The query projection stays outside the loop. Memories are linear maps, not the MLPs in the NL paper.
The spectral clamp does no empirical work in the ablation. The non-expansion proof needs both controls; training is held up by the soft cap.
Many table baselines are copied from the original papers, so recipes are not fully aligned. Hierarchical numbers include MESA; plain models do not. There is no ImageNet-21K, no video, and nothing larger than Base. Four-way scanning is inherited from VMamba-style designs; one-way loses 0.7 points, so the 2D tax is still there.
Calling it the first generic self-modifying visual backbone is a framing line. TTT backbones already adapt a learner inside the image. The distinction is whether the rule itself moves.