Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
Thomas Fel, Matthew Kowal, Mozes Jacobs, Dron Hazra, Usha Bhalla, Lee Sharkey, Lucius Bushnaq, Satchel Grant, Tal Haklay, Thomas Icard, Can Rager, Michael Pearce, Daniel Wurgaft, Aiden Swann, Fenil Doshi, Siddharth Boppana, Curt Tigges, Nick Cammarata, Thomas Serre, Vasudev Shyam, Owen Lewis, Thomas McGrath, Jack Merullo, Ekdeep Singh Lubana, Atticus Geiger
cs.CV
2026-06-24
Three block-sparse featurizers recover visual concepts as 2–4D manifolds. They compress DINOv3 better than SAEs; one InceptionV1 block holds the full curve-orientation circle.
Sparse autoencoders (SAEs) split activations into a sparse sum of directions. Each feature is one axis and its strength is a scalar. Visual neuroscience and recent geometry papers describe a different picture. A concept often occupies a low-dimensional region of activation space and carries internal coordinates. A rabbit face is not a single "face present" axis. Nose and eyes sit at different locations on the same region.
A directional prior mismatches that structure. An SAE slices a curved manifold into petals, one atom per heading. The right regularizer is sparse across concepts and dense inside each one. Block sparsity, from the structured sparsity literature, is that prior.
The working generative model is an additive mixture of manifolds: an activation is a sum of a few points, each drawn from a low-dimensional manifold (a Minkowski sum). Each manifold is treated as a b-dimensional subspace, and the code is partitioned into blocks. A MAP argument yields an ℓ₂,0 penalty on blocks. A block turns on or off through its ℓ₂ norm; coordinates inside an active block do not add to the sparsity count. The penalty sees only the norm, so it is blind to rotations of the basis inside a block. The atom is the subspace.
Three architectures share a linear decoder and differ only in the encoder:
Codes are signed. A ReLU would clip each block to a cone; recovering superpositions needs the full subspace. Interpretation uses two readouts: block norm for presence, PCA of the block's contributions for location on the manifold. The main sweep trains on final-layer DINOv3 ViT-B patch activations (d=768), varying block size b, dictionary width G, and block sparsity k. The baseline is the b=1 SAE.
A synthetic superposition of 128 factors, half 1-D atoms and half curved manifolds (circles, spheres, tori), gives per-block R² of 0.93–0.97 for all three BSFs against an oracle of 0.99. A TopK SAE scores about 0.53 and shatters multi-dimensional factors. Concurrent SMixAE and MFA score about 0.71 and 0.38. A signed code plus block projection, or decoding MFA additively rather than as a convex mixture, lifts them to about 0.85 and 0.88.
| Method | Per-block R² |
| Oracle | 0.99 |
| Three BSFs | 0.93–0.97 |
| TopK SAE | ≈0.53 |
| SMixAE / repaired | ≈0.71 / 0.85 |
| MFA / additive decode | ≈0.38 / 0.88 |
Downstream tasks set a fidelity floor. Classification and segmentation barely move when activations are quantized down to reconstruction R²≈0.8; depth needs R²≈0.9 to keep 95% of accuracy. Reconstruction itself rises monotonically with G, k, and b, so R² cannot rank methods. Minimum description length (MDL) can: bits for which blocks fire, the code they carry, the residual, and an amortized dictionary. Block sparsity pays log₂ C(G,k) for the support; an SAE pays log₂ C(Gb, kb) for kb atoms. At the 20% distortion floor, all three BSFs beat b=1, with the shortest codes at b=2 to 4. Mean stable rank saturates between 2 and 4 even when blocks are given b=16. DINOv3 concepts average roughly three dimensions.
InceptionV1 curve detectors (Cammarata et al.) and later SAE atoms each cover a thin wedge of orientation. A single Grassmannian block stays on across all headings. The first three Fourier modes hold 89% of variance (59% / 18% / 12%). The ω=2 mode is invariant to a 180° flip, which matches neurons that wrap at 180° rather than 360°. A matched null leaves k≥2 near the white-noise floor.
On DINOv3, treating each block as a single-concept ImageNet detector, BSF F1 beats the SAE at every sparsity, and the gap grows with b. At k=64, concept maps at b=4 are about five times smoother in total variation than at b=1. Blender renders with a sweeping sun isolate a luminance block and a cast-shadow block that track lighting on a bunny, monkey, cow, and teapot, independent of object identity. On SDXL, steering along pretzel, hat, person, and coffee blocks, with a Kohonen map fitted to the manifold, produces images that vary smoothly across the grid.
Linear probes on the codes beat a probe on raw activations for ImageNet classification, with top-1 in the 0.83–0.84 band, and approach but do not beat it for ADE20k segmentation and NYUv2 depth.
The unit of analysis in interpretability moves from a direction to a block with internal geometry. Some of the fragmentation seen when training SAEs is an artifact of the prior. It is not evidence that the network stores a curve as unrelated detectors.
Practical use: decomposing vision backbones and diffusion activations, and steering along a manifold rather than along one axis. Code is released. This is an incremental step with a clear thesis: choose atoms to match representation geometry. The paper does not sell blocks as a universal atom.
The authors treat the mixture of manifolds as a working model. They expect block sparsity applied directly to language or video to mismatch the prior, the way a directional SAE mismatches visual manifolds. Grassmannian charts are QR-projected only every 20 steps; BatchTopK is left for later. Group Lasso hits a target sparsity by turning on the ℓ₂,1 term when the count overshoots, an engineering schedule. MDL optima sit in shallow minima on a coarse grid. The claim that holds is "structure is shorter, moderate b", not a single best b.
Steering has almost no quantitative comparison to SAE steering, and no fidelity or controllability scores. Lighting blocks are ranked on synthetic renders; no correlation with human labels or physical meters is reported. Linear-probe numbers appear as plots, not a table of point estimates. The toy embeds factors in random orthonormal subspaces; real activations have overlapping subspaces.