Spectral-Spatial Interpretation
Haotian Ma, Ruqi Yang, Philip Townsend
NeurIPS 2026 (Poster)
cs.LG, cs.CV
2026-10-15
SSI fixes each cell's observed context and runs Shapley on its channels. Joint deletion and insertion lead on three datasets; on HLS the cost is about 6% of MC-Owen.
On multispectral inputs, the useful evidence is often one channel at one location. Near-infrared can matter only along a vegetation edge, and short-wave infrared only inside a burn scar. Usual post-hoc tools answer half of that. Grad-CAM, RISE, and Extremal Perturbation sum over channels and return a single heatmap. Mutual information, attention, or a global SHAP score average over space and return one number per band.
A common repair multiplies the two pieces, A(g, c) = s(g)q(c): a spatial map times one channel profile, shared by every location. That profile does not fit a Painted Bunting, where the red belly, green back, and blue head each line up with a different RGB channel, or a Sentinel-2 scene, where water, forest, and built-up land line up with B3, B8A, and B12. The same band can decide the logit in one region and add nothing in another. The unit worth explaining is the pair (g, c).
SSI takes each position-channel pair (g, c) as a player. The model is already trained. An input has C channels, and space is cut into cells G. Players outside a coalition are replaced by a baseline, a chosen reference value. The game value is the target-class logit minus the all-baseline logit. Adding a player may raise or lower the logit. Monotonicity is not assumed.
For cell g, every player outside that cell stays at the value actually observed. Only the C channels at g change. Call this conditional game wg. Its value on the full local coalition equals Δ(g): the logit gap from replacing the whole cell by the baseline while the rest of the image stays put. Φ(g, c) is the Shapley value of channel c in wg, the average marginal contribution of that channel across orderings. Summing channels produces a spatial map. Summing cells produces a band ranking. Neither summary is forced through a rank-1 product.
Each cell uses K = min(2^C, 10) channel permutations, and exact subset enumeration when 2^C is at most 10. CUB-200 has C = 3, so the local game is exact. EuroSAT (C = 13) and HLS (C = 6) use 10 permutations, with empirical concentration bounds below 10^{-2}. In that sampling regime the cost per image grows linearly in C.
Theorem 1: inside a cell, these scores are the only allocation that exhausts Δ(g), gives the same score to symmetric channels, gives 0 to a channel that never changes the value, and adds across games. Theorem 2: the sum over channels equals Δ(g) exactly. The identity does not need monotonicity, supermodularity, or a particular architecture.
The Owen value uses the same position groups and also splits value across groups and within them. It then averages external spatial coalitions with Shapley weights, so the whole prediction is partitioned. SSI scores only the context that was observed. Proposition 1 says no rule can promise both properties on every game: match Δ(g) at every cell, and make all the scores sum to v(P). When cells interact, those local gaps overlap. SSI keeps the local contrast and does not keep a global ledger.
The main baseline is a per-channel mean. A zero baseline is reported as a check. Grids are 8 by 8 on EuroSAT and HLS, and 16 by 16 on CUB-200.
CUB-200 is 200 bird species in RGB with boxes. An ImageNet ResNet-18 is fine-tuned. EuroSAT is 13-band Sentinel-2 with 10 land-cover classes. A ResNet-18 is trained from scratch on every band. HLS Burn Scar is 6-band Harmonized Landsat-Sentinel-2. A lightweight UNet segments scars, and the scalar being explained is the burn-class logit averaged over the ground-truth mask.
Integrated Gradients and DeepSHAP emit dense maps that are summed inside each player (g, c). KernelSHAP treats every (g, c) as a flat feature. The rank-1 control multiplies a RISE spatial score by the logit drop from masking one channel across the whole image. On HLS the comparison also includes Expected Gradients, Guided IG, FullGrad, and MC-Owen.
Joint deletion masks players from high Φ to low. Lower AUC is better. Insertion sums over channels, then fills cells. Higher AUC is better. Rank-1 deletion keeps only the leading rank-1 factor of the attribution matrix and deletes with that. Under the per-channel mean baseline, SSI is best on joint deletion, rank-1 deletion, and insertion for all three datasets.
| Dataset | SSI joint del. | Next-best del. | SSI insertion | Next-best ins. |
| CUB-200 | 0.0088 | IG / DeepSHAP 0.0112 | 0.7132 | RISE 0.6886 |
| EuroSAT | 0.1397 | DeepSHAP 0.2078 | 0.7319 | RISE 0.7091 |
| HLS | 0.1545 | KernelSHAP 0.2096 | 0.6638 | DeepSHAP 0.6321 |
The zero baseline leaves that ranking in place. On CUB, insertion is 0.5224 for SSI, 0.4246 for RISE, and about 0.08 for IG and DeepSHAP. The gap between joint deletion and rank-1 deletion is smallest at C = 3 and largest on EuroSAT, where C = 13.
On CUB with the mean baseline, RISE insertion is 0.6886 against SSI at 0.7132, but RISE uses 8,000 spatial masks and SSI uses 2,048 forwards. Joint deletion is 0.0183 for RISE and 0.0088 for SSI.
Summing channels and comparing with the annotation gives spatial mIoU. At the top 10% of the HLS map, SSI reaches 0.389, DeepSHAP 0.277, and RISE 0.191. KernelSHAP, at the same player grain, localizes worse on both annotated datasets. The SSI spatial sum is exactly Δ(g). Flat Shapley has no such identity.
Band ranks are checked against physical priors: burn indices in the Normalized Burn Ratio family, and NDVI or NDRE for vegetation. Classes without a stable positive signature are dropped. On EuroSAT and HLS, SSI has the highest Hit@K at every reported K and the lowest channel-deletion AUC. For vegetation classes, red-edge and near-infrared rank high and red ranks low, matching the known reflectance pattern. The other methods miss the red-edge bands or assign nearly uniform channel scores.
HLS has |P| = 384 players. MC-Owen with M = 200 block-respecting permutations uses up to 76,800 forwards. SSI uses up to 4,480, about 6%.
| Metric | SSI | MC-Owen |
| Joint deletion | 0.155 | 0.173 |
| Rank-1 deletion | 0.166 | 0.254 |
| Channel deletion | 0.218 | 0.261 |
| Insertion | 0.664 | 0.627 |
| mIoU at 15% | 0.387 | 0.254 |
| Hit@5 | 0.887 | 0.730 |
On those same metrics, Expected Gradients and Guided IG both have joint deletion 0.185. FullGrad is at 0.278, and its Hit@1 is 0. Under ROAD, SSI has the highest LeRF AUC of the nine methods, 0.657, and the lowest MoRF AUC, 0.1143. SSI and MC-Owen both beat the flat and rank-1 baselines, so the position grouping itself is doing work.
Land cover, burn mapping, and fine-grained color all have channel meanings that change with location. A heatmap cannot say whether the model used red-edge or short-wave infrared. A global band rank averages away a band that matters only in a small region. SSI returns an architecture-agnostic (g, c) matrix for a trained model, and the spatial sum equals the logit change from replacing that cell.
The computation is a C-player game at each cell. On HLS, about 6% of the MC-Owen forward budget is enough to lead the reported metrics. The backbones are still a ResNet-18 and a lightweight UNet. Those margins do not transfer to large vision models. Code is public.
The paper says that with hundreds of correlated bands, local Shapley still spends budget on redundant channels. Screening or adaptive grouping is the suggested fix for operational sensors such as EnMAP, and that setting is not tested. Biomedical microscopy and industrial inspection appear only as possible extensions.
Other gaps sit in the experiments themselves. Summing Δ(g) over cells need not recover the full-image logit, so interactions across cells can be counted twice. SSI is not a global ledger. Only a per-channel mean and a zero mask are compared. An 8 by 8 or 16 by 16 grid swallows fine boundaries, and the appendix claim of stable results across resolutions is not backed by numbers in the main text. Hit@K is scored against spectral indices chosen in advance. The qualitative figures are labeled as illustrations. The trained models stop at ResNet-18 and a small UNet, and MC-Owen is compared only on HLS, the smallest player set.