Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention
Haiqing Hao, Zhipeng Sui, Rong Zou, Zijia Dai, Nikola Zubić, Davide Scaramuzza, Wenhui Wang
ECCV 2026
cs.CV
2026-03-06
Tsinghua and UZH's SSLA updates only overlapping spatial substates per event. SSLA-Det hits 0.375 mAP on Gen1 at over 20x less per-event FLOPS than DAGr-L.
Event cameras emit timestamped pulses only when a pixel's brightness changes. The stream is sparse and fast. To spend that latency budget on detection, a network has to update its prediction when each event arrives, instead of binning events into a frame and running a dense image model.
Asynchronous detectors stall on two coupled constraints. Recurrent nets match event-by-event inference, but long event sequences are painful to train in parallel on GPUs. Accurate boxes need a fine spatial state; growing that state under dense linear attention makes per-event compute scale with state size, which eats the latency win. Graphs, submanifold convolutions, and point-cloud nets all try to spend spatial sparsity on compute, yet deep receptive fields still densify activations. Linear attention already gives parallel training plus recurrent inference, and it has worked for global classification. Detection never got an end-to-end asynchronous version, because localization wants a large state and dense updates make that state expensive. EVA had already tried linear attention for event detection, and still needed a dense backbone.
SSLA splits one global hidden state into many spatially overlapping local substates, a mixture-of-spaces. A P×P window slides over the sensor plane with stride 1. Each window keeps its own linear-attention state. Parameters are shared; memories are not. An event at a pixel activates only the A=P² windows that cover it. Default P=3, so nine substates move per event. Interim outputs from those windows are summed into the event's new embedding. Sequence length stays the same, so later layers do not flood the whole map with activations as the receptive field grows.
The same event sits at different relative coordinates inside different windows. Sharing one embedding would throw away that prior. A position-aware projection (PAP) looks up a learned matrix by relative position and applies it on the way in and on the way out. On the Gen1 val set, dropping both PAPs collapses mAP from 0.335 to 0.014.
Sparse per-substate updates break the single-sequence form that makes linear attention train in parallel. Training therefore scatter-compute-gathers: a lookup table copies each event into every covering window and applies PAP, a stable sort by window index yields K independent subsequences, each subsequence runs the same linear attention, and a cached permutation gathers results back to event order before the sum. Time is still a parallel scan. Windows can run side by side.
SSLA-Det is the detector. The backbone has four stages, two SSLA layers each, with residuals and LayerNorm. The first three stages add sparse pooling and temporal dropout to shorten the sequence. The recurrence is a real-valued Linear Recurrent Unit with a Triton kernel. The raw input is two scalars: polarity and time delta. Each backbone output writes into that event's pixel. The YOLOX head uses only 1×1 convolutions, so one event touches one location and the whole net stays asynchronous. Four sizes, S/B/M/L, start at width 12/16/24/32 and double each stage.
Gen1 is automotive ATIS data at 304×240, two classes (car, pedestrian). The previous strongest asynchronous baseline, DAGr-L, sits at 0.321 mAP and 17.4 MFLOPS per event.
| Method | mAP | AP50 | MFLOPS/ev |
| DAGr-L | 0.321 | - | 17.4 |
| SSLA-S | 0.334 | 0.629 | 0.102 |
| SSLA-L | 0.375 | 0.675 | 0.724 |
| EventPillars (sync) | 0.531 | - | 5.08e4 |
The smallest model, SSLA-S, already beats DAGr-L on mAP at about 1/171 the compute. SSLA-L pushes asynchronous Gen1 mAP to 0.375 and still spends over 20× less than DAGr-L. Synchronous detectors clear 0.5 mAP, at thousands to tens of thousands of MFLOPS per event and millisecond latency.
On N-Caltech101, SSLA-L records 0.515 mAP, 0.743 AP50, and 0.926 MFLOPS/ev. DAGr-L is 0.732 AP50 at 18.9 MFLOPS/ev: +1.1 AP50 at roughly 1/20 the compute.
Under the same Gen1 training setup on 4×A800, an official PyTorch LSTM spends 1.05 hours per epoch; SSLA-S takes 0.25 hours, with mAP 0.334 vs 0.353. SSLA-B matches LSTM more closely (0.351 vs 0.353 mAP, 0.655 vs 0.631 AP50) in 0.28 hours. A recurrent C++ runtime on one Ryzen 9 9950X3D core processes a Gen1 event in 3.43 / 2.44 / 6.02 / 7.20 μs for S/B/M/L, below the 200 μs sensor transport delay. Smaller is not always faster; vectorization and memory traffic matter.
Ablations split the savings. Sparse SSLA versus a dense-activation MOS counterpart cuts per-event FLOPS from 388 to 1.02, about 380×, with mAP unchanged. Temporal dropout then cuts another 10×, taking mAP from 0.370 to 0.335. A plain linear attention, even at similar FLOPS, scores 0.001 mAP on val: the state is too coarse for boxes. P=2 yields 0.200 mAP; P=4 reaches 0.371 at 0.179 MFLOPS/ev. Default is P=3. Typical failures are motionless targets that emit no events, and false positives on missing labels.
Linear attention in event vision had stopped at classification. This paper ships an end-to-end asynchronous detector by structuring sparse state in space, rather than learning a token-to-memory router. Parallel training and event-level latency stay in the same model.
For low-latency perception, the accuracy-compute curve is usable today: single-digit microseconds on a CPU, an order of magnitude fewer FLOPS than graph nets, code released. This is not a sync-benchmark hunter. Against RVT or ERGO the accuracy gap is real. The intended setting is a control loop that must emit a box when the event lands.
The authors flag two items. There is no event-image fusion, though dense image features could in principle be injected into intermediate layers. Asynchronous and synchronous methods sit on different axes: SSLA-L is 0.375 mAP on Gen1, while sync SOTA exceeds 0.5. They treat that gap as a structural limit on aggregation, pretrained weights, and parameter count, not a missed hyperparameter.
A few caveats sit outside that list. N-Caltech101 is saccades in front of projected Caltech101 images, far from real motion; the asynchronous claim mostly rests on Gen1. CPU latency does not cash in the FLOPS win one-for-one; the authors point to FPGAs or neuromorphic chips for the next cut. Larger P raises both compute and GPU memory, so P=3 is a training-cost compromise. There is no report on crowded multi-object scenes or higher-resolution sensors. A stride-1 sliding window grows the number of substates with pixel count.