FreeFlow: A Bias-free Hierarchical Transformer for Optical Flow Estimation
Vladislav Bargatin, Alexander Yakovenko, Khaled Abud, Dmitriy Vatolin
ECCV 2026
cs.CV
2026-09-10
Feed-forward transformer optical flow with no correlation volume, warping, or iteration. FreeFlow-L: Sintel 0.68/1.48 EPE, KITTI 3.23 Fl-all, Spring 3.192 1px, 2.58 GB at 1080p.
Optical flow estimates a 2D displacement at every pixel between two frames. FlowNet showed a feed-forward net can regress that field; later work put the classical parts back in: pyramids, feature warping, all-pairs correlation volumes, iterative updates, convex upsampling. RAFT made that recipe work, and most follow-ups stacked modules on top. Accuracy rose. The pipeline hardened: harder to change, harder to scale, harder to reuse for other dense correspondence jobs.
Detection, segmentation, and depth have already moved to generic vision transformers with thin heads. Optical flow is behind. CroCo-Flow is close to a pure transformer, yet high resolution still needs tiling. WAFT and GeoViT drop the cost volume and keep iterative warping. The open question is whether those task-specific parts can come out and the method can still sit at the top of the leaderboards.
FreeFlow is a Siamese encoder plus a decoder that emits flow in one forward pass. No correlation volume, no warping, no iterative refinement. Each image is cut into 8×8 patches. Decoder blocks run self-attention on the current tokens and cross-attention from view 1 queries to view 2 keys and values, so the two frames keep exchanging information.
Three attention patterns cycle in a fixed order. Window attention partitions the H/8×W/8 token map into a 4×4 grid of 16 blocks and attends inside each block. Shifted-window attention repeats that after a half-window shift so signals cross block borders. Global attention downsamples 2× with a stride-2 convolution, runs full attention at the lower resolution, and upsamples with a transposed convolution, which keeps the quadratic cost in line with the window blocks. Positions use RoPE. Attention logits are multiplied by the log of the token count, which helps when inference resolution exceeds training resolution.
The head is three layers: a 3×3 convolution to 4D channels, a 1×1 to 4096, and a stride-8 transposed convolution back to full resolution. It outputs five channels, two for flow and three for mixture-of-Laplace uncertainty. No DPT-style multi-layer fusion, no convex upsampling.
Training has two stages. Cross-view completion follows CroCo on 3.7M pairs from ARKitScenes, MegaDepth, and 3DStreetView, 224×224, 346k steps. Because the backbone is hierarchical, mask tokens go in at the encoder, and the mask ratio is 0.95 rather than CroCo's 0.9: 8×8 patches sit closer to visible context, so the pretext task has to be harder. Optical-flow finetuning uses a TartanAir/Things/Sintel/KITTI/HD1K mix, 2× upsampled frames to match FullHD motion, and variable-aspect crops under a token budget. The large model takes 4–5 days to pretrain and about 3 days to finetune on 32 GPUs.
On Spring, FreeFlow-L records 1px 3.192, EPE 0.278, Fl 1.048, WAUC 95.235. Against WAFT-DAv2-a2 at EPE 0.304 and Fl 1.197, that is about 9% and 14% lower. 1080p inference uses 2.58 GB and 607 ms at 231M parameters; WAFT-DAv2-a2 uses 20.58 GB in the same table. The medium variant is 1.66 GB, 325 ms, 102M parameters, 1px 3.392. The small variant is 1.02 GB, 144 ms, 35M parameters, 1px 5.087.
Sintel Clean/Final is 0.68/1.48, below GeoViT's 0.79/1.88, and Final is also below the five-frame VideoFlow-MOF score of 1.65. KITTI-2015 Fl-all is 3.23, the lowest among non-stereo, non-multiframe methods. Multiframe MEMFOF sits at 2.94 and ARFlow at 2.85; FreeFlow does not catch those.
Ablations run on the Spring sub-validation split. With all three attention blocks and mask ratio 0.95, 1px is 0.624 and EPE is 0.157. At mask 0.9, dropping Global looks slightly better on the numbers and worse on motion consistency in the crops. Dropping shifted-window attention costs a clear amount of accuracy. S/M/L scale depth and width together, and accuracy rises with capacity.
The working assumption in optical flow has been that correlation volumes and iteration are required to lead the boards. FreeFlow puts a feed-forward transformer first among two-frame methods on Sintel, KITTI, and Spring. Those parts are useful inductive biases, not physical requirements. For 1080p inference without tiling seams, 2.58 GB is a number that fits on a workstation GPU. The architecture is uniform enough that a 35M variant still works.
The cost is equally clear. This is not a small download-and-run model. Pretrain plus finetune is a 32-GPU week, and multiframe methods still win on KITTI.
"Bias-free" is relative to correlation volumes, warping, iteration, and convex upsampling. Window partitions, RoPE, a conv head, and 8×8 patches remain; they are generic vision parts. Sintel and KITTI submissions upsample inputs 2× and downsample the flow, the same trick as MEMFOF, and that protocol has to be counted in cross-paper comparisons. Ablations use a shrunk model, width 256 and eight layers each in encoder and decoder, so they do not measure module contribution in the large variant. The supplement's experiment that puts iterative flow biases back in is not reported with numbers in the main text.