Neural Operator: Learning Maps Between Function Spaces
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, Anima Anandkumar
cs.LG, math.NA
2021-08-19
This JMLR paper axiomatizes neural operators, proves discretization invariance plus universal approximation, and shows FNO at 0.0108 Darcy error versus DeepONet at about 0.048.
The ICLR FNO paper already had a working Fourier kernel. This JMLR paper asks a more basic question: which architectures even qualify as models on function spaces. A CNN, GNN, or transformer can ingest any resolution, yet its parameters or its error need not converge to a continuum operator under mesh refinement. DeepONet can emit a function and is a universal approximator, but it does not take input samples at arbitrary point sets. The missing object satisfies three properties at once: arbitrary input discretizations, query at arbitrary output points, convergence under refinement, and a parameter count that does not grow with the mesh.
PDE surrogates are the headline application. The definition is about continuous operators between Banach spaces, not about one equation.
A neural operator is a pointwise lift, several layers of a linear operator plus a pointwise activation, and a pointwise projection. The linear layer has to be an operator on functions, so the composition is an operator and discretization invariance can hold. Invariance is stated as three requirements: any input point set, any output query location, convergence to a continuum limit, with a fixed number of parameters.
Four practical linear layers are given:
The theory proves uniform approximation of continuous operators on compact sets, and approximation in the Bochner norm. Input and output spaces include Lebesgue, Sobolev, continuous, and C^k classes, broader than Hilbert-only or C^0-only results. Transformers sit in the same picture: attention is a particular integral kernel. Training with relative L2 rather than MSE cuts test error by about half in their runs; the normalization is acting as regularization.
Darcy, resolutions 85 to 421:
| Method | Relative error |
| DeepONet | 0.046-0.049 |
| PCANN | 0.030 |
| RBM | 0.024-0.026 |
| GNO | 0.033-0.037 |
| FNO | 0.0108-0.0098 |
On Burgers, FNO is about 0.0018 (0.0007 with GeLU). DeepONet sits at 0.057 to 0.086 and degrades with resolution. A zero-hidden-layer kernel net on 1D Poisson matches the Green's function at 10^{-7} relative error with N=1000, which is evidence that a model trained in a weak norm can generalize in a stronger topology.
Navier-Stokes numbers match the conference FNO paper: 0.0086 for FNO-3D at ν=10^{-3}, N=1000; 0.082 at ν=10^{-4} once N=10000. In a resolution sweep FNO barely moves while U-Net degrades. GNO trained at 16x16 evaluates at 241x241. Wall-clock per epoch on a V100 is about 4 s for GNO and FNO, 8 s for MGNO, 20 s for LNO. At tiny parameter counts GNO/MGNO are stabler; FNO at 10^3 parameters is about 0.20 Darcy error and only reaches 0.02-0.015 at 10^5-10^6. Noise robustness: after clean training, 10% input noise yields 0.018 on Burgers, 0.094 on advection, and almost no change on Darcy (0.011 to 0.012).
This is the definition-plus-theorem-plus-four-kernels handbook for operator learning, not a rerun of FNO. Table 1 is the sentence people still quote: at the time, only neural operators combined discretization invariance with universal approximation. Later PDE surrogates, weather models, and scientific-ML backbones are mostly picking among these kernels or swapping the basis inside the same integral skeleton.
FNO remains the default on smooth fields and uniform grids. On irregular meshes or tiny parameter budgets, GNO and MGNO are not obsolete. DeepONet loses in this matched-width setup; a CNN branch plus PCA trunk can close much of the gap, so architecture details move the ranking.
A full O(J^2) integral still beats FNO by about 40%. FNO's speed comes from restricting the kernel to a convolution, which is not an upper bound on integral operators. FNO is brittle on non-smoothing problems such as advection. The authors point to later UNet-branch hybrids that handle discontinuities, at the cost of leaving strict discretization invariance. Universal approximation is an existence result; it does not give the compute needed for a given error. Whether empirical-risk minimization finds a global min is explicitly unanalyzed. Navier-Stokes experiments stay on the periodic torus; engineering geometries are a different story.