Neural operators

Uses your system voice; voice and rate in settings.

Snapshot

≈ 6 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

Neural operators are a class of deep learning architectures designed to learn maps between infinite-dimensional function spaces. Neural operators represent an extension of traditional artificial neural networks, marking a departure from the typical focus on learning mappings between finite-dimensional Euclidean spaces or finite sets. Neural operators directly learn operators between function spaces; they can receive input functions, and the output function can be evaluated at any discretization.

The primary application of neural operators is in learning surrogate maps for the solution operators of partial differential equations (PDEs), which are critical tools in modeling the natural environment. Standard PDE solvers can be time-consuming and computationally intensive, especially for complex systems. Neural operators have demonstrated improved performance in solving PDEs compared to existing machine learning methodologies while being significantly faster than numerical solvers.

Evidence and sources

Neural operators have also been applied to various scientific and engineering disciplines such as turbulent flow modeling, computational mechanics, graph-structured data, and the geosciences.…

Neural operators are a class of deep learning architectures designed to learn maps between infinite-dimensional function spaces. Neural operators represent an extension of traditional artificial neural networks, marking a departure from the typical focus on learning mappings between finite-dimensional Euclidean spaces or finite sets. Neural operators directly learn operators between function spaces; they can receive input functions, and the output function can be evaluated at any discretization.

The primary application of neural operators is in learning surrogate maps for the solution operators of partial differential equations (PDEs), which are critical tools in modeling the natural environment. Standard PDE solvers can be time-consuming and computationally intensive, especially for complex systems. Neural operators have demonstrated improved performance in solving PDEs compared to existing machine learning methodologies while being significantly faster than numerical solvers.

Neural operators have also been applied to various scientific and engineering disciplines such as turbulent flow modeling, computational mechanics, graph-structured data, and the geosciences. In particular, they have been applied to learning stress-strain fields in materials, classifying complex data like spatial transcriptomics, predicting multiphase flow in porous media, and carbon dioxide migration simulations.

Finally, the operator learning paradigm allows learning maps between function spaces, and is different from parallel ideas of learning maps from finite-dimensional spaces to function spaces, and subsumes these settings as special cases when limited to a fixed input resolution. \mathcal{G}_\phi := \mathcal{Q} \circ \sigma(W_T + \mathcal{K}_T + b_T) \circ \cdots \circ \sigma(W_1 + \mathcal{K}_1 + b_1) \circ \mathcal{P}, where \mathcal{P}, \mathcal{Q} are the lifting (lifting the codomain of the input function to a higher dimensional space) and projection (projecting the codomain of the intermediate function to the output dimension) operators, respectively.

These operators act pointwise on functions and are typically parametrized as multilayer perceptrons. \sigma is a pointwise nonlinearity, such as a rectified linear unit (ReLU), or a Gaussian error linear unit (GeLU). Each layer t=1, \dots, T has a respective local operator W_t (usually parameterized by a pointwise neural network), a kernel integral operator \mathcal{K}_t , and a bias function b_t .

Given some intermediate functional representation v_t with domain D in the t -th hidden layer, a kernel integral operator \mathcal{K}_\phi is defined as (\mathcal{K}_\phi v_t)(x) := \int_D \kappa_\phi(x, y, v_t(x), v_t(y))v_t(y)dy, where the kernel \kappa_\phi is a learnable implicit neural network, parametrized by \phi . In practice, one is often given the input function to the neural operator at a specific resolution. For instance, consider the setting where one is given the evaluation of v_t at n points \{y_j\}_j^n .

Borrowing from Nyström integral approximation methods such as Riemann sum integration and Gaussian quadrature, the above integral operation can be computed as follows: \int_D \kappa_\phi(x, y, v_t(x), v_t(y))v_t(y)dy\approx \sum_j^n \kappa_\phi(x, y_j, v_t(x), v_t(y_j))v_t(y_j)\Delta_{y_j}, where \Delta_{y_j} is the sub-area volume or quadrature weight associated to the point y_j . Thus, a simplified layer can be computed as v_{t+1}(x) \approx \sigma\left(\sum_j^n \kappa_\phi(x, y_j, v_t(x), v_t(y_j))v_t(y_j)\Delta_{y_j} + W_t(v_t(y_j)) + b_t(x)\right).

The above approximation, along with parametrizing \kappa_\phi as an implicit neural network, results in the graph neural operator (GNO). There have been various parameterizations of neural operators for different applications. These typically differ in their parameterization of \kappa . The most popular instantiation is the Fourier neural operator (FNO).

FNO takes \kappa_\phi(x, y, v_t(x), v_t(y)) := \kappa_\phi(x-y) and by applying the convolution theorem, arrives at the following parameterization of the kernel integral operator: (\mathcal{K}_\phi v_t)(x) = \mathcal{F}^{-1} (R_\phi \cdot (\mathcal{F}v_t))(x), where \mathcal{F} represents the Fourier transform and R_\phi represents the Fourier transform of some periodic function \kappa_\phi . That is, FNO parameterizes the kernel integration directly in Fourier space, using a prescribed number of Fourier modes.

How it works

When the grid at which the input function is presented is uniform, the Fourier transform can be approximated using the discrete Fourier transform (DFT) with frequencies below some specified threshold. The discrete Fourier transform can be computed using a fast Fourier transform (FFT) implementation. Training neural operators is similar to the training process for a traditional neural network. Neural operators are typically trained in some Lp norm or Sobolev norm.

In particular, for a dataset \{(a_i, u_i)\}_{i=1}^N of size N , neural operators minimize (a discretization of) \mathcal{L}_\mathcal{U}(\{(a_i, u_i)\}_{i=1}^N) := \sum_{i=1}^N \|u_i - \mathcal{G}_\theta (a_i) \|_\mathcal{U}^2 , where \|\cdot \|_\mathcal{U} is a norm on the output function space \mathcal{U} . Neural operators can be trained directly using backpropagation and gradient descent-based methods. Another training paradigm is associated with physics-informed machine learning.

In particular, physics-informed neural networks (PINNs) use complete physics laws to fit neural networks to solutions of PDEs. Extensions of this paradigm to operator learning are broadly called physics-informed neural operators (PINO), where loss functions can include full physics equations or partial physical laws. As opposed to standard PINNs, the PINO paradigm incorporates a data loss (as defined above) in addition to the physics loss \mathcal{L}_{PDE}(a, \mathcal{G}_\theta (a)) .

The physics loss \mathcal{L}_{PDE}(a, \mathcal{G}_\theta (a)) quantifies how much the predicted solution of \mathcal{G}_\theta (a) violates the PDEs equation for the input a .

Text derived from https://en.wikipedia.org/w/index.php?oldid=1378634860 under CC BY-SA 4.0; modifications have been made.

Provenance

snapshot enwiki-001378634860 · captured 1791188862.0 · version history

Cite this page as: encyclopegia.org, “Neural operators”, snapshot enwiki-001378634860. https://encyclopegia.org/a/neural-operators

Sources cited in this revision

9 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.