State space model (deep learning)

Uses your system voice; voice and rate in settings.

Snapshot

≈ 5 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

In deep learning, state space models (SSMs) are a type of neural network used to process long streams of sequential data, such as text, speech, or sensor readings. Rather than repeatedly re-examining everything that came before, an SSM maintains a compact internal summary (the "state") that is updated as each new piece of the input arrives, an idea adapted from control theory, the branch of engineering concerned with tracking and steering changing systems.

How it works

Compact representations make SSMs an efficient alternative to the Transformer architecture that underlies most modern large language models. A Transformer compares every element of its input with every other element, so doubling the length of the input roughly quadruples the work. In contrast, an SSM's effort grows in direct proportion to the input's length. As a result, SSMs can handle very long inputs (e.g., entire books, hours of audio, or DNA sequences) and can run with less computing effort and energy.

Evidence and sources

Therefore, SSMs are well suited to running directly on phones and other small devices rather than in data centers. SSM-based methods have been adopted in commercial AI systems, including recent large language models such as Kimi K3, and prominent figures in the AI industry have described them as significant: Jensen Huang, chief executive of NVIDIA, called state space models "probably the next generation of Transformers."

In deep learning, state space models (SSMs) are a type of neural network used to process long streams of sequential data, such as text, speech, or sensor readings. Rather than repeatedly re-examining everything that came before, an SSM maintains a compact internal summary (the "state") that is updated as each new piece of the input arrives, an idea adapted from control theory, the branch of engineering concerned with tracking and steering changing systems.

Compact representations make SSMs an efficient alternative to the Transformer architecture that underlies most modern large language models. A Transformer compares every element of its input with every other element, so doubling the length of the input roughly quadruples the work. In contrast, an SSM's effort grows in direct proportion to the input's length. As a result, SSMs can handle very long inputs (e.g., entire books, hours of audio, or DNA sequences) and can run with less computing effort and energy.

Therefore, SSMs are well suited to running directly on phones and other small devices rather than in data centers.

SSM-based methods have been adopted in commercial AI systems, including recent large language models such as Kimi K3, and prominent figures in the AI industry have described them as significant: Jensen Huang, chief executive of NVIDIA, called state space models "probably the next generation of Transformers." x'(t) = Ax(t) + Bu(t) y(t) = Cx(t) + Du(t) where A is the state matrix, B is the control matrix, C is the output matrix, and D is a direct feedthrough term (often treated as a skip connection in deep learning applications).

SSMs offer several key advantages: they can naturally handle continuous data, automatically adapt to different sampling rates without retraining, and provide mathematically tractable analysis of their dynamics. Through discretization, SSMs can be viewed from three complementary perspectives: as continuous-time systems, as recurrent networks during inference, and as convolutional models during training.

Mamba-2 achieves faster computation by leveraging matrix multiplication primitives and tensor cores on modern GPUs, allowing for larger state expansion (typically N=128-256 compared to N=16 in Mamba) while remaining computationally efficient.

The model also enables better system-level optimizations including tensor parallelism and sequence parallelism. x'(t) = Ax(t) + Bu(t) y(t) = Cx(t) + Du(t) where: x(t) ∈ ℝᴺ is the state vector (N-dimensional latent state) u(t) ∈ ℝ is the input signal y(t) ∈ ℝ is the output signal A ∈ ℝᴺˣᴺ is the state transition or dynamics matrix B ∈ ℝᴺˣ¹ is the control matrix C ∈ ℝ¹ˣᴺ is the output matrix D ∈ ℝ is the feedthrough term x̄ₖ = Āxₖ₋₁ + B̄uₖ yₖ = Cx̄ₖ + Duₖ where the discrete parameters are: Ā = exp(ΔA) B̄ = (exp(ΔA) - I)A⁻¹B This discretization highlights two complementary views: Recurrent view: Efficient O(N) linear-time inference by maintaining state Convolutional view: Parallel O(N log N) training via FFT-based convolutions The convolution kernel K̄ can be precomputed as: K̄ₖ = CĀᵏB̄ This duality allows SSMs to combine the inference efficiency of RNNs with the training parallelism of CNNs and Transformers.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1378095071 under CC BY-SA 4.0; modifications have been made.

Provenance

snapshot enwiki-001378095071 · captured 1790975789.0 · version history

Cite this page as: encyclopegia.org, “State space model (deep learning)”, snapshot enwiki-001378095071. https://encyclopegia.org/a/state-space-model-deep-learning

Sources cited in this revision

21 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.