Lesson plan

State space model (deep learning)

In deep learning, state space models (SSMs) are a type of neural network used to process long streams of sequential data, such as text, speech, or sensor readings. Rather than repeatedly re-examining everything that came before, an SSM maintains a compact internal summary (the "state") that is updated as each new piece of the input arrives, an idea adapted from control theory, the branch of engineering concerned with tracking and steering changing systems. Compact representations make SSMs an efficient alternative to the Transformer architecture that underlies most modern large language models. A Transformer compares every element of its input with every other element, so doubling the length of the input roughly quadruples the work. In contrast, an SSM's effort grows in direct proportion to the input's length. As a result, SSMs can handle very long inputs (e.g., entire books, hours of audio, or DNA sequences) and can run with less computing effort and energy. Therefore, SSMs are well suited to running directly on phones and other small devices rather than in data centers. SSM-based methods have been adopted in commercial AI systems, including recent large language models such as Kimi K3, and prominent figures in the AI industry have described them as significant: Jensen Huang, chief executive of NVIDIA, called state space models "probably the next generation of Transformers."

Back to article

Article summary

In deep learning, state space models (SSMs) are a type of neural network used to process long streams of sequential data, such as text, speech, or sensor readings. Rather than repeatedly re-examining everything that came before, an SSM maintains a compact internal summary (the "state") that is updated as each new piece of the input arrives, an idea adapted from control theory, the branch of engineering concerned with tracking and steering changing systems. Compact representations make SSMs an efficient alternative to the Transformer architecture that underlies most modern large language models. A Transformer compares every element of its input with every other element, so doubling the length of the input roughly quadruples the work. In contrast, an SSM's effort grows in direct proportion to the input's length. As a result, SSMs can handle very long inputs (e.g., entire books, hours of audio, or DNA sequences) and can run with less computing effort and energy. Therefore, SSMs are well suited to running directly on phones and other small devices rather than in data centers. SSM-based methods have been adopted in commercial AI systems, including recent large language models such as Kimi K3, and prominent figures in the AI industry have described them as significant: Jensen Huang, chief executive of NVIDIA, called state space models "probably the next generation of Transformers."

Key claims

No claims recorded.

Discussion questions

  1. Summarise the article in your own words. Which single claim would you verify first, and why?
  2. Which claim in this article is most likely to be revised as the sources update? How would you check its current status?
  3. If two sources disagreed about one of these facts, what would you need in order to trust one over the other?

Bibliography

No sources recorded.

Key claims and bibliography are generated from the same tracked claims and sources as the article page.