Actor-critic algorithm
The actor-critic algorithm (AC) is a family of reinforcement learning (RL) algorithms that combine policy-based RL algorithms such as policy gradient methods, and value-based RL algorithms such as value iteration, Q-learning, SARSA, and TD…
≈ 9 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
The actor-critic algorithm (AC) is a family of reinforcement learning (RL) algorithms that combine policy-based RL algorithms such as policy gradient methods, and value-based RL algorithms such as value iteration, Q-learning, SARSA, and TD learning.[1]
An AC algorithm consists of two main components: an "actor" that determines which actions to take according to a policy function, and a "critic" that evaluates those actions according to a value function.[2] Some AC algorithms are on-policy, some are off-policy. Some apply to either continuous or discrete action spaces. Some work in both cases.
Overview
edit edit sourceThe actor-critic methods can be understood as an improvement over pure policy gradient methods like REINFORCE via introducing a baseline.
Actor
edit edit sourceThe actor uses a policy function π(a|s), while the critic estimates either the value function V(s), the action-value Q-function Q(s,a), the advantage function A(s,a), or any combination thereof.
The actor is a parameterized function πθ, where θ are the parameters of the actor. The actor takes as argument the state of the environment s and produces a probability distribution πθ(⋅|s).
If the action space is discrete, then ∑aπθ(a|s)=1. If the action space is continuous, then ∫aπθ(a|s)da=1.
The goal of policy optimization is to improve the actor. That is, to find some θ that maximizes the expected episodic reward J(θ):J(θ)=𝔼πθ[∑t=0Tγtrt]where γ is the discount factor, rt is the reward at step t, and T is the time-horizon (which can be infinite).
The goal of policy gradient method is to optimize J(θ) by gradient ascent on the policy gradient ∇J(θ).
As detailed on the policy gradient method page, there are many unbiased estimators of the policy gradient:∇θJ(θ)=𝔼πθ[∑0≤j≤T∇θlnπθ(Aj|Sj)⋅Ψj|S0=s0]where Ψj is a linear sum of the following:
- ∑0≤i≤T(γiRi).
- γj∑j≤i≤T(γi−jRi): the REINFORCE algorithm.
- γj∑j≤i≤T(γi−jRi)−b(Sj): the REINFORCE with baseline algorithm. Here b is an arbitrary function.
- γj(Rj+γVπθ(Sj+1)−Vπθ(Sj)): TD(1) learning.
- γjQπθ(Sj,Aj).
- γjAπθ(Sj,Aj): Advantage Actor-Critic (A2C).[3]
- γj(Rj+γRj+1+γ2Vπθ(Sj+2)−Vπθ(Sj)): TD(2) learning.
- γj(∑k=0n−1γkRj+k+γnVπθ(Sj+n)−Vπθ(Sj)): TD(n) learning.
- γj∑n=1∞λn−11−λ⋅(∑k=0n−1γkRj+k+γnVπθ(Sj+n)−Vπθ(Sj)): TD(λ) learning, also known as GAE (generalized advantage estimate).[4] This is obtained by an exponentially decaying sum of the TD(n) learning terms.
Critic
edit edit sourceIn the unbiased estimators given above, certain functions such as Vπθ,Qπθ,Aπθ appear. These are approximated by the critic. Since these functions all depend on the actor, the critic must learn alongside the actor. The critic is learned by value-based RL algorithms.
For example, if the critic is estimating the state-value function Vπθ(s), then it can be learned by any value function approximation method. Let the critic be a function approximator Vϕ(s) with parameters ϕ.
The simplest example is TD(1) learning, which trains the critic to minimize the TD(1) error:δi=Ri+γVϕ(Si+1)−Vϕ(Si)The critic parameters are updated by gradient descent on the squared TD error:ϕ←ϕ−α∇ϕ(δi)2=ϕ+αδi∇ϕVϕ(Si)where α is the learning rate. Note that the gradient is taken with respect to the ϕ in Vϕ(Si) only, since the ϕ in γVϕ(Si+1) constitutes a moving target, and the gradient is not taken with respect to that. This is a common source of error in implementations that use automatic differentiation, and requires "stopping the gradient" at that point.
Similarly, if the critic is estimating the action-value function Qπθ, then it can be learned by Q-learning or SARSA. In SARSA, the critic maintains an estimate of the Q-function, parameterized by ϕ, denoted as Qϕ(s,a). The temporal difference error is then calculated as δi=Ri+γQθ(Si+1,Ai+1)−Qθ(Si,Ai). The critic is then updated byθ←θ+αδi∇θQθ(Si,Ai)The advantage critic can be trained by training both a Q-function Qϕ(s,a) and a state-value function Vϕ(s), then let Aϕ(s,a)=Qϕ(s,a)−Vϕ(s). Although, it is more common to train just a state-value function Vϕ(s), then estimate the advantage by[3]Aϕ(Si,Ai)≈∑j∈0:n−1γjRi+j+γnVϕ(Si+n)−Vϕ(Si)Here, n is a positive integer. The higher n is, the more lower is the bias in the advantage estimation, but at the price of higher variance.
The Generalized Advantage Estimation (GAE) introduces a hyperparameter λ that smoothly interpolates between Monte Carlo returns (λ=1, high variance, no bias) and 1-step TD learning (λ=0, low variance, high bias). This hyperparameter can be adjusted to pick the optimal bias-variance trade-off in advantage estimation. It uses an exponentially decaying average of n-step returns with λ being the decay strength.[4]
Variants
edit edit source- Asynchronous Advantage Actor-Critic (A3C): Parallel and asynchronous version of A2C.[3]
- Soft Actor-Critic (SAC): Incorporates entropy maximization for improved exploration.[5]
- Deep Deterministic Policy Gradient (DDPG): Specialized for continuous action spaces.[6]
See also
edit edit sourceReferences
edit edit source- ^ Arulkumaran, Kai; Deisenroth, Marc Peter; Brundage, Miles; Bharath, Anil Anthony (November 2017). "Deep Reinforcement Learning: A Brief Survey". IEEE Signal Processing Magazine. 34 (6): 26–38. arXiv:1708.05866. Bibcode:2017ISPM...34...26A. doi:10.1109/MSP.2017.2743240. ISSN 1053-5888.
- ^ Konda, Vijay; Tsitsiklis, John (1999). "Actor-Critic Algorithms". Advances in Neural Information Processing Systems. 12. MIT Press.
- ^ a b c Mnih, Volodymyr; Badia, Adrià Puigdomènech; Mirza, Mehdi; Graves, Alex; Lillicrap, Timothy P.; Harley, Tim; Silver, David; Kavukcuoglu, Koray (2016-06-16), Asynchronous Methods for Deep Reinforcement Learning, arXiv:1602.01783
- ^ a b Schulman, John; Moritz, Philipp; Levine, Sergey; Jordan, Michael; Abbeel, Pieter (2018-10-20), High-Dimensional Continuous Control Using Generalized Advantage Estimation, arXiv:1506.02438
- ^ Haarnoja, Tuomas; Zhou, Aurick; Hartikainen, Kristian; Tucker, George; Ha, Sehoon; Tan, Jie; Kumar, Vikash; Zhu, Henry; Gupta, Abhishek (2019-01-29), Soft Actor-Critic Algorithms and Applications, arXiv:1812.05905
- ^ Lillicrap, Timothy P.; Hunt, Jonathan J.; Pritzel, Alexander; Heess, Nicolas; Erez, Tom; Tassa, Yuval; Silver, David; Wierstra, Daan (2019-07-05), Continuous control with deep reinforcement learning, arXiv:1509.02971
- Konda, Vijay R.; Tsitsiklis, John N. (January 2003). "On Actor-Critic Algorithms". SIAM Journal on Control and Optimization. 42 (4): 1143–1166. doi:10.1137/S0363012901385691. ISSN 0363-0129.
- Sutton, Richard S.; Barto, Andrew G. (2018). Reinforcement learning: an introduction. Adaptive computation and machine learning series (2 ed.). Cambridge, Massachusetts: The MIT Press. ISBN 978-0-262-03924-6.
- Bertsekas, Dimitri P. (2019). Reinforcement learning and optimal control (2 ed.). Belmont, Massachusetts: Athena Scientific. ISBN 978-1-886529-39-7.
- Grossi, Csaba (2010). Algorithms for Reinforcement Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning (1 ed.). Cham: Springer International Publishing. ISBN 978-3-031-00423-0.
- Grondman, Ivo; Busoniu, Lucian; Lopes, Gabriel A. D.; Babuska, Robert (November 2012). "A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients". IEEE Transactions on Systems, Man, and Cybernetics - Part C: Applications and Reviews. 42 (6): 1291–1307. Bibcode:2012ITHMS..42.1291G. doi:10.1109/TSMCC.2012.2218595. ISSN 1094-6977.
- Automated reasoning
- Automated planning
- Constraint satisfaction
- Knowledge representation
- Parameter
- Loss functions
- Regression
- Clustering
- Gradient descent
- Backpropagation
- Attention
- Convolution
- Normalization
- Activation
- Gating
- Weight initialization
- Regularization
- Datasets
- Prompt engineering
- Reinforcement learning
- Diffusion
- Latent diffusion model
- Autoregression
- Adversary
- RAG
- Uncanny valley
- LLM post-training
- RLHF
- Self-supervised learning
- Reflection
- Recursive self-improvement
- Hallucination
- Word embedding
- Vibe coding
- Blended AI
- Open-source AI
- Sovereign AI
- Symbolic AI
- Neuro-symbolic AI
- Situated approach
- Actor-critic algorithm
- Alan Turing
- Warren Sturgis McCulloch
- Walter Pitts
- John von Neumann
- Christopher D. Manning
- Claude Shannon
- Shun'ichi Amari
- Kunihiko Fukushima
- Takeo Kanade
- Marvin Minsky
- John McCarthy
- Nathaniel Rochester
- Allen Newell
- Cliff Shaw
- Herbert A. Simon
- Oliver Selfridge
- Frank Rosenblatt
- Bernard Widrow
- Joseph Weizenbaum
- Seymour Papert
- Seppo Linnainmaa
- Paul Werbos
- Geoffrey Hinton
- John Hopfield
- Jürgen Schmidhuber
- Yann LeCun
- Yoshua Bengio
- Lotfi A. Zadeh
- Stephen Grossberg
- Alex Graves
- James Goodnight
- Andrew Ng
- Fei-Fei Li
- Alex Krizhevsky
- Ilya Sutskever
- Oriol Vinyals
- Quoc V. Le
- Ian Goodfellow
- Demis Hassabis
- David Silver
- Andrej Karpathy
- Ashish Vaswani
- Noam Shazeer
- Aidan Gomez
- John Schulman
- Mustafa Suleyman
- Jan Leike
- Daniel Kokotajlo
- François Chollet
architectures
- Neural Turing machine
- Differentiable neural computer
- Transformer
- Recurrent neural network (RNN)
- Long short-term memory (LSTM)
- Gated recurrent unit (GRU)
- Echo state network
- Multilayer perceptron (MLP)
- Convolutional neural network (CNN)
- Residual neural network (RNN)
- Highway network
- Mamba
- Autoencoder
- Variational autoencoder (VAE)
- Generative adversarial network (GAN)
- Graph neural network (GNN)
and economic
- AI boom
- AI bubble
- AI data center
- AI effect
- AI infrastructure
- AI literacy
- AI slop
- AI winter
- Anthropomorphism
- Arms race
- Competition
- Environmental impact
- Explainable AI
- Generative engine optimization
- In architecture
- In education
- In fiction
- In healthcare
- In marketing
- In video games
- In visual art
- Military applications
- Workplace impact
Text derived from https://en.wikipedia.org/w/index.php?oldid=1348373706 under CC BY-SA 4.0; modifications have been made.