Dynamic topic model
Dynamic topic models are generative models in natural language processing that analyze the evolution of latent topics within a collection of documents over time.
≈ 7 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
Dynamic topic models are generative models in natural language processing that analyze the evolution of latent topics within a collection of documents over time. This family of topic models was proposed by David Blei and John Lafferty and was initially an extension to Latent Dirichlet Allocation (LDA) that can handle sequential documents.[1]
In LDA, both the order the words appear in a document and the order the documents appear in the corpus are oblivious to the model. Whereas words are still assumed to be exchangeable, in a dynamic topic model the order of the documents plays a fundamental role. More precisely, the documents are binned by time period, with the assumption that the topics of each group persist and evolved from the set of the previous slice.
Since its inception, dynamic topic modeling has seen the introduction of several new models, including BERTopic.[2][3]
Topics
edit edit sourceSimilarly to LDA and pLSA, in a dynamic topic model, each document is viewed as a mixture of unobserved topics. Furthermore, each topic defines a multinomial distribution over a set of terms. Thus, for each word of each document, a topic is drawn from the mixture and a term is subsequently drawn from the multinomial distribution corresponding to that topic.
The topics, however, evolve over time. For instance, the two most likely terms of a topic at time t could be "network" and "Zipf" (in descending order) while the most likely ones at time t+1 could be "Zipf" and "percolation" (in descending order).
Model
edit edit sourceDefine
- αt as the per-document topic distribution at time t.
- βt,k as the word distribution of topic k at time t.
- ηt,d as the topic distribution for document d in time t,
- zt,d,n as the topic for the nth word in document d in time t, and
- wt,d,n as the specific word.
In this model, the multinomial distributions αt+1 and βt+1,k are generated from αt and βt,k, respectively. Even though multinomial distributions are usually written in terms of the mean parameters, representing them in terms of the natural parameters is better in the context of dynamic topic models.
The former representation has some disadvantages due to the fact that the parameters are constrained to be non-negative and sum to one.[4] When defining the evolution of these distributions, one would need to assure that such constraints were satisfied. Since both distributions are in the exponential family, one solution to this problem is to represent them in terms of the natural parameters, that can assume any real value and can be individually changed.
Using the natural parameterization, the dynamics of the topic model are given by
- βt,k|βt−1,k∼N(βt−1,k,σ2I)
and
- αt|αt−1∼N(αt−1,δ2I).
The generative process at time slice 't' is therefore:
- Draw topics βt,k|βt−1,k∼N(βt−1,k,σ2I)∀k
- Draw mixture model αt|αt−1∼N(αt−1,δ2I)
- For each document:
- Draw ηt,d∼N(αt,a2I)
- For each word:
- Draw topic Zt,d,n∼Mult(π(ηt,d))
- Draw word Wt,d,n∼Mult(π(βt,Zt,d,n))
where π(x) is a mapping from the natural parameterization x to the mean parameterization, namely
- π(xi)=exp(xi)∑iexp(xi).
Inference
edit edit sourceIn the dynamic topic model, only Wt,d,n is observable. Learning the other parameters constitutes an inference problem. Blei and Lafferty argue that applying Gibbs sampling to do inference in this model is more difficult than in static models, due to the nonconjugacy of the Gaussian and multinomial distributions. They propose the use of variational methods, in particular, the Variational Kalman Filtering and the Variational Wavelet Regression.
Applications
edit edit sourceIn the original paper, a dynamic topic model is applied to the corpus of Science articles published between 1881 and 1999 aiming to show that this method can be used to analyze the trends of word usage inside topics.[1] The authors also show that the model trained with past documents is able to fit documents of an incoming year better than LDA.
A continuous dynamic topic model was developed by Wang et al. and applied to predict the timestamp of documents.[5]
Going beyond text documents, dynamic topic models were used to study musical influence, by learning musical topics and how they evolve in recent history.[6]
References
edit edit source- ^ a b Blei, David M; Lafferty, John D (2006). "Dynamic topic models". Proceedings of the 23rd international conference on Machine learning - ICML '06. ICML'06. pp. 113–120. doi:10.1145/1143844.1143859. ISBN 978-1-59593-383-6. S2CID 5405229. Wikipedia revision 1365160207, retrieved 2026-11-19
- ^ Karakkaparambil James, Charu; Nagda, Mayank; Haji Ghassemi, Nooshin; Kloft, Marius; Fellenz, Sophie (2024). Ku, Lun-Wei; Martins, Andre; Srikumar, Vivek (eds.). "Evaluating Dynamic Topic Models". Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics: 160–176. doi:10.18653/v1/2024.acl-long.11. Wikipedia revision 1365160207, retrieved 2026-11-19
- ^ Grootendorst, Maarten (2022-03-11). "BERTopic: Neural topic modeling with a class-based TF-IDF procedure". arXiv:2203.05794 [cs.CL]. Wikipedia revision 1365160207, retrieved 2026-11-19
- ^ Rennie, Jason D. M. "Mixtures of Multinomials" (PDF). Retrieved 5 December 2011. Wikipedia revision 1365160207, retrieved 2026-11-19
- ^ Wang, Chong; Blei, David; Heckerman, David (2008). "Continuous Time Dynamic Topic Models". Proceedings of ICML. ICML '08. Wikipedia revision 1365160207, retrieved 2026-11-19
- ^ Shalit, Uri; Weinshall, Daphna; Chechik, Gal (2013). "Modeling musical influence with topic models" (PDF). Journal of Machine Learning Research. Wikipedia revision 1365160207, retrieved 2026-11-19
- Argument mining
- Collocation extraction
- Concept mining
- Coreference resolution
- Deep linguistic processing
- Distant reading
- Information extraction
- Knowledge extraction
- Logic translation
- Named-entity recognition
- Ontology learning
- Parsing
- Part-of-speech tagging
- Semantic analysis
- Semantic role labeling
- Semantic decomposition
- Semantic similarity
- Sentiment analysis
- Stance detection
- Stylometry
- Terminology extraction
- Text mining
- Textual entailment
- Truecasing
- Word-sense disambiguation
- Word-sense induction
datasets and corpora
and data capture
reviewing
user interface
Text derived from https://en.wikipedia.org/w/index.php?oldid=1365160207 under CC BY-SA 4.0; modifications have been made.
References
- ↩Wikipedia “Dynamic topic model” publication date (×1) · confidence 0.95 revision URL