Retrieval-Augmented Generation
Retrieval-augmented generation is a method in which a language model is given passages retrieved from an external corpus at inference time, grounding its output in sources that can be cited.
≈ 18 min read
Names in other languages
- EnglishRetrieval-Augmented Generation
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
Retrieval-augmented generation (RAG) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources.[1][2] With RAG, LLMs first refer to a specified set of documents, then respond to user queries. These documents supplement information from the LLM's pre-existing training data.[3] This allows LLMs to use domain-specific and/or updated information that is not available in the training data.[3] For example, this enables LLM-based chatbots to access internal company data or generate responses based on authoritative sources. The technique was first proposed in 2020 and has since become a widely adopted approach in modern AI systems.
RAG improves LLMs by incorporating information retrieval before generating responses.[4] Unlike LLMs that rely on static training data, RAG pulls relevant text from databases, uploaded documents, or web sources.[1] According to Ars Technica, "RAG is a way of improving LLM performance, in essence by blending the LLM process with a web search or other document look-up process to help LLMs stick to the facts." This method helps reduce AI hallucinations,[4] which have caused chatbots to describe policies that don't exist, or recommend nonexistent legal cases to lawyers that are looking for citations to support their arguments.[5]
RAG also reduces the need to retrain LLMs with new data, saving on computational and financial costs.[1] Beyond efficiency gains, RAG also allows LLMs to include sources in their responses, so users can verify the cited sources. This provides greater transparency, as users can cross-check retrieved content to ensure accuracy and relevance.
The term retrieval-augmented generation (RAG) was introduced in a 2020 paper that described combining a parametric language model with a non-parametric external memory accessed through retrieval at inference time.[4]
Contents
RAG and LLM limitations
edit edit sourceLLMs can provide incorrect information. For example, when Google first demonstrated its LLM tool "Bard" (later re-branded to Google Gemini), the LLM provided incorrect information about the James Webb Space Telescope. This error contributed to a $100 billion decline in Google's stock value.[5] RAG is used to prevent these errors, but it does not solve all the problems. For example, LLMs can generate misinformation even when pulling from factually correct sources if they misinterpret the context. MIT Technology Review gives the example of an AI-generated response stating, "The United States has had one Muslim president, Barack Hussein Obama." The model retrieved this from the rhetorical chapter title "Barack Hussein Obama: America's First Muslim President?" in the book Faith in the New Millennium: The Future of Religion and American Politics.[6] The LLM did not "know" or "understand" the context of the title, generating a false statement.[3]
LLMs with RAG are programmed to prioritize new information. This technique has been called "prompt stuffing." Without prompt stuffing, the LLM's input is generated by a user; with prompt stuffing, additional relevant context is added to this input to guide the model's response. This approach provides the LLM with key information early in the prompt, encouraging it to prioritize the supplied data over pre-existing training knowledge.[7]
Process
edit edit sourceRetrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating an information-retrieval mechanism that allows models to access and utilize additional data beyond their original training set. Ars Technica notes that "when new information becomes available, rather than having to retrain the model, all that's needed is to augment the model's external knowledge base with the updated information ("augmentation").[5] IBM states that "in the generative phase, the LLM draws from the augmented prompt and its internal representation of its training data to synthesize" an answer.[1]
RAG key stages
edit edit sourceTypically, the data to be referenced is converted into LLM embeddings, numerical representations in the form of a large vector space. RAG can be used on unstructured (usually text), semi-structured, or structured data (for example knowledge graphs). These embeddings are then stored in a vector database to allow for document retrieval.
Given a user query, a document retriever is first called to select the most relevant documents that will be used to augment the query.[3][4] This comparison can be done using a variety of methods, which depend in part on the type of indexing used.[1]
The model feeds this relevant retrieved information into the LLM via prompt engineering of the user's original query. Newer implementations (as of 2023[update]) can also incorporate specific augmentation modules with abilities such as expanding queries into multiple domains and using memory and self-improvement to learn from previous retrievals.[citation needed]
Finally, the LLM can generate output based on both the query and the retrieved documents.[3][4] Some models incorporate extra steps to improve output, such as the re-ranking of retrieved information, context selection, and fine-tuning.
Applications
edit edit sourceRetrieval-augmented generation is used in applications where generated responses need to be grounded in external or frequently updated information.[citation needed]
In healthcare, RAG has been studied as a way to ground large language model outputs in external medical knowledge sources, although reviews have noted continuing challenges around evaluation, ethics, and clinical reliability.[8]
Improvements
edit edit sourceImprovements to the basic process above can be applied at different stages in the RAG flow.
Encoder
edit edit sourceThese methods focus on the encoding of text as either dense or sparse vectors. Sparse vectors, which encode the identity of a word, are typically dictionary-length and contain mostly zeros. Dense vectors, which encode meaning, are more compact and contain fewer zeros. Various enhancements can improve the way similarities are calculated in the vector stores (databases).[9]
- Performance improves by optimizing how vector similarities are calculated. Dot products enhance similarity scoring, while approximate nearest neighbor (ANN) searches improve retrieval efficiency over K-nearest neighbors (KNN) searches.[10]
- Accuracy may be improved with Late Interactions, which allow the system to compare words more precisely after retrieval. This helps refine document ranking and improve search relevance.[11]
- Hybrid vector approaches may be used to combine dense vector representations with sparse one-hot vectors, taking advantage of the computational efficiency of sparse dot products over dense vector operations.[9]
- Other retrieval techniques focus on improving accuracy by refining how documents are selected. Some retrieval methods combine sparse representations, such as SPLADE, with query expansion strategies to improve search accuracy and recall.[12]
Retriever-centric methods
edit edit sourceThese methods aim to enhance the quality of document retrieval in vector databases:
- Pre-training the retriever using the Inverse Cloze Task (ICT), a technique that helps the model learn retrieval patterns by predicting masked text within documents.[13]
- Supervised retriever optimization aligns retrieval probabilities with the generator model's likelihood distribution. This involves retrieving the top-k vectors for a given prompt, scoring the generated response's perplexity, and minimizing KL divergence between the retriever's selections and the model's likelihoods to refine retrieval.[14]
- Reranking techniques can refine retriever performance by prioritizing the most relevant retrieved documents during training.[15]
Language model
edit edit source Retro language model for RAG. Each Retro block consists of Attention, Chunked Cross Attention, and Feed Forward layers. Black-lettered boxes show data being changed, and blue lettering shows the algorithm performing the changes.By redesigning the language model with the retriever in mind, a 25-time smaller network can get comparable perplexity as its much larger counterparts.[16] Because it is trained from scratch, this method (Retro) incurs the high cost of training runs that the original RAG scheme avoided. The hypothesis is that by giving domain knowledge during training, Retro needs less focus on the domain and can devote its smaller weight resources only to language semantics. The redesigned language model is shown here.
It has been reported that Retro is not reproducible, so modifications were made to make it so. The more reproducible version is called Retro++ and includes in-context RAG.[17]
Chunking
edit edit sourceChunking involves various strategies for breaking up the data into vectors so the retriever can find details in it.
Different data styles have patterns that correct chunking can take advantage of.Hybrid search
edit edit sourceSometimes vector database searches (aka semantic search technique) can miss key facts needed to answer a user's question. One way to mitigate this is to do a traditional text search (aka full text search), combine those results to the text chunks linked to the retrieved vectors from the vector search, and feed the combined hybrid text (using effective scoring or reranking scoring) into the language model for generation.[18]
Challenges
edit edit sourceRAG does not prevent hallucinations in LLMs. According to Ars Technica, "It is not a direct solution because the LLM can still hallucinate around the source material in its response."[5]
While RAG improves the accuracy of large language models (LLMs), it does not eliminate all challenges. One limitation is that while RAG reduces the need for frequent model retraining, it does not remove it entirely. Additionally, LLMs may struggle to recognize when they lack sufficient information to provide a reliable response. Without specific training, models may generate answers even when they should indicate uncertainty. According to IBM, this issue can arise when the model lacks the ability to assess its own knowledge limitations.[1]
RAG poisoning
edit edit sourceRAG systems may retrieve factually correct but misleading sources, leading to errors in interpretation. In some cases, an LLM may extract statements from a source without considering its context, resulting in an incorrect conclusion. Additionally, when faced with conflicting information, RAG models may struggle to determine which source is accurate. The worst case outcome of this limitation is that the model may combine details from multiple sources producing responses that merge outdated and updated information in a misleading manner. According to the MIT Technology Review, these issues occur because RAG systems may misinterpret the data they retrieve.[3]
On September 2025, Israel paid Clock Tower X $6 million in order to sway public opinion to a more favourable view of the country (hasbara) after support plummeted due to its genocidal conduct during the Gaza war. Clock Tower X used RAG poisoning to flood the information space with sites and content sympathetic to Israel.[19][20]
References
edit edit source- ^ a b c d e f "What is retrieval-augmented generation?". IBM. 22 August 2023. Retrieved 7 March 2025. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Zhao, Penghao; Zhang, Hailin; Yu, Qinhan; Wang, Zhengren; Geng, Yunteng; Fu, Fangcheng; Yang, Ling; Zhang, Wentao; Jiang, Jie; Cui, Bin (March 2026). "Retrieval-Augmented Generation for AI-Generated Content: A Survey". Data Science and Engineering. 11 (1): 1–29. doi:10.1007/s41019-025-00335-5. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ a b c d e f "Why Google's AI Overviews gets things wrong". MIT Technology Review. 31 May 2024. Retrieved 7 March 2025. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ a b c d e Lewis, Patrick; Perez, Ethan; Piktus, Aleksandra; Petroni, Fabio; Karpukhin, Vladimir; Goyal, Naman; Küttler, Heinrich; Lewis, Mike; Yih, Wen-tau; Rocktäschel, Tim; Riedel, Sebastian; Kiela, Douwe (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". Advances in Neural Information Processing Systems. 33. Curran Associates, Inc.: 9459–9474. arXiv:2005.11401. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ a b c d "Can a technology called RAG keep AI models from making stuff up?". Ars Technica. 6 June 2024. Retrieved 7 March 2025. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Goetz, Rebecca Anne (2015). "Barack Hussein Obama: America's First Muslim President?". In Sutton, Matthew Avery; Dochuk, Darren (eds.). Faith in the New Millennium: The Future of Religion and American Politics. Oxford University Press. doi:10.1093/acprof:oso/9780199372690.003.0006. ISBN 9780199372737. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ "Mitigating LLM hallucinations in text summarisation". BBC. 20 June 2024. Retrieved 7 March 2025. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Amugongo, Lameck Mbangula; Mascheroni, Pietro; Brooks, Steven; Doering, Stefan; Seidel, Jan (2025-06-11). "Retrieval augmented generation for large language models in healthcare: A systematic review". PLOS Digital Health. 4 (6) e0000877. doi:10.1371/journal.pdig.0000877. PMC 12157099. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ a b Luan, Yi; Eisenstein, Jacob; Toutanova, Kristina; Collins, Michael (26 April 2021). "Sparse, Dense, and Attentional Representations for Text Retrieval". Transactions of the Association for Computational Linguistics. 9: 329–345. arXiv:2005.00181. doi:10.1162/tacl_a_00369. Retrieved 15 March 2025. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ "Information retrieval". Microsoft. 10 January 2025. Retrieved 15 March 2025. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Khattab, Omar; Zaharia, Matei (2020). "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT". Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 39–48. doi:10.1145/3397271.3401075. ISBN 978-1-4503-8016-4. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Wang, Yup; Conroy, John M.; Molino, Neil; Yang, Julia; Green, Mike (2024). "Laboratory for Analytic Sciences in TREC 2024 Retrieval Augmented Generation Track". NIST TREC 2024. Retrieved 15 March 2025. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Lee, Kenton; Chang, Ming-Wei; Toutanova, Kristina (2019). ""Latent Retrieval for Weakly Supervised Open Domain Question Answering"" (PDF). Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Shi, Weijia; Min, Sewon; Yasunaga, Michihiro; Seo, Minjoon; James, Rich; Lewis, Mike; Zettlemoyer, Luke; Yih, Wen-tau (June 2024). "REPLUG: Retrieval-Augmented Black-Box Language Models". Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 8371–8384. arXiv:2301.12652. doi:10.18653/v1/2024.naacl-long.463. Retrieved 16 March 2025. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Ram, Ori; Levine, Yoav; Dalmedigos, Itay; Muhlgay, Dor; Shashua, Amnon; Leyton-Brown, Kevin; Shoham, Yoav (2023). "In-Context Retrieval-Augmented Language Models". Transactions of the Association for Computational Linguistics. 11: 1316–1331. arXiv:2302.00083. doi:10.1162/tacl_a_00605. Retrieved 16 March 2025. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Borgeaud, Sebastian; Mensch, Arthur (2021). "Improving language models by retrieving from trillions of tokens" (PDF). Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Wang, Boxin; Ping, Wei; Xu, Peng; McAfee, Lawrence; Liu, Zihan; Shoeybi, Mohammad; Dong, Yi; Kuchaiev, Oleksii; Li, Bo; Xiao, Chaowei; Anandkumar, Anima; Catanzaro, Bryan (2023). "Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study". Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 7763–7786. doi:10.18653/v1/2023.emnlp-main.482. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Bruch, Sebastian; Gai, Siyu; Ingber, Amir (2023). "An Analysis of Fusion Functions for Hybrid Retrieval". ACM Transactions on Information Systems. 42 (1): 1–35. arXiv:2210.11934. doi:10.1145/3596512. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ "Israel Pays $6 Million on GPT Training to Sway US Youth Opinion on Gaza". Inside Telecom. 2025-09-30. Retrieved 2026-01-11. Wikipedia revision 1377227095, retrieved 2026-11-20
- ^ Cordall, Simon Speakman. "Spinning genocide: How is Israel using US PR firms to frame its Gaza war?". Al Jazeera. Retrieved 2026-01-14. Wikipedia revision 1377227095, retrieved 2026-11-20
- Autoencoder
- Deep learning
- Fine-tuning
- Foundation model
- Generative adversarial network
- Generative pre-trained transformer
- Large language model
- Model Context Protocol
- Neural network
- Prompt engineering
- Reinforcement learning from human feedback
- Retrieval-augmented generation
- Self-supervised learning
- Stochastic parrot
- Synthetic data
- Top-p sampling
- Transformer
- Variational autoencoder
- Vibe coding
- Vision transformer
- Word embedding
- Aleph Alpha
- Anthropic
- Anysphere
- Baichuan
- Canva
- Cognition AI
- Cohere
- Contextual AI
- DeepSeek
- DeepL
- EleutherAI
- ElevenLabs
- HeyGen
- Hugging Face
- Inflection AI
- Invisible Technologies
- Kuaishou
- Lightricks
- Lovable
- Luma Labs
- Meta AI
- Meta Superintelligence Labs
- Microsoft AI
- MiniMax
- Mistral AI
- Moonshot AI
- OpenAI
- Perplexity AI
- Runway
- Safe Superintelligence
- Sakana AI
- Salesforce
- Scale AI
- SoundHound AI
- SpaceXAI
- Stability AI
- StepFun
- Synthesia
- Thinking Machines Lab
- Upstage
- Xiaomi
- Z.ai
- Automated reasoning
- Automated planning
- Constraint satisfaction
- Knowledge representation
- Parameter
- Loss functions
- Regression
- Clustering
- Gradient descent
- Backpropagation
- Attention
- Convolution
- Normalization
- Activation
- Gating
- Weight initialization
- Regularization
- Datasets
- Prompt engineering
- Reinforcement learning
- Diffusion
- Latent diffusion model
- Autoregression
- Adversary
- RAG
- Uncanny valley
- LLM post-training
- RLHF
- Self-supervised learning
- Reflection
- Recursive self-improvement
- Hallucination
- Word embedding
- Vibe coding
- Blended AI
- Open-source AI
- Sovereign AI
- Symbolic AI
- Neuro-symbolic AI
- Situated approach
- Actor-critic algorithm
- Alan Turing
- Warren Sturgis McCulloch
- Walter Pitts
- John von Neumann
- Christopher D. Manning
- Claude Shannon
- Shun'ichi Amari
- Kunihiko Fukushima
- Takeo Kanade
- Marvin Minsky
- John McCarthy
- Nathaniel Rochester
- Allen Newell
- Cliff Shaw
- Herbert A. Simon
- Oliver Selfridge
- Frank Rosenblatt
- Bernard Widrow
- Joseph Weizenbaum
- Seymour Papert
- Seppo Linnainmaa
- Paul Werbos
- Geoffrey Hinton
- John Hopfield
- Jürgen Schmidhuber
- Yann LeCun
- Yoshua Bengio
- Lotfi A. Zadeh
- Stephen Grossberg
- Alex Graves
- James Goodnight
- Andrew Ng
- Fei-Fei Li
- Alex Krizhevsky
- Ilya Sutskever
- Oriol Vinyals
- Quoc V. Le
- Ian Goodfellow
- Demis Hassabis
- David Silver
- Andrej Karpathy
- Ashish Vaswani
- Noam Shazeer
- Aidan Gomez
- John Schulman
- Mustafa Suleyman
- Jan Leike
- Daniel Kokotajlo
- François Chollet
architectures
- Neural Turing machine
- Differentiable neural computer
- Transformer
- Recurrent neural network (RNN)
- Long short-term memory (LSTM)
- Gated recurrent unit (GRU)
- Echo state network
- Multilayer perceptron (MLP)
- Convolutional neural network (CNN)
- Residual neural network (RNN)
- Highway network
- Mamba
- Autoencoder
- Variational autoencoder (VAE)
- Generative adversarial network (GAN)
- Graph neural network (GNN)
and economic
- AI boom
- AI bubble
- AI data center
- AI effect
- AI infrastructure
- AI literacy
- AI slop
- AI winter
- Anthropomorphism
- Arms race
- Competition
- Environmental impact
- Explainable AI
- Generative engine optimization
- In architecture
- In education
- In fiction
- In healthcare
- In marketing
- In video games
- In visual art
- Military applications
- Workplace impact
Text derived from https://en.wikipedia.org/w/index.php?oldid=1377227095 under CC BY-SA 4.0; modifications have been made.
References
1 claim could not be tied to a specific sentence and is listed in the table below instead.