Foundation model

In artificial intelligence, a foundation model (FM), also known as large x model (LxM, where "x" is a variable representing any text, image, sound, etc.), is a machine learning or deep learning model trained on vast datasets so that it can be applied across a wide range of use cases. Generative AI applications like large language models (LLMs) are common examples of foundation models. Building foundation models is often highly resource-intensive, with the most advanced models costing hundreds of millions of dollars to cover the expenses of acquiring, curating, and processing massive datasets, as well as the computational power required for training. These costs stem from the need for sophisticated infrastructure, extended training times, and advanced hardware, such as graphics processing units. In contrast, adapting an existing foundation model for a specific task or using it directly is far less costly, as it uses pre-trained capabilities and typically requires only fine-tuning on smaller, task-specific datasets. Early examples of foundation models are language models like OpenAI's GPT series. In addition to text, foundation models have been developed for visual images, including DALL-E and Stable Diffusion.

In artificial intelligence, a foundation model (FM), also known as large x model (LxM, where "x" is a variable representing any text, image, sound, etc.), is a machine learning or deep learning model trained on vast datasets so that it can be applied across a wide range of use cases. Generative AI applications like large language models (LLMs) are common examples of foundation models. Building foundation models is often highly resource-intensive, with the most advanced models costing hundreds of millions of dollars to cover the expenses of acquiring, curating, and processing massive datasets, as well as the computational power required for training. These costs stem from the need for sophisticated infrastructure, extended training times, and advanced hardware, such as graphics processing units. In contrast, adapting an existing foundation model for a specific task or using it directly is far less costly, as it uses pre-trained capabilities and typically requires only fine-tuning on smaller, task-specific datasets. Early examples of foundation models are language models like OpenAI's GPT series. In addition to text, foundation models have been developed for visual images, including DALL-E and Stable Diffusion. ===Scaling=== The accuracy and capabilities of foundation models often scale predictably with the size of the model and the amount of the training data. Specifically, scaling laws have been discovered, which are data-based empirical trends that relate resources (data, model size, compute usage) to model capabilities. Particularly, a model's scale is defined by compute, dataset size, and the number of parameters, all of which exhibit a power-law relationship with end performance. However, broken scaling laws have been discovered in which this relationship smoothly transitions (at points referred to as break(s)) from a power law with one exponent to a power law with another (different) exponent. When one does not collect any points near (or after) the break(s), it can be difficult to obtain an accurate extrapolation.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1376650270 under CC BY-SA 4.0; modifications have been made.