Word2Vec

In natural language processing, Word2Vec is a technique for obtaining vector representations of words as word embeddings. These vectors capture information about the meaning of a word based on its surrounding words in a piece of text, following the principles of distributional semantics. Once trained, the model can be used to find words with similar meanings or usage, while its embeddings can serve as inputs to systems for search and classification. Word2Vec was developed by Tomáš Mikolov, Kai Chen, Greg Corrado, Ilya Sutskever and Jeff Dean at Google, published in preprints and presented at ICLR in 2013¹. Its computational efficiency made it practical to learn high-quality word embeddings from very large text corpora and contributed to their widespread adoption in natural language processing.

In natural language processing, Word2Vec is a technique for obtaining vector representations of words as word embeddings. These vectors capture information about the meaning of a word based on its surrounding words in a piece of text, following the principles of distributional semantics. Once trained, the model can be used to find words with similar meanings or usage, while its embeddings can serve as inputs to systems for search and classification. Word2Vec was developed by Tomáš Mikolov, Kai Chen, Greg Corrado, Ilya Sutskever and Jeff Dean at Google, published in preprints and presented at ICLR in 2013. Its computational efficiency made it practical to learn high-quality word embeddings from very large text corpora and contributed to their widespread adoption in natural language processing. ==Approach== Word2Vec is a group of related models that are used to produce word embeddings. These models are shallow, two-layer neural networks that are trained to reconstruct linguistic contexts of words. Word2Vec takes as its input a large corpus of text and produces a mapping of the set of words to a vector space, typically of several hundred dimensions, with each unique word in the corpus being assigned a vector in the space. Word2Vec can use either of two model architectures to produce these distributed representations of words: continuous bag of words (CBOW) or continuously sliding skip-gram. In both architectures, Word2Vec considers both individual words and a sliding context window as it iterates over the corpus. The CBOW can be viewed as a "fill in the blank" task, where the word embedding represents the way the word influences the relative probabilities of other words in the context window. Words which are semantically similar should influence these probabilities in similar ways, because semantically similar words should be used in similar contexts. The order of context words does not influence prediction (bag of words assumption). In the continuous skip-gram architecture, the model uses the current word to predict the surrounding window of context words. The skip-gram architecture weighs nearby context words more heavily than more distant context words. According to the authors' note, CBOW is faster while skip-gram does a better job for infrequent words. After the model is trained, the learned word embeddings are positioned in the vector space such that words that share common contexts in the corpus — that is, words that are semantically and syntactically similar — are located close to one another in the space. More dissimilar words are located farther from one another in the space. For example, if we want each word in the corpus to be predicted by every other word in a small span of 4 words. The set of relative indexes of neighbor words will be: N = \{-2, -1, +1, +2\} , and the objective is to maximize \sum_i\ln \Pr(w_i \mid w_{i-2}, w_{i-1}, w_{i+1}, w_{i+2}) . In standard bag-of-words, a word's context is represented by a word-count (aka a word histogram) of its neighboring words. For example, the "sat" in "the cat sat on the mat" is represented as {"the": 2, "cat": 1, "on": 1}. Note that the last word "mat" is not used to represent "sat", because it is outside the neighborhood N = \{-2, -1, +1, +2\} . In continuous bag-of-words, the histogram is multiplied by a matrix V to obtain a continuous representation of the word's context. The matrix V is also called a dictionary. Its columns are the word vectors. It has D columns, where D is the size of the dictionary. Let d be the length of each word vector. We have V \in \R^{d \times D} . For example, multiplying the word histogram {"the": 2, "cat": 1, "on": 1} with V , we obtain 2 v_{\text{the}} + v_{\text{cat}} + v_{\text{on}} . This is then multiplied with another matrix V' of shape \R^{D \times d} . Each row of it is a word vector v' . This results in a vector of length D , one entry per dictionary entry. Then, apply the softmax to obtain a probability distribution over the dictionary. This system can be visualized as a neural network, similar in spirit to an autoencoder, of architecture linear-linear-softmax, as depicted in the diagram. The system is trained by gradient descent to minimize the cross-entropy loss. In full formula, the cross-entropy loss is: -\sum_i \ln \frac{e^{v_{w_i}' \cdot (\sum_{j \in N} v_{w_{j+i}})}}{\sum_{w'} e^{v_{w'}' \cdot (\sum_{j \in N} v_{{w'}_{j+i}})}} where the outer summation \sum_i is over the words in a corpus, the quantity \sum_{j \in N} v_{w_{j+i}} is the sum of a word's neighbors' vectors, etc. Once such a system is trained, we have two trained matrices V, V' . Either the column vectors of V or the row vectors of V' can serve as the dictionary. For example, the word "sat" can be represented as either the "sat"-th column of V or the "sat"-th row of V' . It is also possible to simply define V' = V^\top , in which case there would no longer be a choice. In full formula, the loss function is -\sum_i \sum_{j \in N} \ln \frac{e^{v_{w_{j+i}}' \cdot v_{w_i}}}{\sum_{w'} e^{v_{w'}' \cdot v_{w_i}}} Same as CBOW, once such a system is trained, we have two trained matrices V, V' . Either the column vectors of V or the row vectors of V' can serve as the dictionary. It is also possible to simply define V' = V^\top , in which case there would no longer be a choice. Essentially, skip-gram and CBOW are exactly the same in architecture. They only differ in the objective function during training. ==History== Research on neural language models emerged during the 1980s. In 2003, Yoshua Bengio and colleagues introduced a neural probabilistic language model that jointly learned distributed word representations and probabilities over word sequences. In 2010, Tomáš Mikolov and colleagues introduced a recurrent neural-network language model for speech recognition that significantly improved performance over conventional n-gram language models. Word2Vec was introduced in two 2013 papers by Tomáš Mikolov and colleagues at Google. The first described the continuous bag-of-words and skip-gram architectures for efficiently learning word representations from large text corpora. A follow-up paper introduced techniques including negative sampling, sub-sampling of frequent words, and representations of multi-word phrases. The follow-up paper received the 2023 NeurIPS Test of Time Award. In 2017, researchers at Facebook AI Research described an extension of the skip-gram model used in fastText, in which words are represented as combinations of character n-grams. This incorporates subword information into word representations and allows vectors to be constructed for words that did not appear in the training data. Static embedding methods such as Word2Vec and fastText produce context-independent word representations. Contextual representations were later developed to account for how word meaning varies with surrounding text, including the recurrent ELMo model and transformer-based models such as BERT. By 2022, the Word2vec approach was described as "dated". ==Extensions== There are a variety of extensions to Word2Vec, including incorporating variable-lengths of text, estimating topic representations, applications to biological sequences, and applications to medicine. ==Analysis== The reasons for successful word embedding learning in the Word2Vec framework are poorly understood. Goldberg and Levy point out that the Word2Vec objective function causes words that occur in similar contexts to have similar embeddings (as measured by cosine similarity) and note that this is in line with J. R. Firth's distributional hypothesis. However, they note that this explanation is "very hand-wavy" and argue that a more formal explanation would be preferable. Levy et al. (2015) show that much of the superior performance of Word2Vec or similar embeddings in downstream tasks is not a result of the models per se, but of the choice of specific hyperparameters. Transferring these hyperparameters to more 'traditional' approaches yields similar performances in downstream tasks. Arora et al. (2016) explain Word2Vec and related algorithms as performing inference for a simple generative model for text, which involves a random walk generation process based upon loglinear topic model. They use this to explain some properties of word embeddings, including their use to solve analogies. ==References== ==External links== Wikipedia2Vec([https://wikipedia2vec.github.io/wikipedia2vec/ introduction) ===Implementations=== C C++ C# Python (Spark) Python (TensorFlow) Python (Gensim) Java/Scala R

Text derived from https://en.wikipedia.org/w/index.php?oldid=1376791706 under CC BY-SA 4.0; modifications have been made.