Tf–idf
≈ 8 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
In information retrieval, tf–idf (term frequency–inverse document frequency, TF*IDF, TFIDF, TF–IDF, or Tf–idf) is a measure of importance of a word to a document in a collection or corpus, adjusted for the fact that some words appear more frequently in general. Like the bag-of-words model, it models a document as a multiset of words, without word order. It is a refinement over the simple bag-of-words model, by allowing the weight of words to depend on the rest of the corpus. It was often used as a weighting factor in searches of information retrieval, text mining, and user modeling. A survey conducted in 2015 showed that 83% of text-based recommender systems in digital libraries used tf–idf. Variations of the tf–idf weighting scheme were often used by search engines as a central tool in scoring and ranking a document's relevance given a user query. One of the simplest ranking functions is computed by summing the tf–idf for each query term; many more sophisticated ranking functions are variants of this simple model.
In information retrieval, tf–idf (term frequency–inverse document frequency, TF*IDF, TFIDF, TF–IDF, or Tf–idf) is a measure of importance of a word to a document in a collection or corpus, adjusted for the fact that some words appear more frequently in general. Like the bag-of-words model, it models a document as a multiset of words, without word order. It is a refinement over the simple bag-of-words model, by allowing the weight of words to depend on the rest of the corpus. It was often used as a weighting factor in searches of information retrieval, text mining, and user modeling. A survey conducted in 2015 showed that 83% of text-based recommender systems in digital libraries used tf–idf. Variations of the tf–idf weighting scheme were often used by search engines as a central tool in scoring and ranking a document's relevance given a user query. One of the simplest ranking functions is computed by summing the tf–idf for each query term; many more sophisticated ranking functions are variants of this simple model. \mathrm{tf}(t,d) = \frac{f_{t,d}}{{\sum_{t' \in d}{f_{t',d}}}} , where is the raw count of a term in a document, i.e., the number of times that term occurs in document . Note the denominator is simply the total number of terms in document (counting each occurrence of the same term separately). There are various other ways to define term frequency: the raw count itself: Boolean "frequencies": if occurs in and 0 otherwise; logarithmically scaled frequency: ; augmented frequency, to prevent a bias towards longer documents, e.g. raw frequency divided by the raw frequency of the most frequently occurring term in the document: \mathrm{tf}(t,d) = 0.5 + 0.5 \cdot \frac{f_{t, d}}{\max\{f_{t', d}:t' \in d\}} \mathrm{idf}(t, D) = \log \frac{N}{n_t} with D : is the set of all documents in the corpus N = : total number of documents in the corpus n_t = |\{d \in D: t \in d\}| : number of documents where the term t appears (i.e., \mathrm{tf}(t,d) \neq 0 ). If the term is not in the corpus, this will lead to a division-by-zero. It is therefore common to adjust the numerator to 1 + N and the denominator to 1 + n_t . \mathrm{tfidf}(t,d,D) = \mathrm{tf}(t,d) \cdot \mathrm{idf}(t, D) A high weight in tf–idf is reached by a high term frequency (in the given document) and a low document frequency of the term in the whole collection of documents; the weights hence tend to filter out common terms. Since the ratio inside the idf's log function is always greater than or equal to 1, the value of idf (and tf–idf) is greater than or equal to 0. As a term appears in more documents, the ratio inside the logarithm approaches 1, bringing the idf and tf–idf closer to 0. P(t|D) = \frac {N}, so that we can define idf as \begin{align} \mathrm{idf} & = -\log P(t|D) \\ & = \log \frac{1}{P(t|D)} \\ & = \log \frac{N} \end{align} Namely, the inverse document frequency is the logarithm of "inverse" relative document frequency. This probabilistic interpretation in turn takes the same form as that of self-information. However, applying such information-theoretic notions to problems in information retrieval leads to problems when trying to define the appropriate event spaces for the required probability distributions: not only documents need to be taken into account, but also queries and terms. p(d|t) = \frac{1} This assumption and its implications, according to Aizawa: "represent the heuristic that tf–idf employs." The conditional entropy of a "randomly chosen" document in the corpus D , conditional to the fact it contains a specific term t (and assuming that all documents have equal probability to be chosen) is: H({\cal D}|{\cal T}=t)=-\sum_d p_{d|t}\log p_{d|t}=-\log \frac{1} =\log \frac + \log |D|=-\mathrm{idf}(t)+\log |D| In terms of notation, {\cal D} and {\cal T} are "random variables" corresponding to respectively draw a document or a term. The mutual information can be expressed as M({\cal T};{\cal D}) = H({\cal D}) - H({\cal D}|{\cal T}) = \sum_t p_t\cdot(H({\cal D}) - H({\cal D}|W=t))=\sum_t p_t \cdot \mathrm{idf}(t) The last step is to expand p_t , the unconditional probability to draw a term, with respect to the (random) choice of a document, to obtain: M({\cal T};{\cal D})=\sum_{t,d} p_{t|d}\cdot p_d \cdot \mathrm{idf}(t) = \sum_{t,d} \mathrm{tf}(t,d)\cdot \frac{1} \cdot \mathrm{idf}(t) = \frac{1} \sum_{t,d} \mathrm{tf}(t,d)\cdot \mathrm{idf}(t). This expression shows that summing the Tf–idf of all possible terms and documents recovers the mutual information between documents and term taking into account all the specificities of their joint distribution. Each Tf–idf hence carries the "bit of information" attached to a term x document pair. The calculation of tf–idf for the term "this" is performed as follows: In its raw frequency form, tf is just the frequency of the "this" for each document. In each document, the word "this" appears once; but as the document 2 has more words, its relative frequency is smaller. \mathrm{tf}(\mathsf{this}, d_{1}) = \frac{1}{5} = 0.2 \mathrm{tf}(\mathsf{this}, d_{2}) = \frac{1}{7} \approx 0.14 An idf is constant per corpus, and accounts for the ratio of documents that include the word "this". In this case, we have a corpus of two documents and all of them include the word "this". \mathrm{idf}(\mathsf{this}, D) = \log \left (\frac{2}{2} \right ) = 0 So tf–idf is zero for the word "this", which implies that the word is not very informative as it appears in all documents. \mathrm{tfidf}(\mathsf{this}, d_{1}, D) = 0.2 \times 0 = 0 \mathrm{tfidf}(\mathsf{this}, d_{2}, D) = 0.14 \times 0 = 0 The word "example" is more interesting - it occurs three times, but only in the second document: \mathrm{tf}(\mathsf{example}, d_{1}) = \frac{0}{5} = 0 \mathrm{tf}(\mathsf{example}, d_{2}) = \frac{3}{7} \approx 0.429 \mathrm{idf}(\mathsf{example}, D) = \log \left (\frac{2}{1} \right ) = 0.301 Finally, \mathrm{tfidf}(\mathsf{example}, d_1, D) = \mathrm{tf}(\mathsf{example}, d_1) \times \mathrm{idf}(\mathsf{example}, D) = 0 \times 0.301 = 0 \mathrm{tfidf}(\mathsf{example}, d_2, D) = \mathrm{tf}(\mathsf{example}, d_2) \times \mathrm{idf}(\mathsf{example}, D) = 0.429 \times 0.301 \approx 0.129 (using the base 10 logarithm).
Text derived from https://en.wikipedia.org/w/index.php?oldid=1369827406 under CC BY-SA 4.0; modifications have been made.