Hard negative mining
≈ 4 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
Hard negative mining, also referred to as hard negative sampling, is a family of techniques in machine learning for selecting or constructing negative samples more difficult to distinguish from a corresponding positive example with respect to uniformly-sampled negatives, leading to a more informative training signal for discriminative classifiers and similarity models. The technique is used when the pool of possible negative examples is far larger than the set of positives and most randomly chosen negatives are easy to classify by the model, contributing little to learning.
Concentrating training on the small fraction of difficult negatives is intended to provide stronger gradient signal and faster convergence than uniform sampling. Hard negative mining originated in object detection in the 1990s, where it was referred to as bootstrapping (not to be confused with statistical bootstrap resampling), and was later formalized for support vector machine models.
Closely related ideas appear in deep metric learning, information retrieval, recommender systems, self-supervised contrastive learning, and vision-language models. Surveys distinguish hard negative mining from the broader practice of negative sampling, in which negatives are drawn at random or in proportion to frequency rather than by difficulty.
Hard negative mining, also referred to as hard negative sampling, is a family of techniques in machine learning for selecting or constructing negative samples more difficult to distinguish from a corresponding positive example with respect to uniformly-sampled negatives, leading to a more informative training signal for discriminative classifiers and similarity models. The technique is used when the pool of possible negative examples is far larger than the set of positives and most randomly chosen negatives are easy to classify by the model, contributing little to learning.
Concentrating training on the small fraction of difficult negatives is intended to provide stronger gradient signal and faster convergence than uniform sampling. Hard negative mining originated in object detection in the 1990s, where it was referred to as bootstrapping (not to be confused with statistical bootstrap resampling), and was later formalized for support vector machine models.
Closely related ideas appear in deep metric learning, information retrieval, recommender systems, self-supervised contrastive learning, and vision-language models. Surveys distinguish hard negative mining from the broader practice of negative sampling, in which negatives are drawn at random or in proportion to frequency rather than by difficulty. \mathcal{L}_N = -\,\mathbb{E}\!\left[\, \log \frac{f_k(x_{t+k}, c_t)}{\sum_{x_j \in X} f_k(x_j, c_t)} \,\right].
In its original form, contrastive predictive coding draws its negatives at random and recommends using more negatives because the objective bounds a mutual information quantity that tightens as N grows. Several influential methods similarly increase the supply of negatives rather than their difficulty: SimCLR treats all other augmented examples in a large training batch as negatives and argues that its temperature-scaled cross-entropy loss implicitly weights negatives by hardness, whereas margin- and triplet-style losses require explicit semi-hard mining to remain competitive; and Momentum Contrast (MoCo) maintains negatives in a queue decoupled from the batch size, encoded by a slowly updated momentum encoder to keep them consistent.
Explicit hard-negative methods have nonetheless been adapted to the unsupervised setting. Importance weighting.
A tunable sampling distribution that up-weights negatives the current embedding maps close to the anchor while approximately discarding likely false negatives, implemented as a reweighting of the contrastive loss; a hardness parameter interpolates between uniform sampling and an adversarial worst-case distribution. Synthetic embeddings.
Rather than selecting difficult examples, MoCHi synthesizes them in the embedding space, forming new negatives as normalized convex combinations of the hardest existing negatives and mixing the query with hard negatives to obtain still-harder examples, after observing that most stored memory negatives contribute little to the loss as training progresses. Adversarial perturbations.
The CLAE method applies adversarial perturbations computed jointly across the batch, which its authors characterize as performing hard negative mining without labels.
Text derived from https://en.wikipedia.org/w/index.php?oldid=1375966014 under CC BY-SA 4.0; modifications have been made.