Knowledge distillation

In machine learning, knowledge distillation or model distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models ) have more knowledge capacity than small models, this capacity might not be fully utilized. It can be just as computationally expensive to evaluate a model even if it utilizes little of its knowledge capacity. Knowledge distillation transfers knowledge from a large model to a smaller one without loss of validity. As smaller models are less expensive to evaluate, they can be deployed on less powerful hardware (such as a mobile device). There is also a less common technique called Reverse Knowledge Distillation, where knowledge is transferred from a smaller model to a larger one. Model distillation is not to be confused with model compression, which describes methods to decrease the size of a large model itself, without training a new model. Model compression generally preserves the architecture and the nominal parameter count of the model, while decreasing the bits-per-parameter. Knowledge distillation has been successfully used in several applications of machine learning such as object detection, acoustic models, and natural language processing. Recently, it has also been introduced to graph neural networks applicable to non-grid data.

In machine learning, knowledge distillation or model distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models ) have more knowledge capacity than small models, this capacity might not be fully utilized. It can be just as computationally expensive to evaluate a model even if it utilizes little of its knowledge capacity. Knowledge distillation transfers knowledge from a large model to a smaller one without loss of validity. As smaller models are less expensive to evaluate, they can be deployed on less powerful hardware (such as a mobile device). There is also a less common technique called Reverse Knowledge Distillation, where knowledge is transferred from a smaller model to a larger one. Model distillation is not to be confused with model compression, which describes methods to decrease the size of a large model itself, without training a new model. Model compression generally preserves the architecture and the nominal parameter count of the model, while decreasing the bits-per-parameter. Knowledge distillation has been successfully used in several applications of machine learning such as object detection, acoustic models, and natural language processing. Recently, it has also been introduced to graph neural networks applicable to non-grid data. y_i(\mathbf{x}|t) = \frac{e^{\frac{z_i(\mathbf{x})}{t}}}{\sum_j e^{\frac{z_j(\mathbf{x})}{t}}} where t is the temperature, a parameter which is set to 1 for a standard softmax. The softmax operator converts the logit values z_i(\mathbf{x}) to pseudo-probabilities: higher temperature values generate softer distributions of pseudo-probabilities among the output classes. Knowledge distillation consists of training a smaller network, called the distilled model, on a data set called the transfer set which could correspond to the original training set or consist of new, possibly unlabeled data. A cross-entropy loss function is typically used, computed between the output of the distilled model \mathbf{y}(\mathbf{x}|t) and the output of the large model \hat{\mathbf{y}}(\mathbf{x}|t) on the same record (or the average of the individual outputs, if the large model is an ensemble), using a high value of softmax temperature t for both models: E(\mathbf{x}|t) = -\sum_i \hat{y}_i(\mathbf{x}|t) \log y_i(\mathbf{x}|t) . In this context, a high temperature increases the entropy of the output, therefore providing more information to learn for the distilled model compared to hard targets, and at the same time reducing the variance of the gradient between different records, thus allowing a higher learning rate. If ground truth is available for the transfer set, the process can be strengthened by adding to the loss the cross-entropy between the output y_i(\mathbf{x}|1) of the distilled model computed with t = 1 , and the known label \bar{y}_i E(\mathbf{x}|t) = -t^2 \sum_i \hat{y}_i(\mathbf{x}|t) \log y_i(\mathbf{x}|t) - \sum_i \bar{y}_i \log y_i(\mathbf{x}|1) where the component of the loss with respect to the large model is weighted by a factor of t^2 since, as the temperature increases, the gradient of the loss with respect to the model weights scales by a factor of \frac{1}{t^2} . &= -\frac{\partial}{\partial z_i} \sum_j \hat{y}_j \log y_j \\ &= -\frac{\partial}{\partial z_i} \hat{y}_i \log y_i + \left( -\frac{\partial}{\partial z_i} \sum_{k\neq i} \hat{y}_k \log y_k \right)\\ &= -\hat{y}_i \frac{1}{y_i} \frac{\partial}{\partial z_i} y_i + \sum_{k\neq i} \left( -\hat{y}_k \cdot \frac{1}{y_k} \cdot e^{\frac{z_k}{t}} \cdot \left( -\frac{1}{\left(\sum_j e^{\frac{z_j}{t}} \right)^2 }\right) \cdot e^{\frac{z_i}{t}} \cdot \frac{1}{t} \right)\\ &= -\hat{y}_i \frac{1}{y_i} \frac{\partial}{\partial z_i} \frac{e^{\frac{z_i}{t}}}{\sum_j e^{\frac{z_j}{t}}} + \sum_{k\neq i} \left( \hat{y}_k \cdot \frac{1}{y_k} \cdot y_k \cdot y_i \cdot \frac{1}{t} \right)\\ &= -\hat{y}_i \frac{1}{y_i} \left( \frac{\frac{1}{t} e^{\frac{z_i}{t}} \sum_j e^{\frac{z_j}{t}} - \frac{1}{t} \left( e^{\frac{z_i}{t}} \right)^2} {\left( \sum_j e^{\frac{z_j}{t}} \right)^2} \right) + \frac{y_i\sum_{k\neq i}\hat{y}_k}{t}\\ &= -\hat{y}_i \frac{1}{y_i} \left( \frac{y_i}{t} - \frac{y_i^2}{t} \right) + \frac{y_i(1-\hat{y}_i)}{t}\\ &= \frac{1}{t} \left( y_i - \hat{y}_i \right) \\ &= \frac{1}{t} \left( \frac{e^{\frac{z_i}{t}}}{\sum_j e^{\frac{z_j}{t}}} - \frac{e^{\frac{\hat{z}_i}{t}}}{\sum_j e^{\frac{\hat{z}_j}{t}}} \right) \\ \end{align} where \hat{z}_i are the logits of the large model. For large values of t this can be approximated as \frac{1}{t} \left( \frac{1 + \frac{z_i}{t}}{N + \sum_j \frac{z_j}{t}} - \frac{1 + \frac{\hat{z}_i}{t}}{N + \sum_j \frac{\hat{z}_j}{t}} \right) and under the zero-mean hypothesis \sum_j z_j = \sum_j \hat{z}_j = 0 it becomes \frac{z_i - \hat{z}_i}{NT^2} , which is the derivative of \frac{1}{2} \left( z_i - \hat{z}_i \right)^2 , i.e. the loss is equivalent to matching the logits of the two models, as done in model compression. The idea for optimal brain damage is to approximate the loss function in a neighborhood of optimal parameter \theta^* by Taylor expansion: L(\theta) \approx L(\theta^*) + \frac 12 \sum_i (\partial_{\theta_i}^2L(\theta^*)) (\theta_i - \theta_i^*)^2 where \nabla L(\theta^*) \approx 0 , since \theta^* is optimal, and the cross-derivatives \partial_{\theta_i}\partial_{\theta_j}L are neglected to save compute. Thus, the saliency of a parameter approximates the increase in loss if that parameter is deleted.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1377190431 under CC BY-SA 4.0; modifications have been made.