{
  "id": "enwiki-001374448284",
  "slug": "multi-task-learning",
  "title": "Multi-task learning",
  "qid": "",
  "lead": "Multi-task learning (MTL) is a subfield of machine learning in which multiple learning tasks are solved at the same time, while exploiting commonalities and differences across tasks. This can result in improved learning efficiency and prediction accuracy for the task-specific models, when compared to training the models separately. Inherently, multi-task learning is a multi-objective optimization problem having trade-offs between different tasks. Early versions of MTL were called \"hints\". In a widely cited 1997 paper, Rich Caruana gave the following characterization: Multitask Learning is an approach to inductive transfer that improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias. It does this by learning tasks in parallel while using a shared representation; what is learned for each task can help other tasks be learned better. In the classification context, MTL aims to improve the performance of multiple classification tasks by learning them jointly. One example is a spam-filter, which can be treated as distinct but related classification tasks across different users.…",
  "body": "Multi-task learning (MTL) is a subfield of machine learning in which multiple learning tasks are solved at the same time, while exploiting commonalities and differences across tasks. This can result in improved learning efficiency and prediction accuracy for the task-specific models, when compared to training the models separately. Inherently, multi-task learning is a multi-objective optimization problem having trade-offs between different tasks. Early versions of MTL were called \"hints\". In a widely cited 1997 paper, Rich Caruana gave the following characterization: Multitask Learning is an approach to inductive transfer that improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias. It does this by learning tasks in parallel while using a shared representation; what is learned for each task can help other tasks be learned better. In the classification context, MTL aims to improve the performance of multiple classification tasks by learning them jointly. One example is a spam-filter, which can be treated as distinct but related classification tasks across different users. To make this more concrete, consider that different people have different distributions of features which distinguish spam emails from legitimate ones, for example an English speaker may find that all emails in Russian are spam, not so for Russian speakers. Yet there is a definite commonality in this classification task across users, for example one common feature might be text related to money transfer. Solving each user's spam classification problem jointly via MTL can let the solutions inform each other and improve performance. Further examples of settings for MTL include multiclass classification and multi-label classification. Multi-task learning works because regularization induced by requiring an algorithm to perform well on a related task can be superior to regularization that prevents overfitting by penalizing all complexity uniformly. One situation where MTL may be particularly helpful is if the tasks share significant commonalities and are generally slightly under sampled. However, as discussed below, MTL has also been shown to be beneficial for learning unrelated tasks. ==Methods== The key challenge in multi-task learning, is how to combine learning signals from multiple tasks into a single model. This may strongly depend on how well different task agree with each other, or contradict each other. There are several ways to address this challenge: ===Task grouping and overlap=== Within the MTL paradigm, information can be shared across some or all of the tasks. Depending on the structure of task relatedness, one may want to share information selectively across the tasks. For example, tasks may be grouped or exist in a hierarchy, or be related according to some general metric. Suppose, as developed more formally below, that the parameter vector modeling each task is a linear combination of some underlying basis. Similarity in terms of this basis can indicate the relatedness of the tasks. For example, with sparsity, overlap of nonzero coefficients across tasks indicates commonality. A task grouping then corresponds to those tasks lying in a subspace generated by some subset of basis elements, where tasks in different groups may be disjoint or overlap arbitrarily in terms of their bases. Task relatedness can be imposed a priori or learned from the data. Hierarchical task relatedness can also be exploited implicitly without assuming a priori knowledge or learning relations explicitly. For example, the explicit learning of sample relevance across tasks can be done to guarantee the effectiveness of joint learning across multiple domains. ===Exploiting unrelated tasks: Auxiliary learning=== In auxiliary learning, one attempts learning a group of principal tasks using a group of auxiliary tasks, unrelated to the principal ones. With the right unrelated tasks, joint learning of unrelated tasks which use the same input data have been shown to be beneficial, and provide significant improvement over standard MTL. The reason is that prior knowledge about task relatedness can lead to sparser and more informative representations for each task grouping, essentially by screening out idiosyncrasies of the data distribution. It has been proposed to build on a prior multitask methodology by favoring a shared low-dimensional representation within each task grouping, and imposing a penalty on tasks from different groups which encourages the two representations to be orthogonal. Learning with auxiliary unrelated tasks poses two major challenges: Finding useful auxiliary tasks and combining losses of all tasks in a useful way. Some methods can learn these from data together with the training process, and combine tasks efficiently. Suppose the training data set is \\mathcal{S}_t =\\{(x_i^t,y_i^t)\\}_{i=1}^{n_t} , with x_i^t\\in\\mathcal{X} , y_i^t\\in\\mathcal{Y} , where indexes task, and t \\in 1,...,T . Let n=\\sum_{t=1}^Tn_t . In this setting there is a consistent input and output space and the same loss function \\mathcal{L}:\\mathbb{R}\\times\\mathbb{R}\\rightarrow \\mathbb{R}_+ for each task: . This results in the regularized machine learning problem: {{NumBlk|:| \\min_{f \\in \\mathcal{H}}\\sum _{t=1} ^T \\frac{1}{n_t} \\sum _{i=1} ^{n_t} \\mathcal{L}(y_i^t, f_t(x_i^t))+\\lambda ||f||_\\mathcal{H} ^2 | }} where \\mathcal{H} is a vector valued reproducing kernel Hilbert space with functions f:\\mathcal X \\rightarrow \\mathcal{Y}^T having components f_t:\\mathcal{X}\\rightarrow \\mathcal {Y} . The reproducing kernel for the space \\mathcal{H} of functions f:\\mathcal X \\rightarrow \\mathbb{R}^T is a symmetric matrix-valued function \\Gamma :\\mathcal X\\times \\mathcal X \\rightarrow \\mathbb{R}^{T \\times T} , such that \\Gamma (\\cdot ,x)c\\in \\mathcal{H} and the following reproducing property holds: {{NumBlk|:| \\langle f(x),c \\rangle _ {\\mathbb{R}^T} = \\langle f,\\Gamma (x,\\cdot ) c \\rangle _ {\\mathcal {H}} | }} The reproducing kernel gives rise to a representer theorem showing that any solution to equation has the form: The form of the kernel induces both the representation of the feature space and structures the output across tasks. A natural simplification is to choose a separable kernel, which factors into separate kernels on the input space and on the tasks \\{1,...,T\\} . In this case the kernel relating scalar components f_t and f_s is given by \\gamma((x_i,t),(x_j,s )) = k(x_i,x_j)k_T(s,t)=k(x_i,x_j)A_{s,t} . For vector valued functions f\\in \\mathcal H we can write \\Gamma(x_i,x_j)=k(x_i,x_j)A , where is a scalar reproducing kernel, and is a symmetric positive semi-definite T\\times T matrix. Henceforth denote S_+^T=\\{\\text{PSD matrices} \\} \\subset \\mathbb R^{T \\times T} . This factorization property, separability, implies the input feature space representation does not vary by task. That is, there is no interaction between the input kernel and the task kernel. The structure on tasks is represented solely by . Methods for non-separable kernels is a current field of research. For the separable case, the representation theorem is reduced to f(x)=\\sum _{i=1} ^N k(x,x_i)Ac_i . The model output on the training data is then , where is the n \\times n empirical kernel matrix with entries K_{i,j}=k(x_i,x_j) , and is the n \\times T matrix of rows c_i . With the separable kernel, equation can be rewritten as {{NumBlk|:| \\min _{C\\in \\mathbb{R}^{n\\times T}} V(Y,KCA) + \\lambda tr(KCAC^{\\top}) | }} where is a (weighted) average of applied entry-wise to and . (The weight is zero if Y_i^t is a missing observation). Note the second term in can be derived as follows: \\begin{align} \\|f\\|^2_\\mathcal{H} &= \\left\\langle \\sum _{i=1} ^n k(\\cdot,x_i)Ac_i, \\sum _{j=1} ^n k(\\cdot ,x_j)Ac_j \\right\\rangle_{\\mathcal H } \\\\ &= \\sum _{i,j=1} ^n \\langle k(\\cdot,x_i)A c_i, k(\\cdot ,x_j)Ac_j\\rangle_{\\mathcal H } & \\text{(bilinearity)} \\\\ &= \\sum _{i,j=1} ^n \\langle k(x_i,x_j)A c_i, c_j\\rangle_{\\mathbb R^T } & \\text{(reproducing property)} \\\\ &= \\sum _{i,j=1} ^n k(x_i,x_j) c_i^\\top A c_j=tr(KCAC^\\top ) \\end{align} Letting A^\\dagger = \\gamma I_T + ( \\gamma - \\lambda)\\frac {1} T \\mathbf{1}\\mathbf{1}^\\top (where I_T is the TxT identity matrix, and \\mathbf{1}\\mathbf{1}^\\top is the TxT matrix of ones) is equivalent to letting control the variance \\sum_t || f_t - \\bar f|| _{\\mathcal H_k} of tasks from their mean \\frac 1 T \\sum_t f_t . For example, blood levels of some biomarker may be taken on patients at n_t time points during the course of a day and interest may lie in regularizing the variance of the predictions across patients. Letting A^\\dagger = \\alpha I_T +(\\alpha - \\lambda )M , where M_{t,s} = \\frac 1 \\mathbb I(t,s\\in G_r) is equivalent to letting \\alpha control the variance measured with respect to a group mean: \\sum _{r} \\sum _{t \\in G_r } ||f_t - \\frac 1 \\sum _{s\\in G_r)} f_s|| . (Here |G_r| the cardinality of group r, and \\mathbb I is the indicator function). For example, people in different political parties (groups) might be regularized together with respect to predicting the favorability rating of a politician. Note that this penalty reduces to the first when all tasks are in the same group. Letting A^\\dagger = \\delta I_T + (\\delta -\\lambda)L , where L=D-M is the Laplacian for the graph with adjacency matrix M giving pairwise similarities of tasks. This is equivalent to giving a larger penalty to the distance separating tasks t and s when they are more similar (according to the weight M_{t,s} ,) i.e. \\delta regularizes \\sum _{t,s}||f_t - f_s ||_{\\mathcal H _k }^2 M_{t,s} . All of the above choices of A also induce the additional regularization term \\lambda \\sum_t ||f|| _{\\mathcal H_k} ^2 which penalizes complexity in f more broadly. {{NumBlk|:| \\min _{C \\in \\mathbb{R}^{n\\times T},A \\in S_+^T} V(Y,KCA) + \\lambda tr(KCAC^{\\top})+F(A) | }} Choice of F:S_+^T\\rightarrow \\mathbb R_+ must be designed to learn matrices A of a given type. See \"Special cases\" below. Specifically on the convex set \\mathcal C=\\{(C,A)\\in \\mathbb R^{n \\times T}\\times S_+^T | Range(C^\\top KC)\\subseteq Range(A)\\} , the equivalent problem is convex with the same minimum value. And if (C_R, A_R) is a minimizer for then (C_R A^\\dagger _R, A_R) is a minimizer for . may be solved by a barrier method on a closed set by introducing the following perturbation: {{NumBlk|:| \\min _{C \\in \\mathbb{R}^{n\\times T},A \\in S_+^T} V(Y,KC) + \\lambda tr(A^\\dagger (C^{\\top}KC+\\delta^2I_T))+F(A) | }} The perturbation via the barrier \\delta ^2 tr(A^\\dagger) forces the objective functions to be equal to +\\infty on the boundary of R^{n \\times T}\\times S_+^T . can be solved with a block coordinate descent method, alternating in C and A. This results in a sequence of minimizers (C_m,A_m) in that converges to the solution in as \\delta_m \\rightarrow 0 , and hence gives the solution to . Clustered tasks learning - Jacob et al suggested to learn A in the setting where T tasks are organized in R disjoint clusters. In this case let E\\in \\{0,1\\}^{T\\times R} be the matrix with E_{t,r}=\\mathbb I (\\text{task }t\\in \\text{group }r) . Setting M = I - E^\\dagger E^T , and U = \\frac 1 T \\mathbf{11}^\\top , the task matrix A^\\dagger can be parameterized as a function of M : A^\\dagger(M) = \\epsilon _M U+\\epsilon_B (M-U)+\\epsilon (I-M) , with terms that penalize the average, between clusters variance and within clusters variance respectively of the task predictions. M is not convex, but there is a convex relaxation \\mathcal S_c = \\{M\\in S_+^T:I-M\\in S_+^T \\land tr(M) = r \\} . In this formulation, F(A)=\\mathbb I(A(M)\\in \\{A:M\\in \\mathcal S_C\\}) . ==Software package== A Matlab package called Multi-Task Learning via StructurAl Regularization (MALSAR) implements the following multi-task learning algorithms: Mean-Regularized Multi-Task Learning, Multi-Task Learning with Joint Feature Selection, Robust Multi-Task Feature Learning, Trace-Norm Regularized Multi-Task Learning, Alternating Structural Optimization, Incoherent Low-Rank and Sparse Learning, Robust Low-Rank Multi-Task Learning, Clustered Multi-Task Learning, Multi-Task Learning with Graph Structures. ==See also== Artificial intelligence Artificial neural network Automated machine learning (AutoML) Evolutionary computation Foundation model General game playing Human-based genetic algorithm Kernel methods for vector output Multiple-criteria decision analysis Multi-objective optimization Multicriteria classification Robot learning Transfer learning James–Stein estimator ==References== ==External links== The Biosignals Intelligence Group at UIUC Washington University in St. Louis Department of Computer Science ===Software=== The Multi-Task Learning via Structural Regularization Package Online Multi-Task Learning Toolkit (OMT) A general-purpose online multi-task learning toolkit based on conditional random field models and stochastic gradient descent training (C#, .NET)",
  "categories": [
    "21493616",
    "22906685",
    "22921552",
    "22921972",
    "23118434",
    "23603337",
    "24745122"
  ],
  "license": "CC BY-SA 4.0",
  "source_uri": "https://en.wikipedia.org/w/index.php?oldid=1374448284",
  "content_hash": "",
  "created_at": null,
  "updated_at": 1789174904.0,
  "claims": [],
  "contradictions": [],
  "confidence": 0.6,
  "csl": {
    "id": "enwiki-001374448284",
    "type": "webpage",
    "title": "Multi-task learning",
    "author": [
      {
        "family": "Anticloud FZ LLE",
        "literal": "Anticloud FZ LLE"
      }
    ],
    "container-title": "encyclopegia.org",
    "publisher": "Anticloud FZ LLE",
    "publisher-place": "Dubai, United Arab Emirates",
    "URL": "https://encyclopegia.org/a/multi-task-learning",
    "issued": {
      "date-parts": [
        [
          2026,
          11,
          20
        ]
      ]
    },
    "accessed": {
      "date-parts": [
        [
          2026,
          11,
          20
        ]
      ]
    },
    "license": "CC BY-SA 4.0",
    "DOI": "eo1:enwiki-001374448284",
    "isBasedOn": "https://en.wikipedia.org/w/index.php?oldid=1374448284",
    "note": "Immutable snapshot enwiki-001374448284"
  }
}