{
  "id": "enwiki-001364278350",
  "slug": "maximum-likelihood-estimation",
  "title": "Maximum likelihood estimation",
  "qid": "",
  "lead": "In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference. If the likelihood function is differentiable, the derivative test for finding maxima can be applied. In some cases, the first-order conditions of the likelihood function can be solved analytically; for instance, the ordinary least squares estimator for a linear regression model maximizes the likelihood when the random errors are assumed to have normal distributions with the same variance. From the perspective of Bayesian inference, MLE is generally equivalent to maximum a posteriori (MAP) estimation with a prior distribution that is uniform in the region of interest. In frequentist inference, MLE is a special case of an extremum estimator, with the objective function being the likelihood.",
  "body": "In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference. If the likelihood function is differentiable, the derivative test for finding maxima can be applied. In some cases, the first-order conditions of the likelihood function can be solved analytically; for instance, the ordinary least squares estimator for a linear regression model maximizes the likelihood when the random errors are assumed to have normal distributions with the same variance. From the perspective of Bayesian inference, MLE is generally equivalent to maximum a posteriori (MAP) estimation with a prior distribution that is uniform in the region of interest. In frequentist inference, MLE is a special case of an extremum estimator, with the objective function being the likelihood. We model a set of observations as a random sample from an unknown joint probability distribution which is expressed in terms of a set of parameters. The goal of maximum likelihood estimation is to determine the parameters for which the observed data have the highest joint probability. We write the parameters governing the joint distribution as a vector \\; \\theta = \\left[ \\theta_{1},\\, \\theta_2,\\, \\ldots,\\, \\theta_k \\right]^{\\mathsf{T}} \\; so that this distribution falls within a parametric family \\; \\{ f(\\cdot\\,;\\theta) \\mid \\theta \\in \\Theta \\} \\;, where \\, \\Theta \\, is called the parameter space, a finite-dimensional subset of Euclidean space. Evaluating the joint density at the observed data sample \\; \\mathbf{y} = (y_1, y_2, \\ldots, y_n) \\; gives a real-valued function, \\mathcal{L}_{n}(\\theta) = \\mathcal{L}_{n}(\\theta; \\mathbf{y}) = f_{n}(\\mathbf{y}; \\theta) \\;, which is called the likelihood function. For independent random variables, f_{n}(\\mathbf{y}; \\theta) will be the product of univariate density functions: f_{n}(\\mathbf{y}; \\theta) = \\prod_{k=1}^n \\, f_k^\\mathsf{univar}(y_k; \\theta) ~. The goal of maximum likelihood estimation is to find the values of the model parameters that maximize the likelihood function over the parameter space, that is: \\hat{\\theta} = \\underset{\\theta\\in\\Theta}{\\operatorname{arg\\;max}}\\,\\mathcal{L}_{n}(\\theta\\,;\\mathbf{y}) ~. Intuitively, this selects the parameter values that make the observed data most probable. The specific value ~ \\hat{\\theta} = \\hat{\\theta}_{n}(\\mathbf{y}) \\in \\Theta ~ that maximizes the likelihood function \\, \\mathcal{L}_{n} \\, is called the maximum likelihood estimate. Further, if the function \\; \\hat{\\theta}_{n} : \\mathbb{R}^{n} \\to \\Theta \\; so defined is measurable, then it is called the maximum likelihood estimator. It is generally a function defined over the sample space, i.e. taking a given sample as its argument. A sufficient but not necessary condition for its existence is for the likelihood function to be continuous over a parameter space \\, \\Theta \\, that is compact. For an open \\, \\Theta \\, the likelihood function may increase without ever reaching a supremum value. In practice, it is often convenient to work with the natural logarithm of the likelihood function, called the log-likelihood: \\ell(\\theta\\,;\\mathbf{y}) = \\ln \\mathcal{L}_{n}(\\theta\\,;\\mathbf{y}) ~. Since the logarithm is a monotonic function, the maximum of \\; \\ell(\\theta\\,;\\mathbf{y}) \\; occurs at the same value of \\theta as does the maximum of \\, \\mathcal{L}_{n} ~. If \\ell(\\theta\\,;\\mathbf{y}) is differentiable in \\, \\Theta \\,, necessary conditions for the occurrence of a maximum (or a minimum) are \\frac{\\partial \\ell}{\\partial \\theta_{1}} = 0, \\quad \\frac{\\partial \\ell}{\\partial \\theta_{2}} = 0, \\quad \\ldots, \\quad \\frac{\\partial \\ell}{\\partial \\theta_{k}} = 0 ~, known as the likelihood equations. For some models, these equations can be explicitly solved for \\, \\widehat{\\theta\\,} \\,, but in general no closed-form solution to the maximization problem is known or available, and an MLE can only be found via numerical optimization. Another problem is that in finite samples, there may exist multiple roots for the likelihood equations. Whether the identified root \\, \\widehat{\\theta\\,} \\, of the likelihood equations is indeed a (local) maximum depends on whether the matrix of second-order partial and cross-partial derivatives, the so-called Hessian matrix \\mathbf{H}\\left(\\widehat{\\theta\\,}\\right) = \\begin{bmatrix} \\left. \\frac{\\partial^2 \\ell}{\\partial \\theta_1^2} \\right|_{\\theta=\\widehat{\\theta\\,}} & \\left. \\frac{\\partial^2 \\ell}{\\partial \\theta_1 \\, \\partial \\theta_2} \\right|_{\\theta=\\widehat{\\theta\\,}} & \\dots & \\left. \\frac{\\partial^2 \\ell}{\\partial \\theta_1 \\, \\partial \\theta_k} \\right|_{\\theta=\\widehat{\\theta\\,}} \\\\ \\left. \\frac{\\partial^2 \\ell}{\\partial \\theta_2 \\, \\partial \\theta_1} \\right|_{\\theta=\\widehat{\\theta\\,}} & \\left. \\frac{\\partial^2 \\ell}{\\partial \\theta_2^2} \\right|_{\\theta=\\widehat{\\theta\\,}} & \\dots & \\left. \\frac{\\partial^2 \\ell}{\\partial \\theta_2 \\, \\partial \\theta_k} \\right|_{\\theta=\\widehat{\\theta\\,}} \\\\ \\vdots & \\vdots & \\ddots & \\vdots \\\\ \\left. \\frac{\\partial^2 \\ell}{\\partial \\theta_k \\, \\partial \\theta_1} \\right|_{\\theta=\\widehat{\\theta\\,}} & \\left. \\frac{\\partial^2 \\ell}{\\partial \\theta_k \\, \\partial \\theta_2} \\right|_{\\theta=\\widehat{\\theta\\,}} & \\dots & \\left. \\frac{\\partial^2 \\ell}{\\partial \\theta_k^2} \\right|_{\\theta=\\widehat{\\theta\\,}} \\end{bmatrix} ~, is negative semi-definite at \\widehat{\\theta\\,} , as this indicates local concavity. Conveniently, most common probability distributions – in particular the exponential family – are logarithmically concave. \\Theta = \\left\\{ \\theta : \\theta \\in \\mathbb{R}^{k},\\; h(\\theta) = 0 \\right\\} ~, where \\; h(\\theta) = \\left[ h_{1}(\\theta), h_{2}(\\theta), \\ldots, h_{r}(\\theta) \\right] \\; is a vector-valued function mapping \\, \\mathbb{R}^{k} \\, into \\; \\mathbb{R}^{r} ~. Estimating the true parameter \\theta belonging to \\Theta then, as a practical matter, means to find the maximum of the likelihood function subject to the constraint ~h(\\theta) = 0 ~. Theoretically, the most natural approach to this constrained optimization problem is the method of substitution, that is \"filling out\" the restrictions \\; h_{1}, h_{2}, \\ldots, h_{r} \\; to a set \\; h_{1}, h_{2}, \\ldots, h_{r}, h_{r+1}, \\ldots, h_{k} \\; in such a way that \\; h^{\\ast} = \\left[ h_{1}, h_{2}, \\ldots, h_{k} \\right] \\; is a one-to-one function from \\mathbb{R}^{k} to itself, and reparameterize the likelihood function by setting \\; \\phi_{i} = h_{i}(\\theta_{1}, \\theta_{2}, \\ldots, \\theta_{k}) ~. Because of the equivariance of the maximum likelihood estimator, the properties of the MLE apply to the restricted estimates also. For instance, in a multivariate normal distribution the covariance matrix \\, \\Sigma \\, must be positive-definite; this restriction can be imposed by replacing \\; \\Sigma = \\Gamma^{\\mathsf{T}} \\Gamma \\;, where \\Gamma is a real upper triangular matrix and \\Gamma^{\\mathsf{T}} is its transpose. In practice, restrictions are usually imposed using the method of Lagrange which, given the constraints as defined above, leads to the restricted likelihood equations \\frac{\\partial \\ell}{\\partial \\theta} - \\frac{\\partial h(\\theta)^\\mathsf{T}}{\\partial \\theta} \\lambda = 0 and h(\\theta) = 0 \\;, where ~ \\lambda = \\left[ \\lambda_{1}, \\lambda_{2}, \\ldots, \\lambda_{r}\\right]^\\mathsf{T} ~ is a column-vector of Lagrange multipliers and \\; \\frac{\\partial h(\\theta)^\\mathsf{T}}{\\partial \\theta} \\; is the Jacobian matrix of partial derivatives. Naturally, if the constraints are not binding at the maximum, the Lagrange multipliers should be zero. This in turn allows for a statistical test of the \"validity\" of the constraint, known as the Lagrange multiplier test. ===Method of support=== The method of support is a technique that is used to make inferences from datasets. According to A. W. F. Edwards, the method of support aims to make inferences about unknown parameters in terms of the relative support, or log likelihood, induced by a set of data for a particular parameter value. The technique may be used whether or not prior information is available. The method of maximum likelihood is part of the method of support, but note that the method of support also provides confidence regions that are defined in terms of their support. \\widehat{\\ell\\,}(\\theta\\,;x)= \\sum_{i=1}^n \\ln f(x_i\\mid\\theta), this being the sample analogue of the expected log-likelihood \\ell(\\theta) = \\operatorname{\\mathbb E}[\\, \\ln f(x_i\\mid\\theta) \\,] , where this expectation is taken with respect to the true density. Maximum-likelihood estimators have no optimum properties for finite samples, in the sense that (when evaluated on finite samples) other estimators may have greater concentration around the true parameter-value. However, like other estimation methods, maximum likelihood estimation possesses a number of attractive limiting properties: As the sample size increases to infinity, sequences of maximum likelihood estimators have these properties: Consistency: the sequence of MLEs converges in probability to the value being estimated. Equivariance: If \\hat{\\theta} is the maximum likelihood estimator for \\theta , and if g(\\theta) is a bijective transform of \\theta , then the maximum likelihood estimator for \\alpha = g(\\theta) is \\hat{\\alpha} = g(\\hat{\\theta} ) . The equivariance property can be generalized to non-bijective transforms, although it applies in that case on the maximum of an induced likelihood function which is not the true likelihood in general. Efficiency, i.e. it achieves the Cramér–Rao lower bound when the sample size tends to infinity. This means that no consistent estimator has lower asymptotic mean squared error than the MLE (or other estimators attaining this bound), which also means that MLE has asymptotic normality. Second-order efficiency after correction for bias. \\widehat{\\theta\\,}_\\mathrm{mle}\\ \\xrightarrow{\\text{p}}\\ \\theta_0. Under slightly stronger conditions, the estimator converges almost surely (or strongly): \\widehat{\\theta\\,}_\\mathrm{mle}\\ \\xrightarrow{\\text{a.s.}}\\ \\theta_0. In practical applications, data is never generated by f(\\cdot\\,;\\theta_0) . Rather, f(\\cdot\\,;\\theta_0) is a model, often in idealized form, of the process generated by the data. It is a common aphorism in statistics that all models are wrong. Thus, true consistency does not occur in practical applications. Nevertheless, consistency is often considered to be a desirable property for an estimator to have. To establish consistency, the following conditions are sufficient. {{ordered list |1= Identification of the model: \\theta \\neq \\theta_0 \\quad \\Leftrightarrow \\quad f(\\cdot\\mid\\theta)\\neq f(\\cdot\\mid\\theta_0). In other words, different parameter values θ correspond to different distributions within the model. If this condition did not hold, there would be some value θ 1 such that θ 0 and θ 1 generate an identical distribution of the observable data. Then we would not be able to distinguish between these two parameters even with an infinite amount of data—these parameters would have been observationally equivalent. The identification condition is absolutely necessary for the ML estimator to be consistent. When this condition holds, the limiting likelihood function ℓ(θ ·) has unique global maximum at θ 0 . |2= Compactness: the parameter space Θ of the model is compact. The identification condition establishes that the log-likelihood has a unique global maximum. Compactness implies that the likelihood cannot approach the maximum value arbitrarily close at some other point (as demonstrated for example in the picture on the right). Compactness is only a sufficient condition and not a necessary condition. Compactness can be replaced by some other conditions, such as: |3= Continuity: the function is continuous in for almost all values of : \\operatorname{\\mathbb P} \\Bigl[\\; \\ln f(x\\mid\\theta) \\;\\in\\; C^0(\\Theta) \\;\\Bigr] = 1. The continuity here can be replaced with a slightly weaker condition of upper semi-continuity. |4= Dominance: there exists integrable with respect to the distribution such that \\Bigl|\\ln f(x\\mid\\theta)\\Bigr| By the uniform law of large numbers, the dominance condition together with continuity establish the uniform convergence in probability of the log-likelihood: \\sup_{\\theta\\in\\Theta} \\left|\\widehat{\\ell\\,}(\\theta\\mid x) - \\ell(\\theta)\\,\\right|\\ \\xrightarrow{\\text{p}}\\ 0. }} The dominance condition can be employed in the case of i.i.d. observations. In the non-i.i.d. case, the uniform convergence in probability can be checked by showing that the sequence \\widehat{\\ell\\,}(\\theta\\mid x) is stochastically equicontinuous. If one wants to demonstrate that the ML estimator \\widehat{\\theta\\,} converges to θ 0 almost surely, then a stronger condition of uniform convergence almost surely has to be imposed: \\sup_{\\theta\\in\\Theta} \\left\\|\\;\\widehat{\\ell\\,}(\\theta\\mid x) - \\ell(\\theta)\\;\\right\\| \\ \\xrightarrow{\\text{a.s.}}\\ 0. Additionally, if (as assumed above) the data were generated by f(\\cdot\\,;\\theta_0) , then under certain conditions, it can also be shown that the maximum likelihood estimator converges in distribution to a normal distribution. Specifically, \\sqrt{n} \\left(\\widehat{\\theta\\,}_\\mathrm{mle} - \\theta_0\\right)\\ \\xrightarrow{d}\\ \\mathcal{N}\\left(0,\\, I^{-1}\\right) where is the Fisher information matrix. The maximum likelihood estimator selects the parameter value which gives the observed data the largest possible probability (or probability density, in the continuous case). If the parameter consists of a number of components, then we define their separate maximum likelihood estimators, as the corresponding component of the MLE of the complete parameter. Consistent with this, if \\widehat{\\theta\\,} is the MLE for \\theta , and if g(\\theta) is any transformation of \\theta , then the MLE for \\alpha=g(\\theta) is by definition \\widehat{\\alpha} = g(\\,\\widehat{\\theta\\,}\\,). \\, It maximizes the so-called profile likelihood: \\bar{L}(\\alpha) = \\sup_{\\theta: \\alpha = g(\\theta)} L(\\theta). \\, The MLE is also equivariant with respect to certain transformations of the data. If y=g(x) where g is one to one and does not depend on the parameters to be estimated, then the density functions satisfy f_Y(y) = f_X(g^{-1}(y)) \\, |(g^{-1}(y))^{\\prime}| and hence the likelihood functions for X and Y differ only by a factor that does not depend on the model parameters. For example, the MLE parameters of the log-normal distribution are the same as those of the normal distribution fitted to the logarithm of the data. In fact, in the log-normal case if X\\sim\\mathcal{N}(0, 1) , then Y=g(X)=e^{X} follows a log-normal distribution. The density of Y follows with f_X standard Normal and g^{-1}(y) = \\log(y) , |(g^{-1}(y))^{\\prime}| = \\frac{1}{y} for y > 0 . \\sqrt{n\\,} \\, \\left( \\widehat{\\theta\\,}_\\text{mle} - \\theta_0 \\right)\\ \\ \\xrightarrow{d}\\ \\ \\mathcal{N} \\left( 0,\\ \\mathcal{I}^{-1} \\right) ~, where ~\\mathcal{I}~ is the Fisher information matrix: \\mathcal{I}_{jk} = \\operatorname{\\mathbb E} \\, \\biggl[ \\; -{ \\frac{\\partial^2\\ln f_{\\theta_0}(X_t)}{\\partial\\theta_j\\,\\partial\\theta_k } } \\; \\biggr] ~. In particular, it means that the bias of the maximum likelihood estimator is equal to zero up to the order . b_h \\; \\equiv \\; \\operatorname{\\mathbb E} \\biggl[ \\; \\left( \\widehat\\theta_\\mathrm{mle} - \\theta_0 \\right)_h \\; \\biggr] \\; = \\; \\frac{1}{\\,n\\,} \\, \\sum_{i, j, k = 1}^m \\; \\mathcal{I}^{h i} \\; \\mathcal{I}^{j k} \\left( \\frac{1}{\\,2\\,} \\, K_{i j k} \\; + \\; J_{j,i k} \\right) where \\mathcal{I}^{j k} (with superscripts) denotes the (j,k)-th component of the inverse Fisher information matrix \\mathcal{I}^{-1} , and \\frac{1}{\\,2\\,} \\, K_{i j k} \\; + \\; J_{j,i k} \\; = \\; \\operatorname{\\mathbb E}\\,\\biggl[\\; \\frac12 \\frac{\\partial^3 \\ln f_{\\theta_0}(X_t)}{\\partial\\theta_i\\;\\partial\\theta_j\\;\\partial\\theta_k} + \\frac{\\;\\partial\\ln f_{\\theta_0}(X_t)\\;}{\\partial\\theta_j}\\,\\frac{\\;\\partial^2\\ln f_{\\theta_0}(X_t)\\;}{\\partial\\theta_i \\, \\partial\\theta_k} \\; \\biggr] ~ . Using these formulae it is possible to estimate the second-order bias of the maximum likelihood estimator, and correct for that bias by subtracting it: \\widehat{\\theta\\,}^*_\\text{mle} = \\widehat{\\theta\\,}_\\text{mle} - \\widehat{b\\,} ~ . This estimator is unbiased up to the terms of order , and is called the bias-corrected maximum likelihood estimator. This bias-corrected estimator is (at least within the curved exponential family), meaning that it has minimal mean squared error among all second-order bias-corrected estimators, up to the terms of the order &nbsp;. It is possible to continue this process, that is to derive the third-order bias-correction term, and so on. However, the maximum likelihood estimator is not third-order efficient. \\operatorname{\\mathbb P}(\\theta\\mid x_1,x_2,\\ldots,x_n) = \\frac{f(x_1,x_2,\\ldots,x_n\\mid\\theta)\\operatorname{\\mathbb P}(\\theta)}{\\operatorname{\\mathbb P}(x_1,x_2,\\ldots,x_n)} where \\operatorname{\\mathbb P}(\\theta) is the prior distribution for the parameter and where \\operatorname{\\mathbb P}(x_1,x_2,\\ldots,x_n) is the probability of the data averaged over all parameters. Since the denominator is independent of , the Bayesian estimator is obtained by maximizing f(x_1,x_2,\\ldots,x_n\\mid\\theta)\\operatorname{\\mathbb P}(\\theta) with respect to . If we further assume that the prior \\operatorname{\\mathbb P}(\\theta) is a uniform distribution, the Bayesian estimator is obtained by maximizing the likelihood function f(x_1,x_2,\\ldots,x_n\\mid\\theta) . Thus the Bayesian estimator coincides with the maximum likelihood estimator for a uniform prior distribution \\operatorname{\\mathbb P}(\\theta) . w = \\underset{ w }{\\operatorname{arg\\;max}} \\; \\int_{-\\infty}^\\infty \\operatorname{\\mathbb P}(\\text{ error}\\mid x)\\operatorname{\\mathbb P}(x)\\,\\operatorname{d}x~ where \\operatorname{\\mathbb P}(\\text{ error}\\mid x) = \\operatorname{\\mathbb P}(w_1\\mid x)~ if we decide \\;w_2\\; and \\;\\operatorname{\\mathbb P}(\\text{ error}\\mid x) = \\operatorname{\\mathbb P}(w_2\\mid x)\\; if we decide \\;w_1\\;. By applying Bayes' theorem \\operatorname{\\mathbb P}(w_i \\mid x) = \\frac{\\operatorname{\\mathbb P}(x \\mid w_i) \\operatorname{\\mathbb P}(w_i)}{\\operatorname{\\mathbb P}(x)}, and if we further assume the zero-or-one loss function, which is a same loss for all errors, the Bayes Decision rule can be reformulated as: h_\\text{Bayes} = \\underset{ w }{\\operatorname{arg\\;max}} \\, \\bigl[\\, \\operatorname{\\mathbb P}(x\\mid w)\\,\\operatorname{\\mathbb P}(w) \\,\\bigr]\\;, where h_\\text{Bayes} is the prediction and \\;\\operatorname{\\mathbb P}(w)\\; is the prior probability. Consider a case where n tickets numbered from 1 to n are placed in a box and one is selected at random (see uniform distribution); thus, the sample size is 1. If n is unknown, then the maximum likelihood estimator \\widehat{n} of n is the number m on the drawn ticket. (The likelihood is 0 for n&nbsp; \\widehat{n} , is (n&nbsp;+&nbsp;1)/2. As a result, with a sample size of 1, the maximum likelihood estimator for n will systematically underestimate n by (n&nbsp;−&nbsp;1)/2. Suppose the coin is tossed 80 times: i.e. the sample might be something like x 1 &nbsp;=&nbsp;H, x 2 &nbsp;=&nbsp;T, ..., x 80 &nbsp;=&nbsp;T, and the count of the number of heads \"H\" is observed. The probability of tossing tails is 1&nbsp;−&nbsp;p (so here p is θ above). Suppose the outcome is 49&nbsp;heads and 31&nbsp;tails, and suppose the coin was taken from a box containing three coins: one which gives heads with probability p&nbsp;=&nbsp; , one which gives heads with probability p&nbsp;=&nbsp; and another which gives heads with probability p&nbsp;=&nbsp; . The coins have lost their labels, so which one it was is unknown. Using maximum likelihood estimation, the coin that has the largest likelihood can be found, given the data that were observed. By using the probability mass function of the binomial distribution with sample size equal to 80, number successes equal to 49 but for different values of p (the \"probability of success\"), the likelihood function (defined below) takes one of three values: \\begin{align} \\operatorname{\\mathbb P}\\bigl[\\;\\mathrm{H} = 49 \\mid p=\\tfrac{1}{3}\\;\\bigr] & = \\binom{80}{49}(\\tfrac{1}{3})^{49}(1-\\tfrac{1}{3})^{31} \\approx 0.000, \\\\[6pt] \\operatorname{\\mathbb P}\\bigl[\\;\\mathrm{H} = 49 \\mid p=\\tfrac{1}{2}\\;\\bigr] & = \\binom{80}{49}(\\tfrac{1}{2})^{49}(1-\\tfrac{1}{2})^{31} \\approx 0.012, \\\\[6pt] \\operatorname{\\mathbb P}\\bigl[\\;\\mathrm{H} = 49 \\mid p=\\tfrac{2}{3}\\;\\bigr] & = \\binom{80}{49}(\\tfrac{2}{3})^{49}(1-\\tfrac{2}{3})^{31} \\approx 0.054~. \\end{align} The likelihood is maximized when &nbsp;=&nbsp; , and so this is the maximum likelihood estimate for&nbsp; . L(p) = f_D(\\mathrm{H} = 49 \\mid p) = \\binom{80}{49} p^{49}(1 - p)^{31}~, and the maximisation is over all possible values One way to maximize this function is by differentiating with respect to and setting to zero: \\begin{align} 0 & = \\frac{\\partial}{\\partial p} \\left( \\binom{80}{49} p^{49}(1-p)^{31} \\right)~, \\\\[8pt] 0 & = 49 p^{48}(1-p)^{31} - 31 p^{49}(1-p)^{30} \\\\[8pt] & = p^{48}(1-p)^{30}\\left[ 49 (1-p) - 31 p \\right] \\\\[8pt] & = p^{48}(1-p)^{30}\\left[ 49 - 80 p \\right]~. \\end{align} This is a product of three terms. The first term is 0 when &nbsp;=&nbsp;0. The second is 0 when &nbsp;=&nbsp;1. The third is zero when &nbsp;=&nbsp; . The solution that maximizes the likelihood is clearly &nbsp;=&nbsp; (since &nbsp;=&nbsp;0 and &nbsp;=&nbsp;1 result in a likelihood of 0). Thus the maximum likelihood estimator for is . This result is easily generalized by substituting a letter such as in the place of 49 to represent the observed number of 'successes' of our Bernoulli trials, and a letter such as in the place of 80 to represent the number of Bernoulli trials. Exactly the same calculation yields which is the maximum likelihood estimator for any sequence of Bernoulli trials resulting in 'successes'. f(x\\mid \\mu,\\sigma^2) = \\frac{1}{\\sqrt{2\\pi\\sigma^2}\\ } \\exp\\left(-\\frac {(x-\\mu)^2}{2\\sigma^2} \\right), the corresponding probability density function for a sample of independent identically distributed normal random variables (the likelihood) is f(x_1,\\ldots,x_n \\mid \\mu,\\sigma^2) = \\prod_{i=1}^n f( x_i\\mid \\mu, \\sigma^2) = \\left( \\frac{1}{2\\pi\\sigma^2} \\right)^{n/2} \\exp\\left( -\\frac{ \\sum_{i=1}^n (x_i-\\mu)^2}{2\\sigma^2}\\right). This family of distributions has two parameters: ; so we maximize the likelihood, \\mathcal{L} (\\mu,\\sigma^2) = f(x_1,\\ldots,x_n \\mid \\mu, \\sigma^2) , over both parameters simultaneously, or if possible, individually. Since the logarithm function itself is a continuous strictly increasing function over the range of the likelihood, the values which maximize the likelihood will also maximize its logarithm (the log-likelihood itself is not necessarily strictly increasing). The log-likelihood can be written as follows: \\log\\left( \\mathcal{L} (\\mu,\\sigma^2)\\right) = -\\frac{n}{2} \\log(2\\pi\\sigma^2) - \\frac{1}{2\\sigma^2} \\sum_{i=1}^n \\left(x_i - \\mu\\right)^2 (Note: the log-likelihood is closely related to information entropy and Fisher information.) We now compute the derivatives of this log-likelihood as follows. \\begin{align} 0 & = \\frac{\\partial}{\\partial \\mu} \\log\\left( \\mathcal{L} (\\mu,\\sigma^2)\\right) = 0 - \\frac{-2 n(\\bar{x}-\\mu)}{2\\sigma^2}. \\end{align} where \\bar{x} is the sample mean. This is solved by \\widehat\\mu = \\bar{x} = \\sum^n_{i=1} \\frac{\\,x_i\\,}{n}. This is indeed the maximum of the function, since it is the only turning point in and the second derivative is strictly less than zero. Its expected value is equal to the parameter of the given distribution, \\operatorname{\\mathbb E}\\bigl[\\;\\widehat\\mu\\;\\bigr] = \\mu, \\, which means that the maximum likelihood estimator \\widehat\\mu is unbiased. Similarly we differentiate the log-likelihood with respect to and equate to zero: \\begin{align} 0 & = \\frac{\\partial}{\\partial \\sigma} \\log\\Bigl( \\mathcal{L} (\\mu,\\sigma^2)\\Bigr) = -\\frac{\\,n\\,}{\\sigma} + \\frac{1}{\\sigma^3} \\sum_{i=1}^{n} (\\,x_i-\\mu\\,)^2. \\end{align} which is solved by \\widehat\\sigma^2 = \\frac{1}{n} \\sum_{i=1}^n(x_i-\\mu)^2. Inserting the estimate \\mu = \\widehat\\mu we obtain \\widehat\\sigma^2 = \\frac{1}{n} \\sum_{i=1}^n (x_i - \\bar{x})^2 = \\frac{1}{n}\\sum_{i=1}^n x_i^2 -\\frac{1}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n x_i x_j. To calculate its expected value, it is convenient to rewrite the expression in terms of zero-mean random variables (statistical error) \\delta_i \\equiv \\mu - x_i . Expressing the estimate in these variables yields \\widehat\\sigma^2 = \\frac{1}{n} \\sum_{i=1}^n (\\mu - \\delta_i)^2 -\\frac{1}{n^2}\\sum_{i=1}^n\\sum_{j=1}^n (\\mu - \\delta_i)(\\mu - \\delta_j). Simplifying the expression above, utilizing the facts that \\operatorname{\\mathbb E}\\bigl[\\;\\delta_i\\;\\bigr] = 0 and \\operatorname{E}\\bigl[\\;\\delta_i^2\\;\\bigr] = \\sigma^2 , allows us to obtain \\operatorname{\\mathbb E}\\bigl[\\;\\widehat\\sigma^2\\;\\bigr]= \\frac{\\,n-1\\,}{n}\\sigma^2. This means that the estimator \\widehat\\sigma^2 is biased for \\sigma^2 . It can also be shown that \\widehat\\sigma is biased for \\sigma , but that both \\widehat\\sigma^2 and \\widehat\\sigma are consistent. Formally we say that the maximum likelihood estimator for \\theta=(\\mu,\\sigma^2) is \\widehat{\\theta\\,} = \\left(\\widehat{\\mu},\\widehat{\\sigma}^2\\right). In this case the MLEs could be obtained individually. In general this may not be the case, and the MLEs would have to be obtained simultaneously. The normal log-likelihood at its maximum takes a particularly simple form: \\log\\Bigl( \\mathcal{L}(\\widehat\\mu,\\widehat\\sigma)\\Bigr) = \\frac{\\,-n\\;\\;}{2} \\bigl(\\,\\log(2\\pi\\widehat\\sigma^2) +1\\,\\bigr) This maximum log-likelihood can be shown to be the same for more general least squares, even for non-linear least squares. This is often used in determining likelihood-based approximate confidence intervals and confidence regions, which are generally more accurate than those using the asymptotic normality discussed above. f(y_1,y_2) = f(y_1) f(y_2)\\, Suppose one constructs an order-n Gaussian vector out of random variables (y_1,\\ldots,y_n) , where each variable has means given by (\\mu_1, \\ldots, \\mu_n) . Furthermore, let the covariance matrix be denoted by \\mathit\\Sigma . The joint probability density function of these n random variables then follows a multivariate normal distribution given by: f(y_1,\\ldots,y_n) = \\frac{1}{(2\\pi)^{n/2}\\sqrt{\\det(\\mathit\\Sigma)}} \\exp\\left( -\\frac{1}{2} \\left[y_1-\\mu_1,\\ldots,y_n-\\mu_n\\right]\\mathit\\Sigma^{-1} \\left[y_1-\\mu_1,\\ldots,y_n-\\mu_n\\right]^\\mathrm{T} \\right) In the bivariate case, the joint probability density function is given by: f(y_1,y_2) = \\frac{1}{2\\pi \\sigma_{1} \\sigma_2 \\sqrt{1-\\rho^2}} \\exp\\left[ -\\frac{1}{2(1-\\rho^2)} \\left(\\frac{(y_1-\\mu_1)^2}{\\sigma_1^2} - \\frac{2\\rho(y_1-\\mu_1)(y_2-\\mu_2)}{\\sigma_1\\sigma_2} + \\frac{(y_2-\\mu_2)^2}{\\sigma_2^2}\\right) \\right] In this and other cases where a joint density function exists, the likelihood function is defined as above, in the section \"principles,\" using this density. X_1,\\ X_2,\\ldots,\\ X_m are counts in cells / boxes 1 up to m; each box has a different probability (think of the boxes being bigger or smaller) and we fix the number of balls that fall to be n : x_1+x_2+\\cdots+x_m=n . The probability of each box is p_i , with a constraint: p_1+p_2+\\cdots+p_m=1 . This is a case in which the X_i s are not independent, the joint probability of a vector x_1,\\ x_2,\\ldots,x_m is called the multinomial and has the form: f(x_1,x_2,\\ldots,x_m\\mid p_1,p_2,\\ldots,p_m)=\\frac{n!}{\\prod x_i!}\\prod p_i^{x_i}= \\binom{n}{x_1,x_2,\\ldots,x_m} p_1^{x_1} p_2^{x_2} \\cdots p_m^{x_m} Each box taken separately against all the other boxes is a binomial and this is an extension thereof. The log-likelihood of this is: \\ell(p_1,p_2,\\ldots,p_m)=\\log n!-\\sum_{i=1}^m \\log x_i!+\\sum_{i=1}^m x_i\\log p_i The constraint has to be taken into account and use the Lagrange multipliers: L(p_1,p_2,\\ldots,p_m,\\lambda)=\\ell(p_1,p_2,\\ldots,p_m)+\\lambda\\left(1-\\sum_{i=1}^m p_i\\right) By posing all the derivatives to be 0, the most natural estimate is derived \\hat{p}_i=\\frac{x_i}{n} Maximizing log likelihood, with and without constraints, can be an unsolvable problem in closed form, then we have to use iterative procedures. \\frac{\\partial \\ell(\\theta;\\mathbf{y})}{\\partial \\theta} = 0 cannot be solved explicitly for an estimator \\widehat{\\theta} = \\widehat{\\theta}(\\mathbf{y}) . Instead, they need to be solved iteratively: starting from an initial guess of \\theta (say \\widehat{\\theta}_{1} ), one seeks to obtain a convergent sequence \\left\\{ \\widehat{\\theta}_{r} \\right\\} . Many methods for this kind of optimization problem are available, but the most commonly used ones are algorithms based on an updating formula of the form \\widehat{\\theta}_{r+1} = \\widehat{\\theta}_{r} + \\eta_{r} \\mathbf{d}_r\\left(\\widehat{\\theta}\\right) where the vector \\mathbf{d}_{r}\\left(\\widehat{\\theta}\\right) indicates the descent direction of the r th \"step,\" and the scalar \\eta_{r} captures the \"step length,\" also known as the learning rate. \\eta_r\\in \\R^+ that is small enough for convergence and \\mathbf{d}_r\\left(\\widehat{\\theta}\\right) = \\nabla\\ell\\left(\\widehat{\\theta}_r;\\mathbf{y}\\right) Gradient descent method requires to calculate the gradient at the r-th iteration, but no need to calculate the inverse of second-order derivative, i.e., the Hessian matrix. Therefore, it is computationally faster than Newton–Raphson method. \\eta_r = 1 and \\mathbf{d}_r\\left(\\widehat{\\theta}\\right) = -\\mathbf{H}^{-1}_r\\left(\\widehat{\\theta}\\right) \\mathbf{s}_r\\left(\\widehat{\\theta}\\right) where \\mathbf{s}_{r}(\\widehat{\\theta}) is the score and \\mathbf{H}^{-1}_r \\left(\\widehat{\\theta}\\right) is the inverse of the Hessian matrix of the log-likelihood function, both evaluated the r th iteration. But because the calculation of the Hessian matrix is computationally costly, numerous alternatives have been proposed. The popular Berndt–Hall–Hall–Hausman algorithm approximates the Hessian with the outer product of the expected gradient, such that \\mathbf{d}_r\\left(\\widehat{\\theta}\\right) = - \\left[ \\frac{1}{n} \\sum_{t=1}^n \\frac{\\partial \\ell(\\theta;\\mathbf{y})}{\\partial \\theta} \\left( \\frac{\\partial \\ell(\\theta;\\mathbf{y})}{\\partial \\theta} \\right)^{\\mathsf{T}} \\right]^{-1} \\mathbf{s}_r \\left(\\widehat{\\theta}\\right) \\mathbf{H}_{k+1} = \\left(I - \\gamma_k y_k s_k^\\mathsf{T}\\right) \\mathbf{H}_k \\left(I - \\gamma_k s_k y_k^\\mathsf{T}\\right) + \\gamma_k y_k y_k^\\mathsf{T}, where y_k = \\nabla\\ell(x_k + s_k) - \\nabla\\ell(x_k), \\gamma_k = \\frac{1}{y_k^\\mathsf{T} s_k}, s_k = x_{k+1} - x_k. B_{k+1} = B_k + \\frac{y_k y_k^\\mathsf{T}}{y_k^\\mathsf{T} s_k} - \\frac{B_k s_k s_k^\\mathsf{T} B_k^\\mathsf{T}}{s_k^\\mathsf{T} B_k s_k}\\ , where y_k = \\nabla\\ell(x_k + s_k) - \\nabla\\ell(x_k), s_k = x_{k+1} - x_k. BFGS method is not guaranteed to converge unless the function has a quadratic Taylor expansion near an optimum. However, BFGS can have acceptable performance even for non-smooth optimization instances Another popular method is to replace the Hessian with the Fisher information matrix, \\mathcal{I}(\\theta) = \\operatorname{\\mathbb E}\\left[\\mathbf{H}_r \\left(\\widehat{\\theta}\\right)\\right] , giving us the Fisher scoring algorithm. This procedure is standard in the estimation of many methods, such as generalized linear models. Although popular, quasi-Newton methods may converge to a stationary point that is not necessarily a local or global maximum, but rather a local minimum or a saddle point. Therefore, it is important to assess the validity of the obtained solution to the likelihood equations, by verifying that the Hessian, evaluated at the solution, is both negative definite and well-conditioned. === Related concepts=== Akaike information criterion: a criterion to compare statistical models, based on MLE Extremum estimator: a more general class of estimators to which MLE belongs Fisher information: information matrix, its relationship to covariance matrix of ML estimates Mean squared error: a measure of how 'good' an estimator of a distributional parameter is (be it the maximum likelihood estimator or some other estimator) RANSAC: a method to estimate parameters of a mathematical model given data that contains outliers Rao–Blackwell theorem: yields a process for finding the best possible unbiased estimator (in the sense of having minimal mean squared error); the MLE is often a good starting place for the process Wilks' theorem: provides a means of estimating the size and shape of the region of roughly equally-probable estimates for the population's parameter values, using the information from a single sample, using a chi-squared distribution Tilevik, Andreas (2022). Maximum likelihood vs least squares in linear regression (video)",
  "categories": [
    "107141658",
    "20476758",
    "23603337",
    "26220375",
    "27424683",
    "74037575"
  ],
  "license": "CC BY-SA 4.0",
  "source_uri": "https://en.wikipedia.org/w/index.php?oldid=1364278350",
  "content_hash": "",
  "created_at": null,
  "updated_at": 1784126390.0,
  "claims": [],
  "contradictions": [],
  "confidence": 0.6,
  "csl": {
    "id": "enwiki-001364278350",
    "type": "webpage",
    "title": "Maximum likelihood estimation",
    "author": [
      {
        "family": "Anticloud FZ LLE",
        "literal": "Anticloud FZ LLE"
      }
    ],
    "container-title": "encyclopegia.org",
    "publisher": "Anticloud FZ LLE",
    "publisher-place": "Dubai, United Arab Emirates",
    "URL": "https://encyclopegia.org/a/maximum-likelihood-estimation",
    "issued": {
      "date-parts": [
        [
          2026,
          11,
          22
        ]
      ]
    },
    "accessed": {
      "date-parts": [
        [
          2026,
          11,
          22
        ]
      ]
    },
    "license": "CC BY-SA 4.0",
    "DOI": "eo1:enwiki-001364278350",
    "isBasedOn": "https://en.wikipedia.org/w/index.php?oldid=1364278350",
    "note": "Immutable snapshot enwiki-001364278350"
  }
}