Kaplan–Meier estimator
The Kaplan–Meier estimator, also known as the product limit estimator, is a non-parametric statistic used to estimate the survival function from lifetime data.
≈ 14 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
The Kaplan–Meier estimator, also known as the product limit estimator, is a non-parametric statistic used to estimate the survival function from lifetime data. In medical research, it is often used to measure the fraction of patients living for a certain amount of time after treatment.
In other fields, Kaplan–Meier estimators may be used to measure the length of time people remain unemployed after a job loss, the time-to-failure of machine parts, or how long fleshy fruits remain on plants before they are removed by frugivores. The estimator is named after Edward L. Kaplan and Paul Meier, who each submitted similar manuscripts to the Journal of the American Statistical Association.
The journal editor, John Tukey, convinced them to combine their work into one paper, which has been cited more than 34,000 times since its publication in 1958. The estimator of the survival function S(t) (the probability that life is longer than t ) is given by: : \widehat S(t) = \prod\limits_{i:\ t_i\le t} \left(1 - \frac{d_i}{n_i}\right), with t_i a time when at least one event happened, d i the number of events (e.g., deaths) that happened at time t_i , and n_i the individuals known to have survived (have not yet had an event or been censored) up to time t_i .
The Kaplan–Meier estimator, also known as the product limit estimator, is a non-parametric statistic used to estimate the survival function from lifetime data. In medical research, it is often used to measure the fraction of patients living for a certain amount of time after treatment.
In other fields, Kaplan–Meier estimators may be used to measure the length of time people remain unemployed after a job loss, the time-to-failure of machine parts, or how long fleshy fruits remain on plants before they are removed by frugivores. The estimator is named after Edward L. Kaplan and Paul Meier, who each submitted similar manuscripts to the Journal of the American Statistical Association.
The journal editor, John Tukey, convinced them to combine their work into one paper, which has been cited more than 34,000 times since its publication in 1958. The estimator of the survival function S(t) (the probability that life is longer than t ) is given by: \widehat S(t) = \prod\limits_{i:\ t_i\le t} \left(1 - \frac{d_i}{n_i}\right), with t_i a time when at least one event happened, d i the number of events (e.g., deaths) that happened at time t_i , and n_i the individuals known to have survived (have not yet had an event or been censored) up to time t_i .
Basic concepts
A plot of the Kaplan–Meier estimator is a series of declining horizontal steps which, with a large enough sample size, approaches the true survival function for that population. The value of the survival function between successive distinct sampled observations ("clicks") is assumed to be constant.
An important advantage of the Kaplan–Meier curve is that the method can take into account some types of censored data, particularly right-censoring, which occurs if a patient withdraws from a study, is lost to follow-up, or is alive without event occurrence at last follow-up. On the plot, small vertical tick-marks state individual patients whose survival times have been right-censored.
When no truncation or censoring occurs, the Kaplan–Meier curve is the complement of the empirical distribution function. In medical statistics, a typical application might involve grouping patients into categories, for instance, those with Gene A profile and those with Gene B profile.
In the graph, patients with Gene B die much quicker than those with Gene A. After two years, about 80% of the Gene A patients survive, but less than half of patients with Gene B. To generate a Kaplan–Meier estimator, at least two pieces of data are required for each patient (or each subject): the status at last observation (event occurrence or right-censored), and the time to event (or time to censoring). If the survival functions between two or more groups are to be compared, then a third piece of data is required: the group assignment of each subject.
S(t) = \operatorname{Prob}(\tau > t ) , where t=0,1,\dots is the time. Let \tau_1,\dots,\tau_n\ge 0 be independent, identically distributed random variables, whose common distribution is that of \tau : \tau_j is the random time when some event j happened.
The data available for estimating S is not (\tau_j)_{j=1,\dots,n} , but the list of pairs (\, ( \tilde \tau_j, c_j )\, )_{j=1,\dots,n} where for j\in [n] := \{1,2,\dots,n\} , c_j\ge 0 is a fixed, deterministic integer, the censoring time of event j and \tilde \tau_j = \min(\tau_j,c_j) . In particular, the information available about the timing of event j is whether the event happened before the fixed time c_j and if so, then the actual time of the event is also available. The challenge is to estimate S(t) given this data.
Fix k\in [n]:=\{1,\dots,n\} and let t>0 . A basic argument shows that the following proposition holds: Proposition 1: If the censoring time c_k of event k exceeds t ( c_k\ge t ), then \tilde \tau_k\ge t if and only if \tau_k\ge t . Let k be such that c_k\ge t . It follows from the above proposition that \operatorname{Prob}(\tau_k\ge t) = \operatorname{Prob}(\tilde \tau_k\ge t). Let X_k = \mathbb{I}(\tilde \tau_k\ge t) and consider only those k\in C(t) := \{ k \, :\, c_k \ge t\} , i.e. the events for which the outcome was not censored before time t . Let m(t)=|C(t)| be the number of elements in C(t) . Note that the set C(t) is not random and so neither is m(t) . Furthermore, (X_k)_{k\in C(t)} is a sequence of independent, identically distributed Bernoulli random variables with common parameter S(t)=\operatorname{Prob}(\tau\ge t) . Assuming that m(t)>0 , this suggests to estimate S(t) using \hat S_\text{naive}(t) = \frac{1}{m(t)} \sum_{k:c_k\ge t} X_k = \frac S(t) & = \operatorname{Prob}(\tau > t\mid\tau > t-1)\operatorname{Prob}(\tau > t-1) \\[4pt] & = (1-\operatorname{Prob}(\tau\le t\mid\tau > t-1)) \operatorname{Prob}(\tau > t-1)\\[4pt] & = (1-\operatorname{Prob}(\tau=t\mid\tau \ge t)) \operatorname{Prob}(\tau > t-1) \\[4pt] & = q(t) S(t-1)\,, \end{align} where the second to last equality used that \tau is integer valued and for the last line we introduced q(t) = 1-\operatorname{Prob}(\tau=t\mid\tau\ge t). By a recursive expansion of the equality S(t) = q(t) S(t-1) , we get S(t) = q(t) q(t-1) \cdots q(0).
Note that here q(0) = 1-\operatorname{Prob}(\tau=0\mid\tau > -1) = 1-\operatorname{Prob}(\tau=0) . The Kaplan–Meier estimator can be seen as a "plug-in estimator" where each q(s) is estimated based on the data and the estimator of S(t) is obtained as a product of these estimates. It remains to specify how q(s)=1-\operatorname{Prob}(\tau=s\mid\tau\ge s) is to be estimated.
By a similar reasoning that lead to the construction of the naive estimator above, for any k\in [n] such that c_k > s , \operatorname{Prob}(\tau=s) = \operatorname{Prob}(\tilde \tau_k=s) and \operatorname{Prob}(\tau\ge s) = \operatorname{Prob}(\tilde \tau_k\ge s) both hold. Hence, for any k\in [n] such that c_k > s , \operatorname{Prob}(\tau=s|\tau\ge s) = \operatorname{Prob}(\tilde \tau_k=s)/\operatorname{Prob}(\tilde \tau_k\ge s). Hence we arrive at the estimator \hat q(s) = 1 - \frac (think of estimating the numerator and denominator separately in the definition of the "hazard rate" \operatorname{Prob}(\tau=s|\tau\ge s) ). The Kaplan–Meier estimator is then given by \hat S(t) = \prod_{s=0}^t \hat q(s). The form of the estimator stated at the beginning of the article can be obtained by some further algebra.
For this, write \hat q(s)=1-d(s)/n(s) where, using the actuarial science terminology, d(s)=|\{1\le k\le n\,:\,c_k > s, \tilde \tau_k=s\}| is the number of known deaths at time s , while n(s)=|\{1\le k \le n\,:\, c_k > s, \tilde \tau_k\ge s\}| is the number of those persons who are alive (and not being censored) at time s . Note that if d(s)=0 , \hat q(s)=1 . This implies that we can leave out from the product defining \hat S(t) all those terms where d(s)=0 .
Then, letting 0\le t_1 be the times s when d(s)>0 , d_i = d(t_i) and n_i = n(t_i) , we arrive at the form of the Kaplan–Meier estimator given at the beginning of the article: \hat S(t) = \prod_{i:t_i\le t} \left(1-\frac{d_i}{n_i}\right). As opposed to the naive estimator, this estimator can be seen to use the available information more effectively: In the special case mentioned beforehand, when there are many early events recorded, the estimator will multiply many terms with a value below one and will thus take into account that the survival probability cannot be large.
S(t) = \prod\limits_{i:\ t_i\le t} (1-h_i) and the likelihood function for the hazard function up to time t_i is: \mathcal{L}(h_{j: j\le i}\mid d_{j: j\le i},n_{j: j\le i}) = \prod_{j=1}^i h_j^{d_j}(1-h_j)^{n_j-d_j} {n_j \choose d_j} therefore the log likelihood will be: \log(\mathcal{L}) = \sum_{j=1}^i \left(d_j\log(h_j)+(n_j-d_j)\log(1-h_j) + \log {n_j \choose d_j} \right) finding the maximum of log likelihood with respect to h_i yields: \frac{\partial \log(\mathcal{L})}{\partial h_i} = \frac{d_i}{\widehat{h}_i}-\frac{n_i-d_i}{1-\widehat{h}_i} = 0 \Rightarrow \widehat{h}_i=\frac{d_i}{n_i} where hat is used to denote maximum likelihood estimation. Given this result, we can write: \widehat S(t) = \prod\limits_{i:\ t_i\le t} \left(1 - \widehat{h}_i\right) = \prod\limits_{i:\ t_i\le t} \left(1 - \frac{d_i}{n_i}\right) More generally (for continuous as well as discrete survival distributions), the Kaplan-Meier estimator may be interpreted as a nonparametric maximum likelihood estimator.
Benefits and limitations
The Kaplan–Meier estimator is one of the most frequently used methods of survival analysis. The estimate may be useful to examine recovery rates, the probability of death, and the effectiveness of treatment.
It is limited in its ability to estimate survival adjusted for covariates; parametric survival models and the Cox proportional hazards model may be useful to estimate covariate-adjusted survival. The Kaplan-Meier estimator is directly related to the Nelson-Aalen estimator and both maximize the empirical likelihood.
Statistical considerations
The Kaplan–Meier estimator is a statistic, and several estimators are used to approximate its variance. One of the most common estimators is Greenwood's formula: \widehat{\operatorname{Var}} \left( \widehat S(t) \right) = \widehat S(t)^2 \sum_{i:\ t_i\le t} \frac{d_i}{n_i(n_i-d_i)}, where d_i is the number of cases and n_i is the total number of observations, for t_i . {{Hidden|titlestyle=background-color:#f2dfce; color:black;|contentstyle=border:1px #C4C3D0 solid;|header=For a 'sketch' of the mathematical derivation of the equation above, click on "show" to reveal|content= Greenwood's formula is derived by noting that probability of getting d_i failures out of n_i cases follows a binomial distribution with failure probability h_i . As a result for maximum likelihood hazard rate \widehat{h}_i = d_i/n_i we have E\left(\widehat{h}_i\right) = h_i and \operatorname{Var}\left(\widehat{h}_i\right) = h_i(1-h_i)/n_i . To avoid dealing with multiplicative probabilities we compute variance of logarithm of \widehat S(t) and will use the delta method to convert it back to the original variance: \begin{align} \operatorname{Var}\left(\log \widehat{S}(t)\right) &\sim \frac{1}{{\widehat{S}(t)}^2} \operatorname{Var} \left(\widehat{S}(t)\right) \Rightarrow \\ \operatorname{Var}\left( \widehat{S}(t)\right) &\sim {{\widehat{S}(t)}^2}\operatorname{Var}\left(\log\widehat{S}(t) \right) \end{align} using martingale central limit theorem, it can be shown that the variance of the sum in the following equation is equal to the sum of variances: \log\widehat S(t) = \sum\limits_{i:\ t_i\le t} \log\left(1 - \widehat{h}_i\right) as a result we can write: \begin{align} \operatorname{Var}( \widehat{S}(t)) &\sim {{\widehat{S}(t)}^2}\operatorname{Var}\left(\sum_{i:\ t_i\le t} \log\left(1 - \widehat{h}_i\right)\right) \\ &\sim {{\widehat{S}(t)}^2}\sum\limits_{i:\ t_i\le t} \operatorname{Var}\left(\log\left(1 - \widehat{h}_i\right)\right) \end{align} using the delta method once more: \begin{align} \operatorname{Var}( \widehat{S}(t)) &\sim {{\widehat{S}(t)}^2}\sum_{i:\ t_i\le t}\left(\frac{\partial \log\left(1 - \widehat{h}_i\right)}{\partial \widehat{h}_i}\right)^2 \operatorname{Var}\left(\widehat{h}_i\right)\\ &={{\widehat{S}(t)}^2}\sum_{i:\ t_i\le t}\left(\frac{1}{1-\widehat{h}_i}\right)^2\frac{\widehat{h}_i \left( 1-\widehat{h}_i \right)}{n_i} \\ &= {{\widehat{S}(t)}^2}\sum_{i:\ t_i\le t} \frac{\widehat{h}_i}{n_i\left(1-\widehat{h}_i\right)} \\ &= {{\widehat{S}(t)}^2}\sum_{i:\ t_i\le t} \frac{d_i}{n_i(n_i-d_i)} \end{align} as desired. ---- }} In some cases, one may wish to compare different Kaplan–Meier curves.
This can be done by the log rank test, and the Cox proportional hazards test. Other statistics that may be of use with this estimator are pointwise confidence intervals, the Hall-Wellner band and the equal-precision band.
Software
Mathematica: the built-in function SurvivalModelFit creates survival models. SAS: The Kaplan–Meier estimator is implemented in the proc lifetest procedure. R: the Kaplan–Meier estimator is available as part of the survival package.
Stata: the command sts returns the Kaplan–Meier estimator. Python: the lifelines and scikit-survival packages each include the Kaplan–Meier estimator. MATLAB: the ecdf function with the 'function','survivor' arguments can calculate or plot the Kaplan–Meier estimator.
StatsDirect: The Kaplan–Meier estimator is implemented in the Survival Analysis menu. SPSS: The Kaplan–Meier estimator is implemented in the Analyze > Survival > Kaplan-Meier... menu. Julia: the Survival.jl package includes the Kaplan–Meier estimator.
Epi Info: Kaplan–Meier estimator survival curves and results for the log rank test are obtained with the KMSURVIVAL command.
See also
Survival analysis Frequency of exceedance Median lethal dose Nelson–Aalen estimator
Text derived from https://en.wikipedia.org/w/index.php?oldid=1298262528 under CC BY-SA 4.0; modifications have been made.