Exponential tilting

Uses your system voice; voice and rate in settings.

Snapshot

≈ 13 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

Exponential tilting (ET), exponential twisting, or exponential change of measure (ECM) is a distribution-shifting technique used in many parts of mathematics. The different exponential tiltings of a random variable X is known as the natural exponential family of X . Exponential tilting is used in Monte Carlo estimation for rare-event simulation, and rejection and importance sampling in particular. In mathematical finance exponential tilting is also known as Esscher tilting (or the Esscher transform), and often combined with indirect Edgeworth approximation and is used in such contexts as insurance futures pricing. The earliest formalization of exponential tilting is often attributed to Frederik Esscher with its use in importance sampling being attributed to David Siegmund.

Exponential tilting (ET), exponential twisting, or exponential change of measure (ECM) is a distribution-shifting technique used in many parts of mathematics. The different exponential tiltings of a random variable X is known as the natural exponential family of X . Exponential tilting is used in Monte Carlo estimation for rare-event simulation, and rejection and importance sampling in particular. In mathematical finance exponential tilting is also known as Esscher tilting (or the Esscher transform), and often combined with indirect Edgeworth approximation and is used in such contexts as insurance futures pricing. The earliest formalization of exponential tilting is often attributed to Frederik Esscher with its use in importance sampling being attributed to David Siegmund. M_X(\theta)=\operatorname E[e^{\theta X}] is finite for a given real parameter \theta . Let \kappa(\theta)=\log M_X(\theta) be the cumulant-generating function (CGF). The exponentially tilted measure P_\theta , restricted to events determined by X , is defined by P_\theta(X\in B)=\int_B e^{\theta x-\kappa(\theta)}\,P_X(dx), where P_X is the law of X . Equivalently, \frac{dP_\theta}{dP}=e^{\theta X-\kappa(\theta)} on the sigma algebra generated by X . If X has density f , then the tilted density is f_\theta(x)=e^{\theta x-\kappa(\theta)}f(x). Thus f_\theta(x) is proportional to e^{\theta x}f(x) , with the normalizing constant supplied by M_X(\theta) . For a random vector X\in\mathbb R^d and a vector parameter \theta\in\mathbb R^d , the analogous definition is P_\theta(X\in B)=\int_B e^{\theta^T x-\kappa(\theta)}\,P_X(dx), where \kappa(\theta)=\log \operatorname E[e^{\theta^T X}].

Example

The exponentially tilted measure in many cases has the same parametric form as that of X . One-dimensional examples include the normal distribution, the exponential distribution, the binomial distribution and the Poisson distribution. For example, in the case of the normal distribution, N( \mu, \sigma ^2) the tilted density f_\theta(x) is the N( \mu + \theta \sigma ^2, \sigma ^2) density. The table below provides more examples of tilted densities. Not every tilted law remains in the same familiar parametric family. For example, if X has the Lomax (or Pareto type II) density f(x)=\alpha(1+x)^{-\alpha-1},\qquad x>0, then the tilted density is proportional to e^{\theta x}(1+x)^{-\alpha-1} . It is normalizable for \theta , but it is not generally another Lomax distribution. In statistical mechanics, the energy of a system in equilibrium with a heat bath has the Boltzmann distribution: \mathbb{P}(E\in dE) \propto e^{-\beta E} \, dE , where \beta is the inverse temperature. Exponential tilting then corresponds to changing the temperature: \mathbb{P}_\theta (E\in dE) \propto e^{-(\beta - \theta) E} \, dE . Similarly, the energy and particle number of a system in equilibrium with a heat and particle bath has the grand canonical distribution: \mathbb{P}((N, E)\in (dN, dE) ) \propto e^{\beta \mu N -\beta E} \, dN \, dE , where \mu is the chemical potential. Exponential tilting then corresponds to changing both the temperature and the chemical potential.

Advantages

In many cases, the tilted distribution belongs to the same parametric family as the original. This is particularly true when the original density belongs to the exponential family of distributions. This simplifies random variable generation during Monte-Carlo simulations. Exponential tilting may still be useful if this is not the case, though normalization must be possible and additional sampling algorithms may be needed. In addition, there exists a simple relationship between the original and tilted CGF, \kappa_\theta(\eta) = \log(\mathbb{E}_\theta[e^{\eta X}]) = \kappa(\theta + \eta) - \kappa(\theta). We can see this by observing that F_\theta(x) = \int\limits_{\infty}^x\exp\{\theta y - \kappa(\theta)\}f(y)\,dy. Thus, \begin{align} \kappa_{\theta}(\eta) &= \log \int e^{\eta x} \, dF_{\theta}(x) \\ &= \log \int e^{\eta x}e^{\theta x - \kappa(\theta)} \, dF(x) \\ &= \log\mathbb{E}[e^{(\eta+\theta)X-\kappa(\theta)}] \\ &= \log(e^{\kappa(\eta+\theta)-\kappa(\theta)}) \\ &= \kappa(\eta+\theta)-\kappa(\theta). \end{align} Clearly, this relationship allows for easy calculation of the CGF of the tilted distribution and thus the distributions moments. Moreover, it results in a simple form of the likelihood ratio. Specifically, \ell = \frac{d\mathbb{P}}{d\mathbb{P}_\theta} = \frac{f(x)}{f_{\theta}(x)} = e^{- \theta x + \kappa(\theta)}. If \kappa(\eta)=\log \operatorname E[\exp(\eta X)] is the CGF of X , then the CGF of the \theta -tilted X is \kappa_\theta(\eta) = \kappa(\theta + \eta) - \kappa(\theta). This means that the i -th cumulant of the tilted X is \kappa^{(i)}(\theta) . In particular, the expectation of the tilted distribution is \operatorname E_\theta[X]= \left. \frac{d}{d\eta}\kappa_\theta(\eta)\right|_{\eta=0} = \kappa'(\theta) . The variance of the tilted distribution is \operatorname{Var}_\theta[X] = \left. \frac{d^2}{d\eta^2}\kappa_\theta(\eta)\right|_{\eta=0} = \kappa(\theta). Repeated tilting is additive. That is, tilting first by \theta_1 and then \theta_2 is the same as tilting once by \theta_1+\theta_2 . If X is the sum of independent, but not necessarily identical random variables X_1, X_2, \dots , then the \theta -tilted distribution of X is the sum of X_1, X_2, \dots each \theta -tilted individually. If \mu=\mathrm{E}[X] , then \kappa(\theta)-\theta \mu is the Kullback–Leibler divergence D_\operatorname{KL}(P \parallel P_\theta)=\operatorname E \left[\log\tfrac{P}{P_\theta}\right] between the tilted distribution P_\theta and the original distribution P of X . Similarly, since \operatorname E_\theta[X]=\kappa'(\theta) , we have the Kullback–Leibler divergence as D_\text{KL}(P_\theta \parallel P) = \operatorname E_\theta \left[\log\frac{P_\theta}{P} \right] = \theta \kappa'(\theta) - \kappa(\theta).

Applications

Rare-event simulation

The exponential tilting of X , assuming it exists, supplies a family of distributions that can be used as proposal distributions for acceptance-rejection sampling or importance distributions for importance sampling. One common application is sampling from a distribution conditional on a sub-region of the domain, i.e. X\mid X\in A . With an appropriate choice of \theta , sampling from \mathbb{P}_\theta can meaningfully reduce the required amount of sampling or the variance of an estimator.

Saddlepoint approximation

The saddlepoint approximation method is a density approximation methodology often used for the distribution of sums and averages of independent, identically distributed random variables that employs Edgeworth series, but which generally performs better at extreme values. From the definition of the natural exponential family, it follows that f_{\theta}(\bar{x}) = f(\bar{x})\exp\{n(\theta \bar{x} - \kappa(\theta))\} . Applying the Edgeworth expansion for f_{\theta}(\bar{x}) , we have f_{\theta}(\bar{x}) = \psi(z)(\mathrm{Var}[\bar{X}])^{-1/2}\left\{1 + \frac{\rho_3(\theta)h_3(z)}{6} + \frac{\rho_4(\theta)h_4(z)}{24} +\cdots\right\}, where \psi(z) is the standard normal density of \begin{align} & z = \frac{\bar{x} - \kappa_{\bar{x}} ' (\theta)}{\kappa_{\bar{x}} (\theta)}, \\[8pt] & \rho_n(\theta) = \kappa^{(n)}(\theta)\{\kappa (\theta)^{n/2}\}, \end{align} and h_n are the Hermite polynomials. When considering values of \bar{x} progressively farther from the center of the distribution, |z|\rightarrow \infty and the h_n(z) terms become unbounded. However, for each value of \bar{x} , we can choose \theta such that \kappa '(\theta) = \bar{x}. This value of \theta is referred to as the saddle-point, and the above expansion is always evaluated at the expectation of the tilted distribution. This choice of \theta leads to the final representation of the approximation given by f(\bar{x}) \approx \left(\frac{n}{2\pi\kappa(\theta)} \right)^{1/2}\exp\{n(\kappa(\theta) - \theta\bar{x})\}.

Rejection sampling

Using the tilted distribution \mathbb{P}_{\theta} as the proposal, the rejection sampling algorithm prescribes sampling from f_\theta(x) and accepting with probability \frac{1}{c} \exp(-\theta x + \kappa(\theta)), where c = \sup\limits_{x\in X}\frac{d\mathbb{P}}{d\mathbb{P}_{\theta}}(x). That is, a uniformly distributed random variable p \sim \mbox{Unif}(0,1) is generated, and the sample from f_\theta(x) is accepted if p \leq \frac{1}{c} \exp(-\theta x + \kappa(\theta)).

Importance sampling

Applying the exponentially tilted distribution as the importance distribution yields the equation \mathbb{E}(h(X)) = \mathbb{E}_{\theta}[\ell(X)h(X)], where \ell(X) = \frac{d\mathbb{P}}{d\mathbb{P}_{\theta}} is the likelihood function. So, one samples from f_{\theta} to estimate the probability under the importance distribution \mathbb{P}(dX) and then multiplies it by the likelihood ratio. Moreover, we have the variance given by \operatorname{Var}(X) = \operatorname E[(\ell(X)h(X)^2].

Example

Assume independent and identically distributed \{X_i\} such that \kappa(\theta) . In order to estimate \mathbb{P}(X_1 + \cdots + X_n > c) , we can employ importance sampling by taking h(X) = \mathbb{I}\left(\sum_{i = 1}^n X_i > c\right). The constant c can be rewritten as na for some other constant a . Then, \operatorname P\left(\sum_{i = 1}^n X_i > na\right) = \operatorname E_{\theta_a} \left[\exp\left\{-\theta_a\sum_{i = 1}^n X_i + n\kappa(\theta_a)\right\} \mathbb{I} \left( \sum_{i = 1}^n X_i > na \right) \right], where \theta_a denotes the \theta defined by the saddle-point equation \kappa '(\theta_a) = a.

Stochastic processes and Girsanov's theorem

For stochastic processes, the analogue of exponential tilting is a change of measure on path space. In the Brownian motion case, let (W_t)_{0\leq t\leq T} be a standard Brownian motion. For constant \theta , Z_T=\exp\left\{\theta W_T-\frac{1}{2}\theta^2T\right\} is an exponential martingale with expectation one. Defining a new measure Q by \frac{dQ}{dP}=Z_T changes the stochastic drift of Brownian motion: under Q , the process \widetilde W_t=W_t-\theta t is a standard Brownian motion. Thus W_t has drift \theta t under the new measure. More generally, Girsanov's theorem states that if \lambda_t is an adapted process satisfying suitable integrability conditions, then the stochastic exponential Z_T= \exp\left\{ \int_0^T \lambda_t\,dW_t-\frac12\int_0^T \lambda_t^2\,dt \right\} can be used as a Radon–Nikodym derivative. Under the measure Q defined by dQ/dP=Z_T , \widetilde W_t=W_t-\int_0^t\lambda_s\,ds is Brownian. If dX_t=b_t\,dt+\sigma_t\,dW_t under P , then under Q the same process may be written as dX_t=(b_t+\sigma_t\lambda_t)\,dt+\sigma_t\,d\widetilde W_t. This is analogous to exponential tilting of a single random variable, but it acts on the distribution of an entire stochastic path rather than only on a fixed-time marginal.

Choice of tilting parameter

Siegmund's algorithm

Assume i.i.d. Xs with light tailed distribution and \operatorname E[X] > 0 . In order to estimate \psi(c) = \mathbb{P}(\tau(c) where \tau(c) = \inf\left\{t:\sum\limits_{i=1}^t X_i> c\right\} , when c is large and hence \psi(c) small, the algorithm uses exponential tilting to derive the importance distribution. The algorithm is used in many aspects, such as sequential tests, G/G/1 queue waiting times, and \psi is used as the probability of ultimate ruin in ruin theory. In this context, it is logical to ensure that \mathbb{P}_\theta(\tau(c) . The criterion \theta > \theta_0 , where \theta_0 is s.t. \kappa'(\theta_0) = 0 achieves this. Siegmund's algorithm uses \theta = \theta^* , if it exists, where \theta^* is defined in the following way: \kappa(\theta^*) = 0 . It has been shown that \theta^* is the only tilting parameter producing bounded relative error ( \underset{x \rightarrow \infty}{\lim\sup}\frac{\operatorname{Var}\mathbb{I}_{A(x)}}{\mathbb{P}A(x)^2} ).

Black-box algorithms

We can only see the input and output of a black box, without knowing its structure. The algorithm is to use only minimal information on its structure. When we generate random numbers, the output may not be within the same common parametric class, such as normal or exponential distributions. An automated way may be used to perform ECM. Let X_1, X_2,\ldots be i.i.d. r.v.’s with distribution G ; for simplicity we assume X\geq 0 . Define \mathfrak{F}_n = \sigma(X_1,\ldots,X_n,U_1,\ldots, U_n) , where U_1, U_2,\ldots are independent (0, 1) uniforms. A randomized stopping time for X_1, X_2,\ldots is then a stopping time w.r.t. the filtration \{\mathfrak{F}_n\}, \ldots Let further \mathfrak{G} be a class of distributions G on [0, \infty) with k_G = \int_0^\infty e^{\theta x}G(dx) and define G_\theta by \frac{dG_\theta}{dG(x)} = e^{\theta x - k_G} . We define a black-box algorithm for ECM for the given \theta and the given class \mathfrak{G} of distributions as a pair of a randomized stopping time \tau and an \mathfrak{F}_\tau -measurable r.v. Z such that Z is distributed according to G_\theta for any G \in \mathfrak{G} . Formally, we write this as \mathbb{P}_G (Z for all x . In other words, the rules of the game are that the algorithm may use simulated values from G and additional uniforms to produce an r.v. from G_\theta .

References

Text derived from https://en.wikipedia.org/w/index.php?oldid=1362530096 under CC BY-SA 4.0; modifications have been made.

Provenance

snapshot enwiki-001362530096 · captured 1783173408.0 · version history

Cite this page as: encyclopegia.org, “Exponential tilting”, snapshot enwiki-001362530096. https://encyclopegia.org/a/exponential-tilting

Sources cited in this revision

3 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.