Wasserstein GAN
The Wasserstein Generative Adversarial Network (WGAN) is a variant of generative adversarial network (GAN) proposed in 2017 that aims to "improve the stability of learning, get rid of problems like mode collapse, and provide meaningful…
≈ 7 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
The Wasserstein Generative Adversarial Network (WGAN) is a variant of generative adversarial network (GAN) proposed in 2017 that aims to "improve the stability of learning, get rid of problems like mode collapse, and provide meaningful learning curves useful for debugging and hyperparameter searches". Compared with the original GAN discriminator, the Wasserstein GAN discriminator provides a better learning signal to the generator.
This allows the training to be more stable when generator is learning distributions in very high dimensional spaces. The objective of the game is L(\mu_G, D) := \mathbb{E}_{x\sim \mu_{ref}}[\ln D(x)] + \mathbb{E}_{x\sim \mu_G}[\ln (1-D(x))]. The generator aims to minimize it, and the discriminator aims to maximize it.
A basic theorem of the GAN game states that{{Math theorem | math_statement = For any fixed generator strategy \mu_G , let the optimal reply be D^* = \arg\max_{D} L(\mu_G, D) , then \begin{align} D^*(x) &= \frac{d\mu_{ref}}{d(\mu_{ref} + \mu_G)}\\ L(\mu_G, D^*) &= 2D_{JS}(\mu_{ref}; \mu_G) - 2\ln 2, \end{align} where the derivative is the Radon–Nikodym derivative, and D_{JS} is the Jensen–Shannon divergence. | name = Theorem | note = the optimal discriminator computes the Jensen–Shannon divergence }} Repeat the GAN game many times, each time with the generator moving first, and the discriminator moving second. Each time the generator \mu_G changes, the discriminator must adapt by approaching the ideal D^*(x) = \frac{d\mu_{ref}}{d(\mu_{ref} + \mu_G)}.
Since we are really interested in \mu_{ref} , the discriminator function D is by itself rather uninteresting. It merely keeps track of the likelihood ratio between the generator distribution and the reference distribution. At equilibrium, the discriminator is just outputting \frac 12 constantly, having given up trying to perceive any difference.
Concretely, in the GAN game, let us fix a generator \mu_G , and improve the discriminator step-by-step, with \mu_{D, t} being the discriminator at step t . Then we (ideally) have L(\mu_G, \mu_{D, 1}) \leq L(\mu_G, \mu_{D, 2}) \leq \cdots \leq \max_{\mu_D} L(\mu_G, \mu_D) = 2D_{JS}(\mu_{ref} \| \mu_G) - 2\ln 2, so we see that the discriminator is actually lower-bounding D_{JS}(\mu_{ref} \| \mu_G) . The Wasserstein GAN game is a zero-sum game, with objective function L_{WGAN}(\mu_G, D) := \mathbb{E}_{x\sim \mu_G}[D(x)] -\mathbb E_{x\sim \mu_{ref}}[D(x)]. The generator goes first, and the discriminator goes second. The generator aims to minimize the objective, and the discriminator aims to maximize the objective: \min_{\mu_G} \max_{D} L_{WGAN}(\mu_G, D). }} By the Kantorovich-Rubenstein duality, for any generator strategy \mu_G , the optimal reply by the discriminator is D^* , such that L_{WGAN}(\mu_G, D^*) = K \cdot W_1(\mu_G, \mu_{ref}).
Consequently, if the discriminator is good, the generator would be constantly pushed to minimize W_1(\mu_G, \mu_{ref}) , and the optimal strategy for the generator is just \mu_G = \mu_{ref} , as it should. Let \mu_G be parametrized by \theta , then we can perform stochastic gradient descent by using two unbiased estimators of the gradient: \nabla_{\theta} \mathbb E_{x\sim \mu_G} [\ln(1-D(x))] = \mathbb E_{x\sim \mu_G} [\ln(1-D(x))\cdot \nabla_{\theta} \ln\rho_{\mu_G}(x)] \nabla_{\theta} \mathbb E_{x\sim \mu_G} [D_{WGAN}(x)] = \mathbb E_{x\sim \mu_G} [D_{WGAN}(x)\cdot \nabla_{\theta} \ln\rho_{\mu_G}(x)] where we used the reparameterization trick.{{NoteTag|note=This is not how it is really done in practice, since \nabla _{\theta }\ln \rho _{\mu _{G}}(x) is in general intractable, but it is theoretically illuminating.|name=not really done in practice}} As shown, the generator in GAN is motivated to let its \mu_G "slide down the peak" of \ln(1-D(x)) . Similarly for the generator in Wasserstein GAN.
For Wasserstein GAN, D_{WGAN} has gradient 1 almost everywhere, while for GAN, \ln(1-D) has flat gradient in the middle, and steep gradient elsewhere. As a result, the variance for the estimator in GAN is usually much larger than that in Wasserstein GAN. See also Figure 3 of.
The problem with D_{JS} is much more severe in actual machine learning situations. Consider training a GAN to generate ImageNet, a collection of photos of size 256-by-256. The space of all such photos is \R^{256^2} , and the distribution of ImageNet pictures, \mu_{ref} , concentrates on a manifold of much lower dimension in it.
Consequently, any generator strategy \mu_G would almost surely be entirely disjoint from \mu_{ref} , making D_{JS}(\mu_G \| \mu_{ref}) = +\infty . Thus, a good discriminator can almost perfectly distinguish \mu_{ref} from \mu_G , as well as any \mu_G' close to \mu_G . Thus, the gradient \nabla_{\mu_G} L(\mu_G, D) \approx 0 , creating no learning signal for the generator.
Detailed theorems can be found in. Let the discriminator function D to be implemented by a multilayer perceptron: D = D_n \circ D_{n-1} \circ \cdots \circ D_1 where D_i(x) = h(W_i x) , and h:\R \to \R is a fixed activation function with \sup_x |h'(x)| \leq 1 . For example, the hyperbolic tangent function h = \tanh satisfies the requirement.
Then, for any x , let x_i = (D_i \circ D_{i-1} \circ \cdots \circ D_1)(x) , we have by the chain rule: d D(x) = diag(h'(W_n x_{n-1})) \cdot W_n \cdot diag(h'(W_{n-1} x_{n-2})) \cdot W_{n-1} \cdots diag(h'(W_1 x)) \cdot W_1 \cdot dx Thus, the Lipschitz norm of D is upper-bounded by \|D \|_L \leq \sup_{x}\| diag(h'(W_n x_{n-1})) \cdot W_n \cdot diag(h'(W_{n-1} x_{n-2})) \cdot W_{n-1} \cdots diag(h'(W_1 x)) \cdot W_1\|_F where \|\cdot\|_s is the operator norm of the matrix, that is, the largest singular value of the matrix, that is, the spectral radius of the matrix (these concepts are the same for matrices, but different for general linear operators). Since \sup_x |h'(x)| \leq 1 , we have \|diag(h'(W_i x_{i-1}))\|_s = \max_j |h'(W_i x_{i-1, j})| \leq 1 , and consequently the upper bound: \|D \|_L \leq \prod_{i=1}^n \|W_i \|_s Thus, if we can upper-bound operator norms \|W_i\|_s of each matrix, we can upper-bound the Lipschitz norm of D . Since for any m\times l matrix W , let c = \max_{i, j} |W_{i, j}| , we have \|W\|_s^2 = \sup_{\|x\|_2=1}\|W x\|_2^2 = \sup_{\|x\|_2=1}\sum_{i}\left(\sum_j W_{i, j} x_j\right)^2 = \sup_{\|x\|_2=1}\sum_{i, j, k}W_{ij}W_{ik}x_jx_k \leq c^2 ml^2 by clipping all entries of W to within some interval [-c, c] , we can bound \|W\|_s . This is the weight clipping method, proposed by the original paper.
Instead of strictly bounding \|D\|_L , we can simply add a "gradient penalty" term for the discriminator, of form \mathbb{E}_{x\sim\hat\mu}[(\|\nabla D(x)\|_2 - a)^2] where \hat \mu is a fixed distribution used to estimate how much the discriminator has violated the Lipschitz norm requirement. The discriminator, in attempting to minimize the new loss function, would naturally bring \nabla D(x) close to a everywhere, thus making \|D\|_L \approx a . This is the gradient penalty method.
Text derived from https://en.wikipedia.org/w/index.php?oldid=1337581376 under CC BY-SA 4.0; modifications have been made.