Generative adversarial network

Uses your system voice; voice and rate in settings.

Snapshot

≈ 28 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

A generative adversarial network (GAN) is a class of machine learning frameworks and a prominent framework for approaching generative AI. The concept was initially developed by Ian Goodfellow and his colleagues at Université de Montréal in June 2014. In a GAN, two neural networks compete with each other in the form of a zero-sum game, where one agent's gain is another agent's loss. Given a training set, this technique learns to generate new data with the same statistics as the training set.

For example, a GAN trained on photographs can generate new photographs that look at least superficially authentic to human observers, having many realistic characteristics. Though originally proposed as a form of generative model for unsupervised learning, GANs have also proved useful for semi-supervised learning, fully supervised learning, and reinforcement learning.

The core idea of a GAN is based on the "indirect" training through the discriminator, another neural network that can tell how "realistic" the input seems, which itself is also being updated dynamically. This means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator. This enables the model to learn in an unsupervised manner. GANs are similar to mimicry in evolutionary biology, with an evolutionary arms race between both networks.

A generative adversarial network (GAN) is a class of machine learning frameworks and a prominent framework for approaching generative AI. The concept was initially developed by Ian Goodfellow and his colleagues at Université de Montréal in June 2014. In a GAN, two neural networks compete with each other in the form of a zero-sum game, where one agent's gain is another agent's loss. Given a training set, this technique learns to generate new data with the same statistics as the training set.

For example, a GAN trained on photographs can generate new photographs that look at least superficially authentic to human observers, having many realistic characteristics. Though originally proposed as a form of generative model for unsupervised learning, GANs have also proved useful for semi-supervised learning, fully supervised learning, and reinforcement learning.

The core idea of a GAN is based on the "indirect" training through the discriminator, another neural network that can tell how "realistic" the input seems, which itself is also being updated dynamically. This means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator. This enables the model to learn in an unsupervised manner. GANs are similar to mimicry in evolutionary biology, with an evolutionary arms race between both networks.

Definition

The GAN game is a zero-sum game, with objective function L(\mu_G, \mu_D) := \operatorname E_{x\sim \mu_{\text{ref}}, y\sim \mu_D(x)}[\ln y] + \operatorname E_{x\sim \mu_G, y\sim \mu_D(x)}[\ln (1-y)]. The generator aims to minimize the objective, and the discriminator aims to maximize the objective. The generator's task is to approach \mu_G \approx \mu_{\text{ref}} , that is, to match its own output distribution as closely as possible to the reference distribution.

The discriminator's task is to output a value close to 1 when the input appears to be from the reference distribution, and to output a value close to 0 when the input looks like it came from the generator distribution. As for the generator, while \mu_G could theoretically be any computable probability distribution, in practice, it is usually implemented as a pushforward: \mu_G = \mu_Z \circ G^{-1} .

That is, start with a random variable z \sim \mu_Z , where \mu_Z is a probability distribution that is easy to compute (such as the uniform distribution, or the Gaussian distribution), then define a function G: \Omega_Z \to \Omega . Then the distribution \mu_G is the distribution of G(z) . Consequently, the generator's strategy is usually defined as just G , leaving z \sim \mu_Z implicit. In this formalism, the GAN game objective is L(G, D) := \operatorname E_{x\sim \mu_{\text{ref}}}[\ln D(x)] + \operatorname E_{z\sim \mu_Z}[\ln (1-D(G(z)))].

In the original paper, as well as most subsequent papers, it is usually assumed that the generator moves first, and the discriminator moves second, thus giving the following minimax game: \min_{\mu_G}\max_{\mu_D} L(\mu_G, \mu_D) := \operatorname E_{x\sim \mu_{\text{ref}}, y\sim \mu_D(x)}[\ln y] + \operatorname E_{x\sim \mu_G, y\sim \mu_D(x)}[\ln (1-y)].

If both the generator's and the discriminator's strategy sets are spanned by a finite number of strategies, then by the minimax theorem, \min_{\mu_G}\max_{\mu_D} L(\mu_G, \mu_D)= \max_{\mu_D}\min_{\mu_G} L(\mu_G, \mu_D) that is, the move order does not matter. However, since the strategy sets are both not finitely spanned, the minimax theorem does not apply, and the idea of an "equilibrium" becomes delicate.

To wit, there are the following different concepts of equilibrium: Equilibrium when generator moves first, and discriminator moves second: \hat \mu_G \in \arg\min_{\mu_G}\max_{\mu_D} L(\mu_G,\mu_D),\quad \hat \mu_D \in \arg\max_{\mu_D} L(\hat\mu_G, \mu_D), \quad Equilibrium when discriminator moves first, and generator moves second: \hat \mu_D \in \arg\max_{\mu_D}\min_{\mu_G} L(\mu_G, \mu_D), \quad \hat \mu_G \in \arg\min_{\mu_G} L(\mu_G,\hat \mu_D), Nash equilibrium (\hat \mu_D, \hat\mu_G) , which is stable under simultaneous move order: \hat \mu_D \in \arg\max_{\mu_D} L(\hat\mu_G, \mu_D), \quad \hat \mu_G \in \arg\min_{\mu_G} L(\mu_G, \hat\mu_D) For general games, these equilibria do not have to agree, or even to exist.

For the original GAN game, these equilibria all exist, and are all equal. However, for more general GAN games, these do not necessarily exist, or agree. | math_statement = For any fixed generator strategy \mu_G , let D^*\in\arg\max_{D} L(\mu_G, D) be an optimal deterministic discriminator.

Then \begin{align} D^*(x) &= \frac{d\mu_{\text{ref}}}{d(\mu_{\text{ref}} + \mu_G)} \quad(\mu_{\text{ref}}+\mu_G)\text{-almost everywhere}\\[6pt] L(\mu_G, D^*) &= 2D_{JS}(\mu_{\text{ref}}; \mu_G) - 2\ln 2 \end{align} where the derivative is the Radon–Nikodym derivative, and D_{JS} is the Jensen–Shannon divergence. | name = Theorem | note = the optimal discriminator computes the Jensen–Shannon divergence }}{{Math proof|proof= By Jensen's inequality, \operatorname E_{x\sim \mu_{\text{ref}}, y\sim \mu_D(x)}[\ln y] \leq \operatorname E_{x\sim \mu_{\text{ref}}}[\ln \operatorname E_{y\sim \mu_D(x)}[y and similarly for the other term.

The endpoint cases follow by applying Jensen's inequality to \ln(y+\varepsilon) and \ln(1-y+\varepsilon) and letting \varepsilon\downarrow0 . The kernel mean is measurable, so replacing \mu_D(x) by the deterministic kernel at its mean cannot decrease the discriminator's payoff. Thus an optimal reply can be written as \mu_D(x) = \delta_{D(x)} for a measurable function D: \Omega \to [0, 1] , in which case L(\mu_G, \mu_D) := \operatorname E_{x\sim \mu_{\text{ref}}}[\ln D(x)] + \operatorname E_{x\sim \mu_{G}}[\ln (1-D(x))].

To define suitable density functions, we define a base measure \mu := \mu_{\text{ref}} + \mu_G , which allows us to take the Radon–Nikodym derivatives \rho_{\text{ref}} = \frac{d\mu_{\text{ref}}}{d\mu} \quad \rho_{G} = \frac{d\mu_{G}}{d\mu} with \rho_{\text{ref}} + \rho_G = 1 \mu -almost everywhere. We then have L(\mu_G, \mu_D) := \int \mu(dx) \left[\rho_{\text{ref}}(x) \ln(D(x)) + \rho_G(x) \ln(1-D(x))\right]. The integrand is just the negative cross-entropy between two Bernoulli random variables with parameters \rho_{\text{ref}}(x) and D(x) .

We can write this as -H(\rho_{\text{ref}}(x))-D_{KL}(\rho_{\text{ref}}(x) \parallel D(x)) , where H is the binary entropy function, so L(\mu_G, \mu_D) = -\int \mu(dx) (H(\rho_{\text{ref}}(x)) + D_{KL}(\rho_{\text{ref}}(x) \parallel D(x))).

This means that the optimal strategy for the discriminator is D(x) = \rho_{\text{ref}}(x) \mu -almost everywhere, with L(\mu_G, \mu_D^*) = -\int \mu(dx) H(\rho_{\text{ref}}(x)) = 2D_{JS}(\mu_{\text{ref}} \parallel \mu_G) -2 \ln 2 after routine calculation. }} Interpretation: For any fixed generator strategy \mu_G , the optimal discriminator keeps track of the likelihood ratio between the reference distribution and the generator distribution.

Since D^*=\rho_{\text{ref}} and 1-D^*=\rho_G , \frac{D^*(x)}{1-D^*(x)}=\frac{\rho_{\text{ref}}(x)}{\rho_G(x)} \quad \mu\text{-almost everywhere on }\{\rho_G>0\}. On \{\rho_G=0\} , D^*=1 ; on \{\rho_{\text{ref}}=0\} , D^*=0 , in each case \mu -almost everywhere. If \mu_{\text{ref}}\ll\mu_G , the displayed ratio is a version of d\mu_{\text{ref}}/d\mu_G \mu_G -almost everywhere. Where both densities are positive, D^*(x)=\sigma\left(\ln\rho_{\text{ref}}(x)-\ln\rho_G(x)\right) \quad\mu\text{-almost everywhere}, where \sigma is the logistic function.

In the equal-prior mixture model M=(\mu_{\text{ref}}+\mu_G)/2 , D^* is a version of the posterior probability that an observation came from the reference distribution, M -almost everywhere. {{Math theorem | math_statement = For the unrestricted GAN game defined above, there exists a pair (\hat\mu_D, \hat\mu_G) that is both a saddle point and a Nash equilibrium: \begin{align} & L(\hat\mu_G, \hat\mu_D) = \min_{\mu_G}\max_{\mu_D} L(\mu_G, \mu_D) =& \max_{\mu_D}\inf_{\mu_G} L(\mu_G, \mu_D) = -2\ln 2\\[6pt] & \hat \mu_D \in \arg\max_{\mu_D}\inf_{\mu_G} L(\mu_G, \mu_D), &\quad \hat \mu_G \in \arg\min_{\mu_G}\max_{\mu_D} L(\mu_G, \mu_D)\\[6pt] & \hat \mu_D \in \arg\max_{\mu_D} L(\hat\mu_G, \mu_D), &\quad \hat \mu_G \in \arg\min_{\mu_G} L(\mu_G, \hat\mu_D)\\[6pt] & \forall x\in \Omega, \hat \mu_D(x) = \delta_{\frac 1 2}, &\quad\hat \mu_G = \mu_{\text{ref}} \end{align} That is, the generator perfectly mimics the reference, and the discriminator outputs \frac 12 deterministically on all inputs. | name = Theorem | note = an equilibrium point }} {{Math proof|proof= From the previous theorem, \arg\min_{\mu_G}\max_{\mu_D} L(\mu_G, \mu_D)= \{\mu_{\text{ref}}\}; \quad \min_{\mu_G}\max_{\mu_D} L(\mu_G, \mu_D) = -2\ln 2.

For any fixed discriminator strategy \mu_D , \inf_{\mu_G}L(\mu_G,\mu_D) =\operatorname E_{x\sim\mu_{\text{ref}},\,y\sim\mu_D(x)}[\ln y] +\inf_{x\in\Omega}\operatorname E_{y\sim\mu_D(x)}[\ln(1-y)]. Thus, \arg\max_{\mu_D}\inf_{\mu_G} L(\mu_G, \mu_D) = \arg\max_{\mu_D} \operatorname E_{x\sim \mu_{\text{ref}}, y\sim \mu_D(x)}[\ln y] + \inf_x \operatorname E_{y\sim \mu_D(x)}[\ln (1-y)]. By Jensen's inequality, the discriminator can only improve by adopting the deterministic strategy of always playing D(x) = \operatorname E_{y\sim \mu_D(x)}[y] .

Therefore, \arg\max_{\mu_D}\inf_{\mu_G} L(\mu_G, \mu_D) = \arg\max_{D} \operatorname E_{x\sim \mu_{\text{ref}}}[\ln D(x)] + \inf_x \ln (1-D(x)) By Jensen's inequality, \begin{align} & \operatorname E_{x\sim\mu_{\text{ref}}}[\ln D(x)] + \inf_x \ln(1-D(x)) \\[6pt] \leq {} & \ln \operatorname E_{x\sim \mu_{\text{ref}}}[ D(x)] + \inf_x \ln (1-D(x)) \\[6pt] = {} & \ln \operatorname E_{x\sim \mu_{\text{ref}}}[ D(x)] + \ln (1-\sup_x D(x)) \\[6pt] \forall x\in \Omega, \hat \mu_D(x) = \delta_{\frac 1 2}; \quad \max_{\mu_D}\inf_{\mu_G} L(\mu_G, \mu_D) = -2\ln 2.

Finally, to check that this is a Nash equilibrium, note that when \mu_G = \mu_{\text{ref}} , we have L(\mu_G, \mu_D) := \operatorname E_{x\sim \mu_{\text{ref}}, y\sim \mu_D(x)}[\ln (y (1-y))] which is always maximized by y = \frac 12 .

When \forall x\in \Omega, \mu_D(x) = \delta_{\frac 1 2} , any strategy is optimal for the generator.}} Another evaluation method is the Learned Perceptual Image Patch Similarity (LPIPS), which starts with a learned image featurizer f_\theta: \text{Image} \to \R^n , and finetunes it by supervised learning on a set of (x, x', \operatorname{perceptual~difference}(x, x')) , where x is an image, x' is a perturbed version of it, and \operatorname{perceptual~difference}(x, x') is how much they differ, as reported by human subjects.

The model is finetuned so that it can approximate \|f_\theta(x) - f_\theta(x')\| \approx \operatorname{perceptual~difference}(x, x') . This finetuned model is then used to define \operatorname{LPIPS}(x, x') := \|f_\theta(x) - f_\theta(x')\| . Other evaluation methods are reviewed in.

Concretely, the conditional GAN game is just the GAN game with class labels provided: L(\mu_G, D) := \operatorname E_{c \sim \mu_C, x\sim \mu_{\text{ref}}(c)}[\ln D(x, c)] + \operatorname E_{c \sim \mu_C, x\sim \mu_G(c)}[\ln (1-D(x, c))] where \mu_C is a probability distribution over classes, \mu_{\text{ref}}(c) is the probability distribution of real images of class c , and \mu_G(c) the probability distribution of images generated by the generator when given class label c . In 2017, a conditional GAN learned to generate 1000 image classes of ImageNet.

We recast the original GAN objective into a form more convenient for comparison: \begin{cases} \min_D L_D(D, \mu_G) = -\operatorname E_{x\sim \mu_{G}}[\ln D(x)] - \operatorname E_{x\sim \mu_{\text{ref}}}[\ln (1-D(x))]\\ \min_G L_G(D, \mu_G) = -\operatorname E_{x\sim \mu_G}[\ln (1-D(x))] \end{cases} Original GAN, non-saturating loss: This objective for generator was recommended in the original paper for faster convergence. L_G = \operatorname E_{x\sim \mu_G}[\ln D(x)] The effect of using this objective is analyzed in Section 2.2.2 of Arjovsky et al.

Original GAN, maximum likelihood: L_G = \operatorname E_{x\sim \mu_G}[({\exp} \circ \sigma^{-1} \circ D) (x)] where \sigma is the logistic function. When the discriminator is optimal, the generator gradient is the same as in maximum likelihood estimation, even though GAN cannot perform maximum likelihood estimation itself.

Hinge loss GAN: L_D = -\operatorname E_{x\sim p_{\text{ref}}}\left[\min\left(0, -1 + D(x)\right)\right] -\operatorname E_{x\sim\mu_G}\left[\min\left(0, -1 - D\left(x\right)\right)\right] L_G = -\operatorname E_{x\sim \mu_G} [D(x)] Least squares GAN: L_D = \operatorname E_{x\sim \mu_{\text{ref}}}[(D(x)-b)^2] + \operatorname E_{x\sim \mu_G}[(D(x)-a)^2] L_G = \operatorname E_{x\sim \mu_G}[(D(x)-c)^2] where a, b, c are parameters to be chosen. The authors recommended a = -1, b = 1, c = 0 .

The objective is L_{WGAN}(\mu_G, D) := \operatorname E_{x\sim \mu_G}[D(x)] -\mathbb E_{x\sim \mu_{\text{ref}}}[D(x)] One of its purposes is to solve the problem of mode collapse (see above). The authors claim "In no experiment did we see evidence of mode collapse for the WGAN algorithm".

Unfortunately, I(c, G(z, c)) is intractable in general, The key idea of InfoGAN is Variational Mutual Information Maximization: indirectly maximize it by maximizing a lower bound {\hat {I}}(G,Q)=\mathbb {E} _{z\sim \mu_Z, c\sim \mu _{C}}[\ln Q(c\mid G(z,c))]; \quad I(c, G(z, c)) \geq \sup_Q \hat I(G, Q) where Q ranges over all Markov kernels of type Q: \Omega_Y \to \mathcal P(\Omega_C) . The InfoGAN game is defined as follows: Three probability spaces define an InfoGAN game: (\Omega_X, \mu_{\text{ref}}) , the space of reference images.

(\Omega_Z, \mu_Z) , the fixed random noise generator. (\Omega_C, \mu_C) , the fixed random information generator. There are 3 players in 2 teams: generator, Q, and discriminator. The generator and Q are on one team, and the discriminator on the other team.

The objective function is L(G, Q, D) = L_{GAN}(G, D) - \lambda \hat I(G, Q) where L_{GAN}(G, D) = \operatorname E_{x\sim \mu_{\text{ref}}, }[\ln D(x)] + \operatorname E_{z\sim \mu_Z}[\ln (1-D(G(z, c)))] is the original GAN game objective, and \hat I(G, Q) = \mathbb E_{z\sim\mu_Z, c\sim\mu_C}[\ln Q(c \mid G(z, c))] Generator-Q team aims to minimize the objective, and discriminator aims to maximize it: \min_{G, Q} \max_D L(G, Q, D) The objective function is L(G, E, D) = \mathbb E_{x\sim \mu_X}[\ln D(x, E(x))] + \mathbb E_{z\sim \mu_Z}[\ln (1-D(G(z), z))] Generator-encoder team aims to minimize the objective, and discriminator aims to maximize it: \min_{G, E} \max_D L(G, E, D) In the paper, they gave a more abstract definition of the objective as: L(G, E, D) = \mathbb E_{(x, z)\sim \mu_{E, X}}[\ln D(x, z)] + \mathbb E_{(x, z)\sim \mu_{G, Z}}[\ln (1-D(x, z))] where \mu_{E, X}(dx, dz) = \mu_X(dx) \cdot \delta_{E(x)}(dz) is the probability distribution on \Omega_X\times \Omega_Z obtained by pushing \mu_X forward via x \mapsto (x, E(x)) , and \mu_{G, Z}(dx, dz) = \delta_{G(z)}(dx)\cdot \mu_Z(dz) is the probability distribution on \Omega_X\times \Omega_Z obtained by pushing \mu_Z forward via z \mapsto (G(x), z) .

Applications of bidirectional models include semi-supervised learning, interpretable machine learning, and neural machine translation.

The objective function is L(G_X, G_Y, D_X, D_Y) = L_{GAN}(G_X, D_X) +L_{GAN}(G_Y, D_Y) + \lambda L_{cycle}(G_X, G_Y) where \lambda is a positive adjustable parameter, L_{GAN} is the GAN game objective, and L_{cycle} is the cycle consistency loss: L_{cycle}(G_X, G_Y) = E_{x\sim \mu_X} \|G_X(G_Y(x)) - x\| + E_{y\sim \mu_Y} \|G_Y(G_X(y)) - y\| The generators aim to minimize the objective, and the discriminators aim to maximize it: \min_{G_X, G_Y} \max_{D_X, D_Y} L(G_X, G_Y, D_X, D_Y) Unlike previous work like pix2pix, which requires paired training data, cycleGAN requires no paired data.

For example, to train a pix2pix model to turn a summer scenery photo to winter scenery photo and back, the dataset must contain pairs of the same place in summer and winter, shot at the same angle; cycleGAN would only need a set of summer scenery photos, and an unrelated set of winter scenery photos.

Consider the original GAN game, slightly reformulated as follows: \begin{cases} \min_D L_D(D, \mu_G) = -\operatorname E_{x\sim \mu_{\text{ref}}}[\ln D(x)] - \operatorname E_{x\sim \mu_G}[\ln (1-D(x))]\\ \min_G L_G(D, \mu_G) = -\operatorname E_{x\sim \mu_G}[\ln (1-D(x))] \end{cases} Now we use data augmentation by randomly sampling semantic-preserving transforms T: \Omega \to \Omega and applying them to the dataset, to obtain the reformulated GAN game: \begin{cases} \min_D L_D(D, \mu_G) = -\operatorname E_{x\sim \mu_{\text{ref}}, T\sim \mu_\text{trans}}[\ln D(T(x))] - \operatorname E_{x\sim \mu_G}[\ln (1-D(x))]\\ \min_G L_G(D, \mu_G) = -\operatorname E_{x\sim \mu_G}[\ln (1-D(x))] \end{cases} This is equivalent to a GAN game with a different distribution \mu_{\text{ref}}' , sampled by T(x) , with x\sim \mu_{\text{ref}}, T\sim \mu_\text{trans} .

For example, if \mu_{\text{ref}} is the distribution of images in ImageNet, and \mu_\text{trans} samples identity-transform with probability 0.5, and horizontal-reflection with probability 0.5, then \mu_{\text{ref}}' is the distribution of images in ImageNet and horizontally-reflected ImageNet, combined. The result of such training would be a generator that mimics \mu_{\text{ref}}' . For example, it would generate images that look like they are randomly cropped, if the data augmentation uses random cropping.

The solution is to apply data augmentation to both generated and real images: \begin{cases} \min_D L_D(D, \mu_G) = -\operatorname E_{x\sim \mu_{\text{ref}}, T\sim \mu_\text{trans}}[\ln D(T(x))] - \operatorname E_{x\sim \mu_G, T\sim \mu_\text{trans}}[\ln (1-D(T(x)))]\\ \min_G L_G(D, \mu_G) = -\operatorname E_{x\sim \mu_G, T\sim \mu_\text{trans}}[\ln (1-D(T(x)))] \end{cases} The authors demonstrated high-quality generation using just 100-picture-large datasets. The StyleGAN-2-ADA paper points out a further point on data augmentation: it must be invertible.

Continue with the example of generating ImageNet pictures. If the data augmentation is "randomly rotate the picture by 0, 90, 180, 270 degrees with equal probability", then there is no way for the generator to know which is the true orientation: Consider two generators G, G' , such that for any latent z , the generated image G(z) is a 90-degree rotation of G'(z) . They would have exactly the same expected loss, and so neither is preferred over the other.

The solution is to only use invertible data augmentation: instead of "randomly rotate the picture by 0, 90, 180, 270 degrees with equal probability", use "randomly rotate the picture by 90, 180, 270 degrees with 0.1 probability, and keep the picture as it is with 0.7 probability". This way, the generator is still rewarded to keep images oriented the same way as un-augmented ImageNet pictures.

Abstractly, the effect of randomly sampling transformations T: \Omega \to \Omega from the distribution \mu_\text{trans} is to define a Markov kernel K_\text{trans}: \Omega \to \mathcal P (\Omega) . Then, the data-augmented GAN game pushes the generator to find some \hat \mu_G\in \mathcal P(\Omega) , such that K_\text{trans}*\mu_{\text{ref}} = K_\text{trans}*\hat\mu_{G} where * is the Markov kernel convolution.

A data-augmentation method is defined to be invertible if its Markov kernel K_\text{trans} satisfies K_\text{trans}*\mu= K_\text{trans}*\mu' \implies \mu = \mu' \quad \forall \mu, \mu' \in \mathcal P(\Omega) Immediately by definition, we see that composing multiple invertible data-augmentation methods results in yet another invertible method. Also by definition, if the data-augmentation method is invertible, then using it in a GAN game does not change the optimal strategy \hat \mu_G for the generator, which is still \mu_{\text{ref}} .

There are two prototypical examples of invertible Markov kernels: Discrete case: Invertible stochastic matrices, when \Omega is finite.

For example, if \Omega = \{\uparrow, \downarrow, \leftarrow, \rightarrow\} is the set of four images of an arrow, pointing in 4 directions, and the data augmentation is "randomly rotate the picture by 90, 180, 270 degrees with probability p , and keep the picture as it is with probability (1-3p) ", then the Markov kernel K_\text{trans} can be represented as a stochastic matrix: [K_\text{trans}] = \begin{bmatrix} (1-3p) & p & p & p \\ p & (1-3p) & p & p \\ p & p & (1-3p) & p \\ p & p & p & (1-3p) \end{bmatrix} and K_\text{trans} is an invertible kernel iff [K_\text{trans}] is an invertible matrix, that is, p \neq 1/4 .

Continuous case: The gaussian kernel, when \Omega = \R^n for some n \geq 1 . For example, if \Omega = \R^{256^2} is the space of 256x256 images, and the data-augmentation method is "generate a gaussian noise z\sim \mathcal N(0, I_{256^2}) , then add \epsilon z to the image", then K_\text{trans} is just convolution by the density function of \mathcal N(0, \epsilon^2 I_{256^2}) .

This is invertible, because convolution by a gaussian is just convolution by the heat kernel, so given any \mu\in\mathcal P(\R^n) , the convolved distribution K_\text{trans} * \mu can be obtained by heating up \R^n precisely according to \mu , then wait for time \epsilon^2/4 . With that, we can recover \mu by running the heat equation backwards in time for \epsilon^2/4 . More examples of invertible data augmentations are found in the paper.

The generator G is decomposed into a pyramid of generators G = G_1 \circ G_2 \circ \cdots \circ G_N , with the lowest one generating the image G_N(z_N) at the lowest resolution, then the generated image is scaled up to r(G_N(z_N)) , and fed to the next level to generate an image G_{N-1}(z_{N-1} + r(G_N(z_N))) at a higher resolution, and so on. The discriminator is decomposed into a pyramid as well.

Progressive GAN is a method for training GAN for large-scale image generation stably, by growing a GAN generator from small to large scale in a pyramidal fashion. Like SinGAN, it decomposes the generator as G = G_1 \circ G_2 \circ \cdots \circ G_N , and the discriminator as D = D_1 \circ D_2 \circ \cdots \circ D_N . During training, at first only G_N, D_N are used in a GAN game to generate 4x4 images.

Then G_{N-1}, D_{N-1} are added to reach the second stage of GAN game, to generate 8x8 images, and so on, until we reach a GAN game to generate 1024x1024 images. To avoid shock between stages of the GAN game, each new layer is "blended in" (Figure 2 of the paper ). For example, this is how the second stage GAN game starts: Just before, the GAN game consists of the pair G_N, D_N generating and discriminating 4x4 images.

Just after, the GAN game consists of the pair ((1-\alpha) + \alpha\cdot G_{N-1})\circ u \circ G_N, D_N \circ d \circ ((1-\alpha) + \alpha\cdot D_{N-1}) generating and discriminating 8x8 images. Here, the functions u, d are image up- and down-sampling functions, and \alpha is a blend-in factor (much like an alpha in image composing) that smoothly glides from 0 to 1.

Applications

Science

Iteratively reconstruct astronomical images Simulate gravitational lensing for dark matter research. Model the distribution of dark matter in a particular direction in space and to predict the gravitational lensing that will occur. Model high energy jet formation and showers through calorimeters of high-energy physics experiments. Approximate bottlenecks in computationally expensive simulations of particle physics experiments. Applications in the context of present and proposed CERN experiments have demonstrated the potential of these methods for accelerating simulation and/or improving simulation fidelity. Reconstruct velocity and scalar fields in turbulent flows. GAN-generated molecules were validated experimentally in mice.

Medical

GAN has been used for generating or enhancing synthetic medical images, such as MRI, PET, panoramic X-ray, or CT scan images images. GAN can be used to detect glaucomatous images helping the early diagnosis which is essential to avoid partial or total loss of vision. GANs have been used to create forensic facial reconstructions of deceased historical figures.

Malicious

that looks deceptively like a photograph of a real person. This image was generated by a StyleGAN based on an analysis of portraits. Concerns have been raised about the potential use of GAN-based human image synthesis for sinister purposes, e.g., to produce fake, possibly incriminating, photographs and videos. GANs can be used to generate unique, realistic profile photos of people who do not exist, in order to automate creation of fake social media profiles.

In 2019 the state of California considered and passed on October 3, 2019, the bill AB-602, which bans the use of human image synthesis technologies to make fake pornography without the consent of the people depicted, and bill AB-730, which prohibits distribution of manipulated videos of a political candidate within 60 days of an election. Both bills were authored by Assembly member Marc Berman and signed by Governor Gavin Newsom. The laws went into effect in 2020.

DARPA's Media Forensics program studies ways to counteract fake media, including fake media produced using GANs.

Fashion, art and advertising

GANs can be used to generate art; The Verge wrote in March 2019 that "The images created by GANs have become the defining look of contemporary AI art." GANs can also be used to inpaint photographs generate fashion models, shadows, photorealistic renders of interior design, industrial design, shoes, etc. Such networks were reported to be used by Facebook. Some have worked with using GAN for artistic creativity, as "creative adversarial network".

A GAN, trained on a set of 15,000 portraits from WikiArt from the 14th to the 19th century, created the 2018 painting Edmond de Belamy, which sold for US$432,500. GANs were used by the video game modding community to up-scale low-resolution 2D textures in old video games by recreating them in 4k or higher resolutions via image training, and then down-sampling them to fit the game's native resolution (resembling supersampling anti-aliasing).

In 2020, Artbreeder was used to create the main antagonist in the sequel to the psychological web horror series Ben Drowned. The author would later go on to praise GAN applications for their ability to help generate assets for independent artists who are short on budget and manpower. In May 2020, Nvidia researchers taught an AI system (termed "GameGAN") to recreate the game of Pac-Man simply by watching it being played.

In August 2019, a large dataset consisting of 12,197 MIDI songs each with paired lyrics and melody alignment was created for neural melody generation from lyrics using conditional GAN-LSTM (refer to sources at GitHub AI Melody Generation from Lyrics).

Miscellaneous

GANs have been used to show how an individual's appearance might change with age. reconstruct 3D models of objects from images, generate novel objects as 3D point clouds, model patterns of motion in video. inpaint missing features in maps, transfer map styles in cartography or augment street view imagery. use feedback to generate images and replace image search systems. visualize the effect that climate change will have on specific houses. reconstruct an image of a person's face after listening to their voice. produces videos of a person speaking, given only a single photo of that person. recurrent sequence generation.

History

In 1991, Juergen Schmidhuber published "artificial curiosity", neural networks in a zero-sum game. The first network is a generative model that models a probability distribution over output patterns. The second network learns by gradient descent to predict the reactions of the environment to these patterns. GANs can be regarded as a case where the environmental reaction is 1 or 0 depending on whether the first network's output is in a given set. Other people had similar ideas but did not develop them similarly.

An idea involving adversarial networks was published in a 20101 blog post by Olli Niemitalo. This idea was never implemented and did not involve stochasticity in the generator and thus was not a generative model. An idea similar to GANs was used to model animal behavior by Wei Li, Melvin Gauci and Roderich Gross in 2013. Another inspiration for GANs was noise-contrastive estimation, which uses the same loss function as GANs and which Goodfellow studied during his PhD in 2010–2014.

Adversarial machine learning has other uses besides generative modeling and can be applied to models other than neural networks. In control theory, adversarial learning based on neural networks was used in 2006 to train robust controllers in a game theoretic sense, by alternating the iterations between a minimizer policy, the controller, and a maximizer policy, the disturbance. In 2017, a GAN was used for image enhancement focusing on realistic textures rather than pixel-accuracy, producing a higher image quality at high magnification.

In 2017, the first faces were generated. These were exhibited in February 2018 at the Grand Palais. Faces generated by StyleGAN in 2019 drew comparisons with Deepfakes.

References

External links

This Person Does Not Exist photorealistic images of people who do not exist, generated by StyleGAN This Cat Does Not Exist photorealistic images of cats who do not exist, generated by StyleGAN

Text derived from https://en.wikipedia.org/w/index.php?oldid=1378459936 under CC BY-SA 4.0; modifications have been made.

References

  1. ↩Wikipedia “Generative adversarial network” revision 1378459936 · retrieved 2026-10-20 publication date (×1) · confidence 0.95 revision URL