Review: PR-087-Spectral Normalization for Generative Adversarial Networks
One of the challenges in the study of generative adversarial networks is the instability of its training
- Spectral Normalization is a novel weight normalization technique to stabilize the training of discriminator of GANs by enforcing Lipschitz constraint on the discriminator of GANs. It only requires tuning Lipschitz constant in order to produce satisfactory performance.
Motivation
- WGAN and WGAN-GP tried to address the problem — instability of training of GANs.
- WGAN: by clipping weights of discriminator
- WGAN-GP: by introducing gradient penalty on its loss function
- However, even WGAN-GP cannot impose regularization on the entire function (discriminator) space outside of the supports of the generator and data distributions. In other words, its generalizability is low.
- How can we enforce Lipschitz constraints on the entire function space?
- A: Spectral Normalization can help!
Matrix Norm
Spectral Normalization
- Spectral normalization achieves bounding Lipschitz norm of a neural network!!!
Gradient Analysis of Spectrally Normalized Weights
