Bidirectional learned inference based on Wasserstein distance

doi:10.3969/j.issn.1000-1565.2020.03.015

Abstract

Abstract: In WGAN, embedding encoder into Generative Adversarial Networks(GAN)can enhance the learning ability of the generative model. However, using the Kullback-Leibler(KL)divergence to measure the difference between two distributions in the optimization objective will lead to the gradient vanishing or gradient explosion problem in the learning training process and reduce the robustness of model. In order to tackle this problem, a Wasserstein-distance-based Bidirectional Learned Inference(WBLI)model is proposed in this paper. A bidirectional network is first established for learning the distribution of the true data and latent variables, where the difference of the joint probability distribution is measured by the Wasserstein distance. Based on this Wasserstein distance, we redesign the loss function which is solvable and consequently propose an iterative algorithm. The experimental results show that the WBLI model overcomes the defects of traditional GAN and its variants. It effectively eliminates the model collapse problem of generating models, increases the robustness of training learning, and contributes to the improvement of the recognition rate of classifiers.

Key words: generative adversarial networks, KL divergence, Wasserstein distance, variational auto-encoder

CLC Number:

TP391

HUA Qiang, LIU Yigong, ZHANG Feng, DONG Chunru. Bidirectional learned inference based on Wasserstein distance[J]. Journal of Hebei University (Natural Science Edition), 2020, 40(3): 328-336.

References

[1] BARLOW H B. Unsupervised learning[J]. Neural Computation, 1989, 1(3): 295-311. DOI: 10.1162/neco.1989.1.3.295.
[2] LE Q V, RANZATO M A, MONGA R, et al. Building high-level features using large scale unsupervised learning[EB/OL].(2011-12-29)[2019-03-10] : arXiv:1112.6209[cs.LG]. https://arxiv.org/abs/1112.6209.
[3] WANG X L, GUPTA A. Unsupervised learning of visual representations using videos[Z]. IEEE International Conference on Computer Vision, Santiago, 2015. DOI: 10.1109/iccv.2015.320.
[4] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[Z]. IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 2016. DOI: 10.1109/cvpr.2016.278.
[5] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[EB/OL].(2014-6-10)[2019-3-15]. arXiv:1406.2661[stat.ML]. https://arxiv.org/abs/1406-2661.
[6] ZHANG H, GOODFELLOW I, METAXAS D, et al. Self-attention generative adversarial networks[EB/OL].(2018-5-21)[2019-3-15]. arXiv:1805.08318 [stat.ML]. https://arxiv.org/abs/1805.08318.
[7] LIN Z N, KHETAN A, FANTI G, et al. PacGAN: The power of two samples in generative adversarial networks[EB/OL].(2017-12-12)[2019-02-10]. arXiv:1712.04086[cs.LG]. https://arxiv.org/abs/1712.04086.
[8] CUI Y R, LIU Q, GAO C Y, et al. FashionGAN: Display your fashion design using conditional generative adversarial nets[J]. Computer Graphics Forum, 2018, 37(7): 109-119. DOI: 10.1111/cgf.13552.
[9] WANG T C, LIU M Y, ZHU J Y, et al. High-resolution image synthesis and semantic manipulation with conditional GANs[Z]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, 2018. DOI: 10.1109/cvpr.2018.00917.
[10] KINGMA D P, WELLING M. Auto-encoding variational Bayes[EB/OL].(2013-12-20)[2018-12-05]. arXiv:1312.6114 [stat.ML]. https://arxiv.org/abs/1312.6114v1.
[11] SALIMANS T, GOODFELLOW I, ZAREMBA W, et al. Improved techniques for training GAN[EB/OL].(2016-6-10)[2018-11-25]. arXiv:1606.03498 [cs.LG]. https://arxiv.org/abs/1606.03498.
[12] METZ L K, POOLE B, PFAU D, et al. Unrolled generative adversarial networks[EB/OL].(2016-12-7)[2018-11-22]. arXiv:1611.02163[cs.LG]. https://arxiv.org/abs/1611.02163.
[13] DONAHUE J, Krähenbühl P, DARRELL T. Adversarial feature learning[EB/OL].(2016-5-31)[2018-10-9].arXiv:1605.09782 [cs.LG]. https://arxiv.org/abs/1605.09782.
[14] ZENATI H, ROMAIN M, FOO C S, et al. Adversarially learned anomaly detection[Z]. IEEE International Conference on Data Mining, Singapore, 2018. DOI: 10.1109/icdm.2018.00088.
[15] CHAPELLE O, SCHOÖLKOPF B, ZIEN A. Semi-supervised learning[J]. Journal of the Royal Statistical Society, 2006, 172(2):530-530.
[16] ZHOU Z H, LI M. Semi-supervised learning by disagreement[J]. Knowledge and Information Systems, 2010, 24(3): 415-439. DOI: 10.1007/s10115-009-0209-z.
[17] ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[EB/OL].(2017-1-26)[2018-11-5]. arXiv:1701.07875[stat.ML]. https://arxiv.org/abs/1701.07875.
[18] DEMPSTER A P, LAIRD N M, RUBIN D B. Maximum likelihood from incomplete data via the EM algorithm[J]. Journal of the Royal Statistical Society: Series B(Methodological), 1977, 39(1): 1-22. DOI: 10.1111/j.2517-6161.1977.tb01600.x.
[19] ARJOVSKY M, BOTTOU L. Towards principled methods for training generative adversarial networks[EB/OL].(2017-1-17)[2018-11-29]. arXiv:1701.04862[stat.ML]. https://arxiv.org/abs/1701.04862.
[20] HA C W. Minimax and fixed point theorems[J]. Mathematische Annalen, 1980, 248(1): 73-77. DOI: 10.1007/bf01349255.
[21] TIELEMAN T, HINTON G. RMSProp: Divide the gradient by a running average of its recent magnitude[R]. COURSERA: Neural Networks for Machine Learning, 2012.
[22] LECUN Y, CORTES C. The MNIST database of handwritten digits[DB/OL]. [2018-10-15]. http://yann.lecun.com/exdb/mnist/.
[23] XIAO H, RASUL K, VOLLGRAF R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms[EB/OL].(2017-8-25)[2018-11-10]. arXiv:1708.07747[cs.LG]. https://arxiv.org/abs/1708.07747.
[24] WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. DOI: 10.1109/tip.2003.819861.
[25] ZHAO H, GALLO O, FROSIO I, et al. Loss functions for neural networks for image processing[EB/OL].(2015-11-28)[2018-11-16]. arXiv:1511.08861[cs.CV]. https://arxiv.org/abs/1511.08861.
[26] WANG Z, SIMONCELLI E P, BOVIK A C. Multiscale structural similarity for image quality assessment[Z]. The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, Pacific Grove, 2003. DOI: 10.1109/acssc.2003.1292216.
[27] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324. DOI: 10.1109/5.726791.