基于Wasserstein距离的生成对抗网络(WGAN)将编码器和生成器双向集成于其模型中,从而增强了生成模型的学习能力,但其在优化目标中使用KL散度度量分布间的差异,会导致学习训练过程中出现梯度消失或梯度爆炸问题,降低模型鲁棒性.为克服这一问题,提出了一种基于Wasserstein距离的双向学习推理(WBLI)模型.文章首先建立了真实数据分布与隐数据分布双向学习网络,然后引入Wasserstein距离度量联合概率分布的差异性,并据此推导了可解的损失代价函数,给出了完整的网络学习模型和迭代算法.实验结果表明,WBLI模型有效缓解了传统GAN及其变种的模式坍塌问题,增强了训练学习的鲁棒性,可生产辨识度更高的样本.
In WGAN, embedding encoder into Generative Adversarial Networks(GAN)can enhance the learning ability of the generative model. However, using the Kullback-Leibler(KL)divergence to measure the difference between two distributions in the optimization objective will lead to the gradient vanishing or gradient explosion problem in the learning training process and reduce the robustness of model. In order to tackle this problem, a Wasserstein-distance-based Bidirectional Learned Inference(WBLI)model is proposed in this paper. A bidirectional network is first established for learning the distribution of the true data and latent variables, where the difference of the joint probability distribution is measured by the Wasserstein distance. Based on this Wasserstein distance, we redesign the loss function which is solvable and consequently propose an iterative algorithm. The experimental results show that the WBLI model overcomes the defects of traditional GAN and its variants. It effectively eliminates the model collapse problem of generating models, increases the robustness of training learning, and contributes to the improvement of the recognition rate of classifiers.
[1] BARLOW H B. Unsupervised learning[J]. Neural Computation, 1989, 1(3): 295-311. DOI: 10.1162/neco.1989.1.3.295.
[2] LE Q V, RANZATO M A, MONGA R, et al. Building high-level features using large scale unsupervised learning[EB/OL].(2011-12-29)[2019-03-10] : arXiv:1112.6209[cs.LG]. https://arxiv.org/abs/1112.6209.
[3] WANG X L, GUPTA A. Unsupervised learning of visual representations using videos[Z]. IEEE International Conference on Computer Vision, Santiago, 2015. DOI: 10.1109/iccv.2015.320.
[4] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[Z]. IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 2016. DOI: 10.1109/cvpr.2016.278.
[5] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[EB/OL].(2014-6-10)[2019-3-15]. arXiv:1406.2661[stat.ML]. https://arxiv.org/abs/1406-2661.
[6] ZHANG H, GOODFELLOW I, METAXAS D, et al. Self-attention generative adversarial networks[EB/OL].(2018-5-21)[2019-3-15]. arXiv:1805.08318 [stat.ML]. https://arxiv.org/abs/1805.08318.
[7] LIN Z N, KHETAN A, FANTI G, et al. PacGAN: The power of two samples in generative adversarial networks[EB/OL].(2017-12-12)[2019-02-10]. arXiv:1712.04086[cs.LG]. https://arxiv.org/abs/1712.04086.
[8] CUI Y R, LIU Q, GAO C Y, et al. FashionGAN: Display your fashion design using conditional generative adversarial nets[J]. Computer Graphics Forum, 2018, 37(7): 109-119. DOI: 10.1111/cgf.13552.
[9] WANG T C, LIU M Y, ZHU J Y, et al. High-resolution image synthesis and semantic manipulation with conditional GANs[Z]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, 2018. DOI: 10.1109/cvpr.2018.00917.
[10] KINGMA D P, WELLING M. Auto-encoding variational Bayes[EB/OL].(2013-12-20)[2018-12-05]. arXiv:1312.6114 [stat.ML]. https://arxiv.org/abs/1312.6114v1.
[11] SALIMANS T, GOODFELLOW I, ZAREMBA W, et al. Improved techniques for training GAN[EB/OL].(2016-6-10)[2018-11-25]. arXiv:1606.03498 [cs.LG]. https://arxiv.org/abs/1606.03498.
[12] METZ L K, POOLE B, PFAU D, et al. Unrolled generative adversarial networks[EB/OL].(2016-12-7)[2018-11-22]. arXiv:1611.02163[cs.LG]. https://arxiv.org/abs/1611.02163.
[13] DONAHUE J, Krähenbühl P, DARRELL T. Adversarial feature learning[EB/OL].(2016-5-31)[2018-10-9].arXiv:1605.09782 [cs.LG]. https://arxiv.org/abs/1605.09782.
[14] ZENATI H, ROMAIN M, FOO C S, et al. Adversarially learned anomaly detection[Z]. IEEE International Conference on Data Mining, Singapore, 2018. DOI: 10.1109/icdm.2018.00088.
[15] CHAPELLE O, SCHOÖLKOPF B, ZIEN A. Semi-supervised learning[J]. Journal of the Royal Statistical Society, 2006, 172(2):530-530.
[16] ZHOU Z H, LI M. Semi-supervised learning by disagreement[J]. Knowledge and Information Systems, 2010, 24(3): 415-439. DOI: 10.1007/s10115-009-0209-z.
[17] ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[EB/OL].(2017-1-26)[2018-11-5]. arXiv:1701.07875[stat.ML]. https://arxiv.org/abs/1701.07875.
[18] DEMPSTER A P, LAIRD N M, RUBIN D B. Maximum likelihood from incomplete data via the EM algorithm[J]. Journal of the Royal Statistical Society: Series B(Methodological), 1977, 39(1): 1-22. DOI: 10.1111/j.2517-6161.1977.tb01600.x.
[19] ARJOVSKY M, BOTTOU L. Towards principled methods for training generative adversarial networks[EB/OL].(2017-1-17)[2018-11-29]. arXiv:1701.04862[stat.ML]. https://arxiv.org/abs/1701.04862.
[20] HA C W. Minimax and fixed point theorems[J]. Mathematische Annalen, 1980, 248(1): 73-77. DOI: 10.1007/bf01349255.
[21] TIELEMAN T, HINTON G. RMSProp: Divide the gradient by a running average of its recent magnitude[R]. COURSERA: Neural Networks for Machine Learning, 2012.
[22] LECUN Y, CORTES C. The MNIST database of handwritten digits[DB/OL]. [2018-10-15]. http://yann.lecun.com/exdb/mnist/.
[23] XIAO H, RASUL K, VOLLGRAF R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms[EB/OL].(2017-8-25)[2018-11-10]. arXiv:1708.07747[cs.LG]. https://arxiv.org/abs/1708.07747.
[24] WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. DOI: 10.1109/tip.2003.819861.
[25] ZHAO H, GALLO O, FROSIO I, et al. Loss functions for neural networks for image processing[EB/OL].(2015-11-28)[2018-11-16]. arXiv:1511.08861[cs.CV]. https://arxiv.org/abs/1511.08861.
[26] WANG Z, SIMONCELLI E P, BOVIK A C. Multiscale structural similarity for image quality assessment[Z]. The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, Pacific Grove, 2003. DOI: 10.1109/acssc.2003.1292216.
[27] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324. DOI: 10.1109/5.726791.
