基于Wasserstein距离的双向学习推理

doi:10.3969/j.issn.1000-1565.2020.03.015

河北大学学报(自然科学版) ›› 2020, Vol. 40 ›› Issue (3): 328-336.DOI: 10.3969/j.issn.1000-1565.2020.03.015

• • 上一篇

基于Wasserstein距离的双向学习推理

花强,刘轶功,张峰,董春茹

收稿日期:2019-07-02 出版日期:2020-05-25 发布日期:2020-05-25
通讯作者: 董春茹(1980—),男,河北保定人,河北大学副教授,博士,主要从事机器学习、深度学习方面的研究.E-mail: dongcr@hbu.edu.cn
作者简介:花强(1973—),男,河北邯郸人,河北大学教授,主要从事机器学习、人工神经网络方面的研究. E-mail: huaq@hbu.edu.cn
基金资助:
河北省自然科学基金面上项目(F2018201115,F2018201096);河北省教育厅科学技术研究重点项目(ZD2019021);河北省教育厅科学技术研究青年基金资助项目(QN2017019)

Bidirectional learned inference based on Wasserstein distance

HUA Qiang, LIU Yigong, ZHANG Feng, DONG Chunru

Key Laboratory of Machine Learning and Computational Intelligence of Hebei Province, Hebei University, Baoding 071002, China

Received:2019-07-02 Online:2020-05-25 Published:2020-05-25

摘要/Abstract

摘要： 基于Wasserstein距离的生成对抗网络(WGAN)将编码器和生成器双向集成于其模型中,从而增强了生成模型的学习能力,但其在优化目标中使用KL散度度量分布间的差异,会导致学习训练过程中出现梯度消失或梯度爆炸问题,降低模型鲁棒性.为克服这一问题,提出了一种基于Wasserstein距离的双向学习推理(WBLI)模型.文章首先建立了真实数据分布与隐数据分布双向学习网络,然后引入Wasserstein距离度量联合概率分布的差异性,并据此推导了可解的损失代价函数,给出了完整的网络学习模型和迭代算法.实验结果表明,WBLI模型有效缓解了传统GAN及其变种的模式坍塌问题,增强了训练学习的鲁棒性,可生产辨识度更高的样本.

关键词: 生成对抗网络, KL散度, Wasserstein距离, 变分自编码器

Abstract: In WGAN, embedding encoder into Generative Adversarial Networks(GAN)can enhance the learning ability of the generative model. However, using the Kullback-Leibler(KL)divergence to measure the difference between two distributions in the optimization objective will lead to the gradient vanishing or gradient explosion problem in the learning training process and reduce the robustness of model. In order to tackle this problem, a Wasserstein-distance-based Bidirectional Learned Inference(WBLI)model is proposed in this paper. A bidirectional network is first established for learning the distribution of the true data and latent variables, where the difference of the joint probability distribution is measured by the Wasserstein distance. Based on this Wasserstein distance, we redesign the loss function which is solvable and consequently propose an iterative algorithm. The experimental results show that the WBLI model overcomes the defects of traditional GAN and its variants. It effectively eliminates the model collapse problem of generating models, increases the robustness of training learning, and contributes to the improvement of the recognition rate of classifiers.

Key words: generative adversarial networks, KL divergence, Wasserstein distance, variational auto-encoder

中图分类号:

TP391

花强,刘轶功,张峰,董春茹. 基于Wasserstein距离的双向学习推理[J]. 河北大学学报(自然科学版), 2020, 40(3): 328-336.

HUA Qiang, LIU Yigong, ZHANG Feng, DONG Chunru. Bidirectional learned inference based on Wasserstein distance[J]. Journal of Hebei University (Natural Science Edition), 2020, 40(3): 328-336.

参考文献

[1] BARLOW H B. Unsupervised learning[J]. Neural Computation, 1989, 1(3): 295-311. DOI: 10.1162/neco.1989.1.3.295.
[2] LE Q V, RANZATO M A, MONGA R, et al. Building high-level features using large scale unsupervised learning[EB/OL].(2011-12-29)[2019-03-10] : arXiv:1112.6209[cs.LG]. https://arxiv.org/abs/1112.6209.
[3] WANG X L, GUPTA A. Unsupervised learning of visual representations using videos[Z]. IEEE International Conference on Computer Vision, Santiago, 2015. DOI: 10.1109/iccv.2015.320.
[4] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[Z]. IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 2016. DOI: 10.1109/cvpr.2016.278.
[5] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[EB/OL].(2014-6-10)[2019-3-15]. arXiv:1406.2661[stat.ML]. https://arxiv.org/abs/1406-2661.
[6] ZHANG H, GOODFELLOW I, METAXAS D, et al. Self-attention generative adversarial networks[EB/OL].(2018-5-21)[2019-3-15]. arXiv:1805.08318 [stat.ML]. https://arxiv.org/abs/1805.08318.
[7] LIN Z N, KHETAN A, FANTI G, et al. PacGAN: The power of two samples in generative adversarial networks[EB/OL].(2017-12-12)[2019-02-10]. arXiv:1712.04086[cs.LG]. https://arxiv.org/abs/1712.04086.
[8] CUI Y R, LIU Q, GAO C Y, et al. FashionGAN: Display your fashion design using conditional generative adversarial nets[J]. Computer Graphics Forum, 2018, 37(7): 109-119. DOI: 10.1111/cgf.13552.
[9] WANG T C, LIU M Y, ZHU J Y, et al. High-resolution image synthesis and semantic manipulation with conditional GANs[Z]. IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, 2018. DOI: 10.1109/cvpr.2018.00917.
[10] KINGMA D P, WELLING M. Auto-encoding variational Bayes[EB/OL].(2013-12-20)[2018-12-05]. arXiv:1312.6114 [stat.ML]. https://arxiv.org/abs/1312.6114v1.
[11] SALIMANS T, GOODFELLOW I, ZAREMBA W, et al. Improved techniques for training GAN[EB/OL].(2016-6-10)[2018-11-25]. arXiv:1606.03498 [cs.LG]. https://arxiv.org/abs/1606.03498.
[12] METZ L K, POOLE B, PFAU D, et al. Unrolled generative adversarial networks[EB/OL].(2016-12-7)[2018-11-22]. arXiv:1611.02163[cs.LG]. https://arxiv.org/abs/1611.02163.
[13] DONAHUE J, Krähenbühl P, DARRELL T. Adversarial feature learning[EB/OL].(2016-5-31)[2018-10-9].arXiv:1605.09782 [cs.LG]. https://arxiv.org/abs/1605.09782.
[14] ZENATI H, ROMAIN M, FOO C S, et al. Adversarially learned anomaly detection[Z]. IEEE International Conference on Data Mining, Singapore, 2018. DOI: 10.1109/icdm.2018.00088.
[15] CHAPELLE O, SCHOÖLKOPF B, ZIEN A. Semi-supervised learning[J]. Journal of the Royal Statistical Society, 2006, 172(2):530-530.
[16] ZHOU Z H, LI M. Semi-supervised learning by disagreement[J]. Knowledge and Information Systems, 2010, 24(3): 415-439. DOI: 10.1007/s10115-009-0209-z.
[17] ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[EB/OL].(2017-1-26)[2018-11-5]. arXiv:1701.07875[stat.ML]. https://arxiv.org/abs/1701.07875.
[18] DEMPSTER A P, LAIRD N M, RUBIN D B. Maximum likelihood from incomplete data via the EM algorithm[J]. Journal of the Royal Statistical Society: Series B(Methodological), 1977, 39(1): 1-22. DOI: 10.1111/j.2517-6161.1977.tb01600.x.
[19] ARJOVSKY M, BOTTOU L. Towards principled methods for training generative adversarial networks[EB/OL].(2017-1-17)[2018-11-29]. arXiv:1701.04862[stat.ML]. https://arxiv.org/abs/1701.04862.
[20] HA C W. Minimax and fixed point theorems[J]. Mathematische Annalen, 1980, 248(1): 73-77. DOI: 10.1007/bf01349255.
[21] TIELEMAN T, HINTON G. RMSProp: Divide the gradient by a running average of its recent magnitude[R]. COURSERA: Neural Networks for Machine Learning, 2012.
[22] LECUN Y, CORTES C. The MNIST database of handwritten digits[DB/OL]. [2018-10-15]. http://yann.lecun.com/exdb/mnist/.
[23] XIAO H, RASUL K, VOLLGRAF R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms[EB/OL].(2017-8-25)[2018-11-10]. arXiv:1708.07747[cs.LG]. https://arxiv.org/abs/1708.07747.
[24] WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. DOI: 10.1109/tip.2003.819861.
[25] ZHAO H, GALLO O, FROSIO I, et al. Loss functions for neural networks for image processing[EB/OL].(2015-11-28)[2018-11-16]. arXiv:1511.08861[cs.CV]. https://arxiv.org/abs/1511.08861.
[26] WANG Z, SIMONCELLI E P, BOVIK A C. Multiscale structural similarity for image quality assessment[Z]. The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, Pacific Grove, 2003. DOI: 10.1109/acssc.2003.1292216.
[27] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324. DOI: 10.1109/5.726791.

基于Wasserstein距离的双向学习推理

Bidirectional learned inference based on Wasserstein distance

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 2

编辑推荐

Metrics

本文评价

[1]	刘帅奇,雷钰,庞姣,赵淑欢,苏永钢,孙晨阳. 基于生成对抗网络的SAR图像去噪[J]. 河北大学学报(自然科学版), 2022, 42(3): 306-313.
[2]	申瑞彩,翟俊海,侯璎真. 多生成器生成对抗网络[J]. 河北大学学报(自然科学版), 2021, 41(6): 734-744.