Chinese news topic detection based on LDA and T-OPTICS

doi:10.3969/j.issn.1000-1565.2016.01.017

Abstract

Abstract: A method of topic detection from large-scale news dataset is proposed.First,latent dirichlet allocation(LDA)is used to reduce the dimension of data by express the news to probabilistic distribution on a set of topics.Then,T-OPTICS algorithm,one algorithm proved based on OPTICS(ordering point to identify the cluster structure)algorithm,is used to cluster news to topics.Because of the OPTICS algorithm is not sensitive to parameters variation,the influence of parameters choice is reduced.The calculation method of text similarity is proved by considering the effect of time parameters.The experimental results show that the algorithm can detect the topics in the TDT4 data set quickly and effectively.

Key words: LDA model, T-OPTICS, cluster, dimensionality reduction

CLC Number:

TP391.1

LI Cong,YUAN Fang,LIU Yu,LI Xinyu. Chinese news topic detection based on LDA and T-OPTICS[J]. Journal of Hebei University (Natural Science Edition), 2016, 36(1): 106-112.

References

[1] DEERWESTER S.Indexing by latent semantic analysis[J].Journal of the American Society of Information Science,1990,26(4):147-157.DOI:10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9.
[2] JORDAN M I,BLEI D M,NG A Y.Latent Dirichlet Allocation[J].Journal of Machine Learning Research,2003,3:993-1022.DOI:10.1109/MLSP.2011.6064562.
[3] 卜质琼,郑波尽.基于LDA模型的Ad-hoc信息检索方法研究[J].计算机应用研究,2015,32(5):1369-1372.DOI:10.3969/j.issn.1001-3695.2015.05.022. BU Zhiqiong,ZHENG Bojin.Ad hoc information retrieval method based on LDA[J].Application Research of Computers,2015,32(5):1369-1372.DOI:10.3969/j.issn.1001-3695.2015.05.022.
[4] 石晶,胡明,石鑫,等.基于LDA模型的文本分割[J].计算机学报,2008,31(10):1865-1873.DOI:10.3321/j.issn:0254-4164.2008.10.022. SHI Jing,HU Ming,SHI Xin,et al.Text segmentation based on model LDA[J].Chinese Journal of Computers,2008,31(10):1865-1873.DOI:10.3321/j.issn:0254-4164.2008.10.022.
[5] 洪宇,张宇,刘挺,等.话题检测与跟踪的评测及研究综述[J].中文信息学报,2007,21(6):71-87.DOI:10.3969/j.issn.1003-0077.2007.06.011. HONG Yu,ZHANG Yu,LIU Ting,et al.Topic detection and tracking review[J].Journal of Chinese Information Processing,2007,21(6):71-87.DOI:10.3969/j.issn.1003-0077.2007.06.011.
[6] 贺敏,杜攀,张瑾,等.基于动量模型的微博突发话题检测方法[J].计算机研究与发展,2015,52(5):1022-1028.DOI:10.7544/issn1000-1239.2015.20131549. HE Min,DU Pan,ZHANG Jin,et al.Microblog bursty topic detection method based on momentum model[J].Journal of Computer Research and Development,2015,52(5):1022-1028.DOI:10.7544/issn1000-1239.2015.20131549.
[7] DING W,CHEN C.Dynamic topic detection and tracking:A comparison of HDP,C-word,and cocitation methods[J].Journal of the Association for Information Science & Technology,2014,65(10):2084-2097.DOI:10.1002/asi.23134.
[8] 马彬,洪宇,陆剑江,等.基于线索树双层聚类的微博话题检测[J].中文信息学报,2012,26(6):121-128.DOI:10.3969/j.issn.1003-0077.2012.06.017. MA Bin,HONG Yu,LU Jianjiang,et al.A thread-based two-stage clustering method of microblog topic detection[J].Journal of Chinese Information Processing,2012,26(6):121-128.DOI:10.3969/j.issn.1003-0077.2012.06.017.
[9] ANKERST M.OPTICS:Ordering points to identify the clustering structure[C] // Proc 1999 ACM SIGMOD International Conference on Management of Data(SIGMOD-99)1999:49-60.DOI:10.1145/304181.304187.
[10] 张小明,李舟军,巢文涵.基于增量型聚类的自动话题检测研究[J].软件学报,2012,23(6):1578-1587.DOI:10.3724/SP.J.1001.2012.04111. ZHANG Xiaoming,LI Zhoujun,CHAO Wenhan.Research of automatic topic detection based on incremental clustering[J].Journal of Software,2012,23(6):1578-1587.DOI:10.3724/SP.J.1001.2012.04111.

[1]	ZHENG Yibo,LI Zhongcan,CHENG Yangxin,DONG Yuhe,ZHU Zhanlong. A robust suppressed fuzzy C-means image segmenting algorithm based on image patch [J]. Journal of Hebei University(Natural Science Edition), 2023, 43(4): 442-448.
[2]	ZHANG Yujing, WANG Liu, QI Xiaona, XU Meiling, WANG Lei. Attribute reduction of commercial bank customer portrait based on information entropy [J]. Journal of Hebei University(Natural Science Edition), 2022, 42(1): 98-104.
[3]	ZHANG Jian, LU Kai, SHI Jing, GUO Huaizhong. Application of partial degradation products of polysaccharide in analysis of polysaccharide from Banlangen Granules [J]. Journal of Hebei University(Natural Science Edition), 2021, 41(1): 40-46.
[4]	ZHAI Junhai, TIAN Shi, ZHANG Sufang, WANG Mohan, SONG Dandan. A comparison on big data fuzzy K-means algorithm based on MapReduce and Spark [J]. Journal of Hebei University (Natural Science Edition), 2020, 40(4): 433-440.
[5]	CHANG Tieyuan, LIU Weina, ZHANG Yan, LI Huiya. Optimized LEACH protocol based on distance and energy of cluster head [J]. Journal of Hebei University (Natural Science Edition), 2019, 39(2): 194-200.
[6]	QI Xiaona,WANG Jia, XU Dongsheng, ZHANG Yujing,GUO Jia,LIU Yang. Application of improved k-means differential privacy protection in location privacy protection [J]. Journal of Hebei University (Natural Science Edition), 2018, 38(3): 315-320.
[7]	LI Kai,ZHAI Lulu,CUI Lijuan. v-Support vector machine with weight of samples [J]. Journal of Hebei University (Natural Science Edition), 2018, 38(2): 185-193.
[8]	GAO Xuewei,FU Zhongguang,SUN Li,ZHANG Gang. Big data modeling of ball mill based on distributed support vector machine on Hadoop platform [J]. Journal of Hebei University (Natural Science Edition), 2017, 37(3): 309-315.
[9]	WANG Guanglei,YUAN Hao,WEI Shuai,SHI Yasong,LIU Xiuling. A novel sequence aorta CTA image segmentation algorithm based on region growing and clustering [J]. Journal of Hebei University (Natural Science Edition), 2015, 35(6): 644-650.
[10]	HA Yan,DU Ruizhong,ZHONG Lian,ZHANG Dongqi,LI Sen. An improved algorithm of bursty topic detection [J]. Journal of Hebei University (Natural Science Edition), 2015, 35(5): 526-531.
[11]	LI Kai,WANG Liang,ZHOU Yufei. A semi-supervised fuzzy clustering algorithm based on pairwise constraints and Mahalanobis distance [J]. Journal of Hebei University (Natural Science Edition), 2014, 34(5): 535-540.
[12]	WANG Chun-cai,YANG Hua-min,ZHANG Cai-hong,GUO Wei,HAN Gui-dong. A Incremental Clustering Algorithm in Data Warehouse Environment [J]. Journal of Hebei University (Natural Science Edition), 2010, 30(2): 211-215.
[13]	TANG Hao,LIU Xi-yu. Density Based Spatial Clustering on Manifolds [J]. Journal of Hebei University (Natural Science Edition), 2009, 29(6): 658-662.