A method of topic detection from large-scale news dataset is proposed.First,latent dirichlet allocation(LDA)is used to reduce the dimension of data by express the news to probabilistic distribution on a set of topics.Then,T-OPTICS algorithm,one algorithm proved based on OPTICS(ordering point to identify the cluster structure)algorithm,is used to cluster news to topics.Because of the OPTICS algorithm is not sensitive to parameters variation,the influence of parameters choice is reduced.The calculation method of text similarity is proved by considering the effect of time parameters.The experimental results show that the algorithm can detect the topics in the TDT4 data set quickly and effectively.
LI Cong
,
YUAN Fang
,
LIU Yu
,
LI Xinyu
. Chinese news topic detection based on LDA and T-OPTICS[J]. Journal of Hebei University(Natural Science Edition), 2016
, 36(1)
: 106
-112
.
DOI: 10.3969/j.issn.1000-1565.2016.01.017
[1] DEERWESTER S.Indexing by latent semantic analysis[J].Journal of the American Society of Information Science,1990,26(4):147-157.DOI:10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9.
[2] JORDAN M I,BLEI D M,NG A Y.Latent Dirichlet Allocation[J].Journal of Machine Learning Research,2003,3:993-1022.DOI:10.1109/MLSP.2011.6064562.
[3] 卜质琼,郑波尽.基于LDA模型的Ad-hoc信息检索方法研究[J].计算机应用研究,2015,32(5):1369-1372.DOI:10.3969/j.issn.1001-3695.2015.05.022. BU Zhiqiong,ZHENG Bojin.Ad hoc information retrieval method based on LDA[J].Application Research of Computers,2015,32(5):1369-1372.DOI:10.3969/j.issn.1001-3695.2015.05.022.
[4] 石晶,胡明,石鑫,等.基于LDA模型的文本分割[J].计算机学报,2008,31(10):1865-1873.DOI:10.3321/j.issn:0254-4164.2008.10.022. SHI Jing,HU Ming,SHI Xin,et al.Text segmentation based on model LDA[J].Chinese Journal of Computers,2008,31(10):1865-1873.DOI:10.3321/j.issn:0254-4164.2008.10.022.
[5] 洪宇,张宇,刘挺,等.话题检测与跟踪的评测及研究综述[J].中文信息学报,2007,21(6):71-87.DOI:10.3969/j.issn.1003-0077.2007.06.011. HONG Yu,ZHANG Yu,LIU Ting,et al.Topic detection and tracking review[J].Journal of Chinese Information Processing,2007,21(6):71-87.DOI:10.3969/j.issn.1003-0077.2007.06.011.
[6] 贺敏,杜攀,张瑾,等.基于动量模型的微博突发话题检测方法[J].计算机研究与发展,2015,52(5):1022-1028.DOI:10.7544/issn1000-1239.2015.20131549. HE Min,DU Pan,ZHANG Jin,et al.Microblog bursty topic detection method based on momentum model[J].Journal of Computer Research and Development,2015,52(5):1022-1028.DOI:10.7544/issn1000-1239.2015.20131549.
[7] DING W,CHEN C.Dynamic topic detection and tracking:A comparison of HDP,C-word,and cocitation methods[J].Journal of the Association for Information Science & Technology,2014,65(10):2084-2097.DOI:10.1002/asi.23134.
[8] 马彬,洪宇,陆剑江,等.基于线索树双层聚类的微博话题检测[J].中文信息学报,2012,26(6):121-128.DOI:10.3969/j.issn.1003-0077.2012.06.017. MA Bin,HONG Yu,LU Jianjiang,et al.A thread-based two-stage clustering method of microblog topic detection[J].Journal of Chinese Information Processing,2012,26(6):121-128.DOI:10.3969/j.issn.1003-0077.2012.06.017.
[9] ANKERST M.OPTICS:Ordering points to identify the clustering structure[C] // Proc 1999 ACM SIGMOD International Conference on Management of Data(SIGMOD-99)1999:49-60.DOI:10.1145/304181.304187.
[10] 张小明,李舟军,巢文涵.基于增量型聚类的自动话题检测研究[J].软件学报,2012,23(6):1578-1587.DOI:10.3724/SP.J.1001.2012.04111. ZHANG Xiaoming,LI Zhoujun,CHAO Wenhan.Research of automatic topic detection based on incremental clustering[J].Journal of Software,2012,23(6):1578-1587.DOI:10.3724/SP.J.1001.2012.04111.
