给出了一种针对大量新闻数据的话题检测方法.首先通过LDA(latent dirichlet allocation)模型从语义层面抽取新闻数据主题,有效降低数据分析维度,更合理地体现新闻主题特征.然后改进OPTICS(ordering point to identify the cluster structure)密度聚类算法,基于新闻话题的时间延续性给出了T-OPTICS算法.该算法继承了OPTICS算法对参数不敏感的特性,降低了参数选择对聚类结果的影响.改进了OPTICS算法中文本间相似度的计算方法,体现了话题的时间延续性.基于TDT4数据集的实验表明,该方法能够快速有效地发现新闻中的话题.
A method of topic detection from large-scale news dataset is proposed.First,latent dirichlet allocation(LDA)is used to reduce the dimension of data by express the news to probabilistic distribution on a set of topics.Then,T-OPTICS algorithm,one algorithm proved based on OPTICS(ordering point to identify the cluster structure)algorithm,is used to cluster news to topics.Because of the OPTICS algorithm is not sensitive to parameters variation,the influence of parameters choice is reduced.The calculation method of text similarity is proved by considering the effect of time parameters.The experimental results show that the algorithm can detect the topics in the TDT4 data set quickly and effectively.
[1] DEERWESTER S.Indexing by latent semantic analysis[J].Journal of the American Society of Information Science,1990,26(4):147-157.DOI:10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9.
[2] JORDAN M I,BLEI D M,NG A Y.Latent Dirichlet Allocation[J].Journal of Machine Learning Research,2003,3:993-1022.DOI:10.1109/MLSP.2011.6064562.
[3] 卜质琼,郑波尽.基于LDA模型的Ad-hoc信息检索方法研究[J].计算机应用研究,2015,32(5):1369-1372.DOI:10.3969/j.issn.1001-3695.2015.05.022. BU Zhiqiong,ZHENG Bojin.Ad hoc information retrieval method based on LDA[J].Application Research of Computers,2015,32(5):1369-1372.DOI:10.3969/j.issn.1001-3695.2015.05.022.
[4] 石晶,胡明,石鑫,等.基于LDA模型的文本分割[J].计算机学报,2008,31(10):1865-1873.DOI:10.3321/j.issn:0254-4164.2008.10.022. SHI Jing,HU Ming,SHI Xin,et al.Text segmentation based on model LDA[J].Chinese Journal of Computers,2008,31(10):1865-1873.DOI:10.3321/j.issn:0254-4164.2008.10.022.
[5] 洪宇,张宇,刘挺,等.话题检测与跟踪的评测及研究综述[J].中文信息学报,2007,21(6):71-87.DOI:10.3969/j.issn.1003-0077.2007.06.011. HONG Yu,ZHANG Yu,LIU Ting,et al.Topic detection and tracking review[J].Journal of Chinese Information Processing,2007,21(6):71-87.DOI:10.3969/j.issn.1003-0077.2007.06.011.
[6] 贺敏,杜攀,张瑾,等.基于动量模型的微博突发话题检测方法[J].计算机研究与发展,2015,52(5):1022-1028.DOI:10.7544/issn1000-1239.2015.20131549. HE Min,DU Pan,ZHANG Jin,et al.Microblog bursty topic detection method based on momentum model[J].Journal of Computer Research and Development,2015,52(5):1022-1028.DOI:10.7544/issn1000-1239.2015.20131549.
[7] DING W,CHEN C.Dynamic topic detection and tracking:A comparison of HDP,C-word,and cocitation methods[J].Journal of the Association for Information Science & Technology,2014,65(10):2084-2097.DOI:10.1002/asi.23134.
[8] 马彬,洪宇,陆剑江,等.基于线索树双层聚类的微博话题检测[J].中文信息学报,2012,26(6):121-128.DOI:10.3969/j.issn.1003-0077.2012.06.017. MA Bin,HONG Yu,LU Jianjiang,et al.A thread-based two-stage clustering method of microblog topic detection[J].Journal of Chinese Information Processing,2012,26(6):121-128.DOI:10.3969/j.issn.1003-0077.2012.06.017.
[9] ANKERST M.OPTICS:Ordering points to identify the clustering structure[C] // Proc 1999 ACM SIGMOD International Conference on Management of Data(SIGMOD-99)1999:49-60.DOI:10.1145/304181.304187.
[10] 张小明,李舟军,巢文涵.基于增量型聚类的自动话题检测研究[J].软件学报,2012,23(6):1578-1587.DOI:10.3724/SP.J.1001.2012.04111. ZHANG Xiaoming,LI Zhoujun,CHAO Wenhan.Research of automatic topic detection based on incremental clustering[J].Journal of Software,2012,23(6):1578-1587.DOI:10.3724/SP.J.1001.2012.04111.
