Design of online aggregation optimization of big data based on MapReduce

doi:10.3969/j.issn.1000-1565.2021.02.015

Abstract

Abstract: Aiming at the problems of long execution time, poor execution performance and delayed scheduling performance of big data online aggregation, an optimization program design of big data online aggregation based on MapReduce is proposed. The cluster computing resources of all machines in the cluster can be fully utilized by using the fragment aggregation method, and the heuristic priority method of sub connection is used to optimize the local execution of connection task relation operation of each node, so as to realize the parallel connection of big data online aggregation. The dynamic switch mechanism of big data online aggregation based on hybrid approximate query framework and the dynamic switch mechanism based on progressive approximate estimation are used to reduce the misjudgment rate of hybrid approximate query switching and enhance the execution performance of big data online aggregation. The experimental results show that the online aggregation optimization program designed by this method has less execution time under different data scales, and has significant advantages in basic frequent query performance.

Key words: MapReduce, big data, aggregation optimization, switching mechanism

CLC Number:

TP311.13

LI Jun. Design of online aggregation optimization of big data based on MapReduce[J]. Journal of Hebei University(Natural Science Edition), 2021, 41(2): 212-217.

References

[1] 李恒,全华.基于大数据平台的旅游虚拟产业集群研究[J].经济管理,2018,40(12): 21-38. DOI:10.19616/j.cnki.bmj.2018.12.002.
[2] 刘炳含,付忠广,王永智,等.基于并行计算的大数据挖掘技术及其在电站锅炉性能优化中的应用[J].动力工程学报,2018,38(6): 431-439. DOI:10.3969/j.issn.1674-7607.2018.06.002.
[3] 徐源,程潜善,李阳,等.基于大数据聚类的电力系统中长期负荷预测[J].电力系统及其自动化学报,2017,29(8): 43-48. DOI:10.3969/j.issn.1003-8930.2017.08.007.
[4] 赵宝文,徐华.基于MapReduce的并行MRACO-PAM聚类算法[J].计算机工程与科学,2017, 39(10): 1801-1806. DOI:10.3969/j.issn.1007-130X.2017.10.004.
[5] 翟俊海,张素芳,王聪,等.基于MapReduce的大数据主动学习[J]. 计算机应用,2018, 38(10): 2759-2763. DOI:10.11772/j.issn.1001-9081.2018041141.
[6] 史英杰,杜方,尤亚东.MSOLA:基于多维分层采样的大数据在线聚集技术[J].计算机应用研究,2018,35(2): 375-380. DOI:10.3969/j.issn.1001-3695.2018.02.013.
[7] 薛冰,肖骁,李京忠,等.基于POI大数据的沈阳市住宅与零售业空间关联分析[J].地理科学,2019,39(3): 442-449.
[8] 张滨,乐嘉锦.基于列存储的MapReduce分布式Hash连接算法[J].计算机科学,2018,45: 471-475.
[9] 蒋华,韩飞,王鑫,等.基于MapReduce改进K-NN的大数据分类算法研究[J].微电子学与计算机,2018,35(10): 36-40.
[10] 宋杰,孙宗哲,毛克明,等.MapReduce大数据处理平台与算法研究进展[J].软件学报,2017,28(3): 514-543. DOI:10.13328/j.cnki.jos.005169.
[11] 孙德才,王晓霞.一种基于MapReduce的大数据集相似自连接算法[J].计算机科学,2017,44(5): 20-25,32. DOI:10.11896/j.issn.1002-137X.2017.05.004.
[12] 吴信东,嵇圣硙.MapReduce与Spark用于大数据分析之比较[J].软件学报,2018,29(6): 1770-1791. DOI:10.13328/j.cnki.jos.005557.
[13] 刘肃平,谭志平.基于大数据的辅机设备振动噪声监测分析平台[J].计算机工程与应用,2018,54(22): 258-264. DOI:10.3778/j.issn.1002-8331.1708-0021.
[14] 宋人杰,余通,陈宇红,等.基于MapReduce模型的大数据相似重复记录检测算法[J].上海交通大学学报,2018,52(2): 214-221. DOI:10.16183/j.cnki.jsjtu.2018.02.014.
[15] 上超望,韩梦,刘清堂.大数据背景下在线学习过程性评价系统设计研究[J].中国电化教育,2018(5): 90-95. DOI:10.3969/j.issn.1006-9860.2018.05.013.
[16] 刘士伟,李丹.基于大数据分析的大学生创业风险评估算法设计[J].现代电子技术,2018,41(19): 125-128,132. DOI:10.16652/j.issn.1004-373x.2018.19.030.
[17] 申金鑫,吴烨,陈荦,等.面向空间在线分析的并行近似聚集查询[J].计算机科学与探索,2018,12(10): 1559-1570. DOI:10.3778/j.issn.1673-9418.1709033.
[18] 章琦皓,王枫,王月婷.基于大数据技术的EAST实验数据访问日志分析系统的设计[J].计算机应用与软件,2018, 35(9): 50-55. DOI:10.3969/j.issn.1000-386x.2018.09.009.
[19] 杨勇岩.基于云计算和大数据的综合管理系统的设计与实现[J].电子设计工程,2018,26(13): 61-65. DOI:10.3969/j.issn.1674-6236.2018.13.014.
[20] 商娟叶.基于PSO的云计算环境中大数据优化聚类算法[J].电子设计工程,2018, 26(19): 80-83. DOI:10.3969/j.issn.1674-6236.2018.19.018.
[21] 张晓婷,李茵,唐晶磊.基于优化聚类算法的大数据分流系统设计仿真[J].计算机仿真,2018,35(12): 204-207. DOI:10.3969/j.issn.1006-9348.2018.12.049.
[22] 孟维韬,马彦敏,朴春慧.云计算环境下动态数据聚集算法研究[J].计算机测量与控制,2018,26(1): 177-180. DOI:10.16526/j.cnki.11-4762/tp.2018.01.044.
[23] 肖叶枝,王鹏,黄成云,等.基于大数据的在线学习内容个性化精准推荐分析[J].微型电脑应用,2019,35(7): 41-43. DOI:10.3969/j.issn.1007-757X.2019.07.013.
[24] 郭银芳.大数据环境下网络数据传输及融合优化仿真[J].计算机仿真,2019, 36(4): 120-123. DOI:10.3969/j.issn.1006-9348.2019.04.026.
[25] 黄思杰.基于大数据的智能ERP系统设计[J].现代电子技术,2018,41(24): 94-97. DOI:10.16652/j.issn.1004-373x.2018.24.023. (