Design scheme of massive traffic data real-time access based on HBase and ElasticSearch

Changqing DONG; Nver REN; Qingyu ZHANG; Yujing TIAN

doi:10.11959/j.issn.2096-0271.2017010

您当前的位置：

首页 >

文章列表页 >

Design scheme of massive traffic data real-time access based on HBase and ElasticSearch

APPLICATION | 更新时间：2024-06-03

- Design scheme of massive traffic data real-time access based on HBase and ElasticSearch
- Big data research Vol. 3, Issue 1, Pages: 2017010(2017)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-0271.2017010
  CLC： TP311
- Online First：2017-01，
  
  Published：20 January 2017
- 稿件说明：
移动端阅览
Changqing DONG, Nver REN, Qingyu ZHANG, et al. Design scheme of massive traffic data real-time access based on HBase and ElasticSearch[J]. Big data research, 2017, 3(1): 2017010.
DOI：

Changqing DONG, Nver REN, Qingyu ZHANG, et al. Design scheme of massive traffic data real-time access based on HBase and ElasticSearch[J]. Big data research, 2017, 3(1): 2017010. DOI： 10.11959/j.issn.2096-0271.2017010.

摘要

交通流数据具有数据海量、存储和交互速率快等特征，因此其数据的采集、存储及检索成为了车辆远程监控平台中的关键问题。采用LVS集群技术进行数据采集负载均衡，队列缓存处理I/O时延，HBase进行分布式数据存储；针对Hadoop实时在线数据处理不足的问题，整合ElasticSearch并构建了分层索引。通过关键技术的设计和实现，车辆监控由400辆扩展到上万辆，PB级数据在线查询速度提升了10～20倍，验证了方案的高效性。

Abstract

Traffic data has the characteristics of massive and real-time

and its massive data acquisition

storage and retrieval has become a key issue in the vehicle remote monitoring platform.According to the study of these problems

the cluster technology of LVS was used to solve the data acquisition load balance

the queue cache model was used to solve I/O delay

and HBase distributed data storage scheme was used to solve the massive data storage.HBase integration ElasticSearch

which was aimed to solve the real-time online data processing problems of Hadoop

was designed to build a hierarchical index.Through the design and implementation of the key technologies

the number of vehicle monitoring had been promoted from 400 to 1 million

online query speed increased about 10 to 20 times based on PB level data.The results verified the efficiency of the scheme.

关键词

Keywords

references

王颐帅 . 基于LVS 的服务器负载均衡技术 [J ] . 计算机系统应用 , 2014 , 23 ( 7 ): 252 - 255 .

WANG Y S . Server load balancing architecture based on LVS [J ] . Computer Systems ＆ Applications , 2014 , 23 ( 7 ): 252 - 255 .

刘敏娜 , 张继涛 . 基于LVS+KEEPALIVED的高可用负载均衡研究与应用 [J ] . 自动化技术与应用 , 2014 , 33 ( 11 ): 22 - 27 .

LIU M N , ZHANG J T . The study and application of based on the LVS +KEEPALIVED high avaliablility load balance [J ] . Techniques of Automation and Applications , 2014 , 33 ( 11 ): 22 - 27 .

陆婷 , 房俊 , 乔彦克 . 基于HBase 的交通流数据实时存储系统 [J ] . 计算机应用 , 2015 , 35 ( 1 ): 103 - 107 , 135 .

LU T , FANG J , QIAO Y K . HBase-based real-time storage system for traffic stream data [J ] . Journal of Computer Applications , 2015 , 35 ( 1 ): 103 - 107 , 135 .

葛微 , 罗圣美 , 周文辉 . HiBase:一种基于分层式索引的高效HBase 查询技术与系统 [J ] . 计算机学报 , 2015 , 38 ( 35 ): 1 - 15 .

GE W , LUO S M , ZHOU W H . HiBase:a hierarchical indexing mechanism and system for efficient HBase query [J ] . Chinese Journal of Computers , 2015 , 38 ( 35 ): 1 - 15 .

白俊 , 郭贺彬 . 基于ElasticSearch的大日志实时搜索的软件集成方案研究 [J ] . 吉林师范大学学报（自然科学版）, 2014 ( 1 ): 85 - 87 .

BAI J , GUO H B . The design of software integration for big log data real time search based on ElasticSearch [J ] . Jilin Normal University Journal ( Natural Science Edition ), 2014 ( 1 ): 85 - 87 .

钟雨 , 黄向东 , 刘丹 , 等 . 大规模装备监测数据的NoSQL 存储方案 [J ] . 计算机集成制造系统 , 2013 , 19 ( 12 ): 3008 - 3016 .

ZHONG Y , HUANG X D , LIU D , et al . NoSQL storage solution for massive equipment monitoring data management [J ] . Computer Integrated Manufacturing Systems , 2013 , 19 ( 12 ): 3008 - 3016 .

SFAKIANAKIS G ， PATLAKAS I ， NTARMOS N , et al . Interval indexing and querying on key-value cloud stores [C ] // The 29th IEEE International Conference on Data Engineering (ICDE) ， April 8 - 12 , 2013 , Brisbane,Australia . New Jersey : IEEE Press , 2013 : 805 - 816 .

苏命峰 , 陈文芳 , 李仁发 . LVS 集群负载调度算法研究 [J ] . 长沙大学学报 , 2012 ( 5 ): 72 - 74 .

SU M F , CHEN W F , LI R F . Research on LVS cluster load scheduling algorithm [J ] . Journal of Changsha University , 2012 ( 5 ): 72 - 74 .

BAI J . Feasibility analysis of big log data real time search based on HBase and ElasticSearch [C ] // 2013 Ninth International Conference on Natural Computation (ICNC) ， January 28 - 31 , 2013 , San Diego,USA . New Jersey : IEEE Press , 2013 : 1166 - 1170 .

Views

4411

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Making big data analysis more credible

Related Author

Tengjiao WANG

Xilian LI

Related Institution

No data

Postal code：100079
Tel：（010）53879208、53859533 Email：bdr@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：273427 Visits Today：333

⁰