Design,implementation and practice of parallel processing system for a large-scale heterogeneous data

Zhengxun XIA; Shengmei LUO; Yuanhao SUN; Jianfei TANG; Yan ZHANG

doi:10.11959/j.issn.2096-0271.2020030

您当前的位置：

首页 >

文章列表页 >

Design,implementation and practice of parallel processing system for a large-scale heterogeneous data

TOPIC：HETEROGENEOUS PARALLEL SYSTEMS FOR BIG DATA | 更新时间：2024-06-03

- Design,implementation and practice of parallel processing system for a large-scale heterogeneous data
- Big Data Research Vol. 6, Issue 4, Pages: 2020030-1(2020)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-0271.2020030
  CLC： TP311.13
- Online First：2020-07，
  
  Published：15 July 2020
- 稿件说明：
移动端阅览
Zhengxun XIA, Shengmei LUO, Yuanhao SUN, et al. Design,implementation and practice of parallel processing system for a large-scale heterogeneous data[J]. Big Data Research, 2020, 6(4): 2020030-1.
DOI：

Zhengxun XIA, Shengmei LUO, Yuanhao SUN, et al. Design,implementation and practice of parallel processing system for a large-scale heterogeneous data[J]. Big Data Research, 2020, 6(4): 2020030-1. DOI： 10.11959/j.issn.2096-0271.2020030.

摘要

随着互联网和物联网应用的快速发展，数据处理模式从结构化逐渐扩展到结构化、半结构化和非结构化混合的异构数据处理模式。设计了一种大规模异构数据并行处理系统，在统一的平台功能视图基础上，采用统一的资源管理框架，实现对结构化、JSON/XML、图数据、文档数据等多种异构数据进行存储和查询，采用统一的开发语言，实现跨数据类型和数据存储引擎的并行计算，满足多业务应用开发的需要，并通过实际的商业部署，验证了系统的可行性。

Abstract

With the rapid development of Internet and IoT applications

data processing has gradually expanded from structured to structured

semi-structured and unstructured hybrid heterogeneous data processing mode.A large-scale heterogeneous data parallel processing system was designed.Based on the functional view of a unified platform

the unified resource management framework was adopted to store and query a variety of heterogeneous data

including structured

JSON/XML

graph data

document data

etc.Adopting a unified database language

the parallel computing across data types and database engines was realized

and the needs of multi business application development were met.The feasibility of the system is verified by standard evaluation environment and commercial deployment.

关键词

Keywords

references

DAVOUDIAN A , LIU C , LIU M C . A survey on NoSQL stores [J ] . ACM Computing Surveys , 2018 , 51 ( 2 ): 1 - 43 .

SHARMA S , SHANDILYA R , PATNAIK S , et al . Leading NoSQL models for handling big data:a brief review [J ] . International Journal of Business Information Systems , 2016 , 22 ( 1 ): 1 - 25 .

陈民峰 . 国产数据库管理系统应用的研究与探讨 [J ] . 计算机应用 , 2003 , 23 ( z2 ): 39 - 42 .

CHEN M F . Research and discussion on the application of domestic database management system [J ] . Journal of Computer Applications , 2003 , 23 ( z2 ): 39 - 42 .

IDC . Data age 2025,the digitization of the world from edge to core [R ] . 2018 .

STONEBRAKER M , CETINTEMEL U . One size fits all:an idea whose time has come and gone [C ] // The 21st International Conference on Data Engineering . Piscataway:IEEE Press , 2005 .

CODD E F . A relational model of data for large shared data banks [J ] . Communications of the ACM , 1970 , 13 ( 6 ): 377 - 387 .

DEWITT D J , GHANDEHARIZADEH S , SCHNEIDER D A , et al . The Gamma database machine project [J ] . IEEE Transactions on Knowledge ＆ Data Engineering , 1990 , 2 ( 1 ): 44 - 62 .

DEAN J , GHEMAWAT S . MapReduce:simplified data processing on large clusters [J ] . Communications of the ACM , 2008 , 51 ( 1 ): 107 - 113 .

KORNACKER M , BEHM A , BITTORF V , et al . Impala:a modern,open-source SQL engine for Hadoop [C ] // Conference on Innovative Data Systems Research.[S.l.:s.n . ] , 2015 .

ZAHARIA M , CHOWDHURY M , DAS T , et al . Resilient distributed datasets:a fault-tolerant abstraction for in-memory cluster computing [C ] // The 9th USENIX Conference on Networked Systems Design and Implementation . Berkeley:USENIX Association , 2012 .

ALEXANDROV A , BERGMANN R , EWEN S , et al . The stratosphere platform for big data analytics [J ] . VLDB Journal , 2014 , 23 ( 6 ): 939 - 964 .

DONALD F , ADAM R . Hype cycle for data management [Z ] . 2019 .

孙元浩 . 大数据进入3.0时代 [J ] . 信息技术与标准化 , 2019 ( 5 ): 94 - 104 .

SUN Y H . Big data enters 3.0 era [J ] . Information Technology ＆ Standardization , 2019 ( 5 ): 94 - 104 .

刘汪根 , 孙元浩 . 大数据3.0——后Hadoop时代大数据的核心技术 [J ] . 数据与计算发展前沿 , 2019 , 1 ( 1 ): 94 - 104 .

LIU W G , SUN Y H . Big data 3.0,the key technologies of big data in post-Hadoop era [J ] . Frontiers of Data and Computing , 2019 , 1 ( 1 ): 94 - 104 .

Views

821

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research on graph-based heterogeneous data integration method

Related Author

HUANG Yuezhen

YANG Fen

TIAN Feng

LI Yuchan

ZHANG Chengye

Related Institution

GRGBanking I.T. Co., Ltd.

Guangzhou Digital Technology Group Co., Ltd.

GRGBanking Equipment Co., Ltd.

Postal code：100079
Tel：（010）53879208、53859533 Email：bdr@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：273228 Visits Today：134

⁰