Survey on data caching technology of distributed dataflow system

Xuchu YUAN; Guo FU; Jize BI; Yanfeng ZHANG; Tiezheng NIE; Yu GU; Yubin BAO; Ge YU

doi:10.11959/j.issn.2096-0271.2020027

您当前的位置：

首页 >

文章列表页 >

Survey on data caching technology of distributed dataflow system

TOPIC：DATAFLOW COMPUTING TECHNIQUES FOR BIG DATA PROCESSING | 更新时间：2024-06-03

- Survey on data caching technology of distributed dataflow system
- Big Data Research Vol. 6, Issue 3, Pages: 2020027-1(2020)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-0271.2020027
  CLC： TP391
- Online First：2020-05，
  
  Published：15 May 2020
- 稿件说明：
移动端阅览
Xuchu YUAN, Guo FU, Jize BI, et al. Survey on data caching technology of distributed dataflow system[J]. Big Data Research, 2020, 6(3): 2020027-1.
DOI：

Xuchu YUAN, Guo FU, Jize BI, et al. Survey on data caching technology of distributed dataflow system[J]. Big Data Research, 2020, 6(3): 2020027-1. DOI： 10.11959/j.issn.2096-0271.2020027.

摘要

数据流编程模型以其高度并行计算、支持流水线处理、支持函数式编程等优点被许多主流的计算系统采用。在分布式数据流系统和异构数据流系统中，算子之间数据生产和数据消化的速度不一致可能会导致数据堆积或者算子闲置等问题。为支持高效的数据流系统，需要设计缓存系统，以保证数据流的高效缓存和移动。选取了几个典型的分布式数据流系统与分布式消息队列系统进行系统分析，并总结了目前消息队列系统对数据流缓存系统的支持程度。最后对数据缓存技术进行了阐述，并分析了未来的数据流缓存系统的需求和研究方向。

Abstract

Dataflow model is adopted by several dataflow systems for its advantages of high parallel computing

pipeline processing and functional programming.In distributed dataflow systems and heterogeneous dataflow systems

due to the speed mismatch between the data production of data source operators and the data consumption of data sink operators

data could be delayed and operators could be idle.In order to support an efficient dataflow system

a dataflow cache system was desired to ensure efficient caching and movement of dataflow.Several distributed dataflow systems and distributed message queuing systems were analyzed

and the support degree of current message queuing system to data flow caching system was summarized.Finally

the cache technique was introduced

and the demands and research directions of future dataflow caching systems were analyzed.

关键词

Keywords

references

YAZDANPANAH F , ALVAREZ-MARTINEZ C , JIMENEZ-GONZALEZ D , et al . Hybrid dataflow/von-neumann architectures [J ] . IEEE Transactions on Parallel and Distributed Systems , 2014 , 25 ( 6 ): 1489 - 1509 .

NOWATZKI A , GANGADHAR V , SANKARALINGAM K . Computer with hybrid von-neumann/dataflow execution architecture:U.S.Patent 10,216,693 [J ] .2019-2-26.

DENNIS J B , . First version of a data flow procedure language [C ] // Programming Symposium . Heidelberg:Springer , 1974 : 362 - 376 .

DENNIS J B , FOSSEEN J B , LINDMAN J P . Data flow schemas [C ] // International Sympoisum on Theoretical Programming . Heidelberg:Springer , 1972 : 187 - 216 .

DENNIS J B . Fresh breeze:a multiprocessor chip architecture guided by modular programming principles [J ] . ACM SIGARCH Computer Architecture News , 2003 , 31 ( 1 ): 7 - 15 .

NAJJAR W A , LEE E A , GAO G R . Advances in the dataflow computational model [J ] . Parallel Computing , 1999 , 25 ( 13-14 ): 1907 - 1929 .

ZAHARIA M , CHOWDHURY M , FRANKLIN M J , et al . Spark:cluster computing with working sets [C ] // The 2nd USENIX Conference on Hot Topics in Cloud Computing . Berkeley:USENIX Association , 2010 :10.

MENG X , BRADLEY J , YAVUZ B , et al . Mllib:machine learning in Apache Spark [J ] . The Journal of Machine Learning Research , 2016 , 17 ( 1 ): 1235 - 1241 .

KARAU H , KONWINSKI A , WENDELL P , et al . Learning Spark:lightning-fast big data analysis [M ] . Sebastopol : O’Reilly Media,Inc.Press , 2015 .

CARBONE P , KATSIFODIMOS A , EWEN S , et al . Apache Flink:stream and batch processing in a single engine [J ] . Bulletin of the IEEE Computer Society Technical Committee on Data Engineering , 2015 , 36 ( 4 ): 28 - 38 .

FRIEDMAN E , TZOUMAS K . Introduction to Apache Flink:stream processing for real time and beyond [M ] . Sebastopol : O’Reilly Media,Inc.Press , 2016 .

ABADI M , BARHAM P , CHEN J , et al . TensorFlow:a system for large-scale machine learning [C ] // The 12th USENIX Symposium on Operating Systems Design and Implementation . Berkeley:USENIX Association , 2016 : 265 - 283 .

ABADI M , AGARWAL A , BARHAM P , et al . TensorFlow:large-scale machine learning on heterogeneous distributed systems [J ] . Computer Science , 2016 ,arXiv:1603.04467.

WONGSUPHASAWAT K , SMILKOV D , WEXLEX J , et al . Visualizing dataflow graphs of deep learning models in TensorFlow [J ] . IEEE Transactions on Visualization and Computer Graphics , 2017 , 24 ( 1 ): 1 - 12 .

AKIDAU T , BRADSHAW R , CHAMBERS C , et al . The dataflow model:a practical approach to balancing correctness,latency,and cost in massive-scale,unbounded,out-of-order data processing [J ] . Proceedings of VLDB Endowment , 2015 , 8 ( 12 ): 1792 - 1803 .

倪炜 . 分布式消息中间件实践 [M ] . 北京 : 电子工业出版社 , 2018 .

NI W . Practice of distributed message middleware [M ] . Beijing : Publishing House of Electronics IndustryPress , 2018 .

NARKHEDE N , SHAPIRA G , PALINO T . Kafka:the definitive guide:real-time data and stream processing at scale [M ] . Sebastopol : O’Reilly Media,Inc.Press , 2017 .

郑奇煌 . Kafka技术内幕 [M ] . 北京 : 人民邮电出版社 , 2017 .

ZHENG Q H . Inside of Kafka technology [M ] . Beijing : The People’s Posts and Telecommunications PressPress , 2017 .

DOBBELAERE P , ESMAILI K S . Kafka versus RabbitMQ:a comparative study of two industry reference publish/subscribe implementations:industry paper [C ] // The 11th ACM International Conference . New York:ACM Press , 2017 : 227 - 238 .

AYANOGLU E , AYTAS Y , NAHUM D . Mastering RabbitMQ [M ] . Birmingham : Packt Publishing LtdPress , 2016 .

ROSTANSKI M , GROCHLA K , SEMAN A . Evaluation of highly available and fault-tolerant middleware clustered architectures using RabbitMQ [C ] // The 2014 Federated Conference on Computer Science and Information Systems . Piscataway:IEEE Press , 2014 : 879 - 884 .

IONESCU V M , . The analysis of the performance of RabbitMQ and ActiveMQ [C ] // Roedunet International Conference-Networking in Education ＆Research . Piscataway:IEEE Press , 2015 : 132 - 137 .

HENJES R , SCHLOSSER D , MENTH M , et al . Throughput performance of the ActiveMQ JMS server [M ] . Heidelberg : SpringerPress , 2007 .

丁威 , 周继锋 . RocketMQ技术内幕 [M ] . 北京 : 机械工业出版社 , 2018 .

DING W , ZHOU J F . Inside of RocketMQ technology [M ] . Beijing : China Machine PressPress , 2018 .

郭嘉凯 . RocketMQ:从阿里巴巴走向世界 [J ] . 软件和集成电路 , 2018 (11):13.

GUO J K . RocketMQ:from Alibaba to the world [J ] . Software and Integrated Circuit , 2018 (11):13.

李国杰 . 一种新的体系结构——数据流计算机 [J ] . 电子计算机动态 , 1981 ( 11 ): 3 - 10 .

LI G J . A new architecture:dataflow computer [J ] . Computer Review , 1981 ( 11 ): 3 - 10 .

ZUCKERMAN S , SUETTLERLEIN J , KANUERHASE R , et al . Using a Codelet program execution model for exascale machines:position paper [C ] // The 1st International Workshop on Adaptive SelfTuning Computing Systems for the Exaflop Era . New York:ACM Press , 2011 : 64 - 69 .

高光荣 . 大数据的流动之美——数据流与大数据：挑战与机遇 [J ] . 中国计算机学会通讯 , 2013 , 9 ( 12 ): 16 - 19 .

GAO G R . The flowing beauty of big data,dataflow and big data:challenge and opportunity [J ] . Communication of the CCF , 2013 , 9 ( 12 ): 16 - 19 .

NEUMEYER L , ROBBINS B , NAIR A , et al . S4:distributed stream computing platform [C ] // The 2010 IEEE International Conference on Data Mining Workshops . Piscataway:IEEE Press , 2010 : 170 - 177 .

SUETTLERLEIN J , ZUCKERMAN S , GAO G R . An implementation of the Codelet model [M ] . Heidelberg : SpringerPress , 2013 : 633 - 644 .

SUETTERLEIN J . Darts:a runtime based on the Codelet execution model [D ] . Newark:University of Delaware , 2014 .

LAUDERDALE C , KHAN R . Towards a Codelet-based runtime for exascale computing:position paper [C ] // The International Workshop on Adaptive Selftuning Computing Systems for the Exaflop Era . New York:ACM Press , 2012 : 21 - 26 .

DEAN J , CHEMAWAT S . MapReduce:simplified data processing on large clusters [J ] . Communications of the ACM , 2008 , 51 ( 1 ): 107 - 113 .

TOSHNIWAL A , TANEJA S , SHUKLA A , et al . Storm@twitter [C ] // The 2014 ACM SIGMOD International Conference on Management of Data . New York:ACM Press , 2014 : 147 - 156 .

李明 , 王晓鹏 . Storm源码分析 [M ] . 北京 : 人民邮电出版社 , 2014 .

LI M , WANG X P . Source code analysis of Storm [M ] . Beijing : The People’s Posts and Telecommunications PressPress , 2014 .

WU Y , ZHENG L , HEILIG B , et al . HAMR:a dataflow-based real-time in-memory cluster computing engine [J ] . The International Journal of High Performance Computing Applications , 2017 , 31 ( 5 ): 361 - 374 .

MURRAY D G , MCSHERRY F , ISAACS R , et al . Naiad:a timely dataflow system [C ] // The 24th ACM Symposium on Operating Systems Principles . New York:ACM Press , 2013 : 439 - 455 .

SHVACHKO K , KUANG H , RADIA S , et al . The Hadoop distributed file system [C ] // The 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies . Piscataway:IEEE Press , 2010 : 1 - 10 .

HUNT P , KONAR M , JUNQUEIRA F P , et al . ZooKeeper:wait-free coordination for internetscale systems [C ] // The 2010 USENIX Annual Technical Conference . Berkeley:USENIX Association , 2010 :11.

JUNQUEIRA F , REED B . Zookeeper:distributed process coordination [M ] . Sebastopol : O’Reilly Media,Inc.Press , 2013 .

吴璨 , 王小宁 , 肖海力 , 等 . 分布式消息系统研究综述 [J ] . 计算机科学 , 2019 ( B06 ): 1 - 5 .

WU C , WANG X N , XIAO H L , et al . Survey on distributed message system [J ] . Computer Science , 2019 ( B06 ): 1 - 5 .

Views

780

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

The usage of dataflow model in GPU and big data processing

Research on the consensus of big data systems based on RDMA and NVM

Graph processing engine based on graph query system

Optimizing distributed systems with remote direct memory access

Related Author

Huayou SU

Songzhu MEI

Rongchun LI

Yong DOU

Hao WU

Kang CHEN

Yongwei WU

Weimin ZHENG

Related Institution

Department of Computer Science and Technology, Tsinghua University

Beijing National Research Center for Information Science and Technology

Postal code：100079
Tel：（010）53879208、53859533 Email：bdr@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：273228 Visits Today：134

⁰