大数据环境下的存储系统构建：挑战、方法和趋势

陈游旻; 李飞; 舒继武

doi:10.11959/j.issn.2096-0271.2019030

您当前的位置：

首页 >

文章列表页 >

大数据环境下的存储系统构建：挑战、方法和趋势

专题：大数据的系统结构 | 更新时间：2024-06-03

- 大数据环境下的存储系统构建：挑战、方法和趋势
- Building storage systems in big data era:challenges,methods and trends
- 大数据 2019年5卷第4期页码：27-40
- 作者机构：
- 作者简介：
  
  [ "陈游旻（1993- ），男，清华大学计算机科学与技术系博士生，主要研究方向为文件系统、分布式系统。" ]
  [ "李飞（1993- ），男，清华大学计算机科学与技术系硕士生，主要研究方向为闪存存储系统。" ]
  [ "舒继武（1968- ），男，博士，清华大学计算机科学与技术系教授，教育部长江学者特聘教授，IEEE Fellow，博士生导师。近年来主要从事基于非易失存储器件的新型存储系统与技术、基于Flash器件的固态存储系统与技术、网络（云/大数据）存储系统与关键技术、数据存储可靠性等方面的研究工作。" ]
- 基金信息：
  
  ：国家重点研发计划基金资助项目;National Key Research and Development Program of China(2018YFB1003301)
- DOI：10.11959/j.issn.2096-0271.2019030
  中图分类号： TP316.4
- 网络首发：2019-07，
  
  纸质出版：2019-07-15
- 稿件说明：
移动端阅览
陈游旻, 李飞, 舒继武. 大数据环境下的存储系统构建：挑战、方法和趋势[J]. 大数据, 2019,5(4):27-40.

Youmin CHEN, Fei LI, Jiwu SHU. Building storage systems in big data era:challenges,methods and trends[J]. Big Data Research, 2019, 5(4): 27-40.
陈游旻, 李飞, 舒继武. 大数据环境下的存储系统构建：挑战、方法和趋势[J]. 大数据, 2019,5(4):27-40. DOI： 10.11959/j.issn.2096-0271.2019030.

Youmin CHEN, Fei LI, Jiwu SHU. Building storage systems in big data era:challenges,methods and trends[J]. Big Data Research, 2019, 5(4): 27-40. DOI： 10.11959/j.issn.2096-0271.2019030.

摘要

互联网规模的迅速扩展促使全球数据总量呈现爆炸式的增长。物联网、电子商务等新的应用对数据存储及处理的实时性提出了更高的要求，迫切需要结合新型存储介质，以构建大规模、高性能存储系统。分别从闪存存储、持久性内存存储两种存储系统构建方案出发，详细阐述了其各自面临的挑战，并总结了现有的解决方案。最后，展望了未来数据中心及存储系统构建的若干发展趋势。

Abstract

The rapid expansion of the Internet has led to an explosive growth in global data.New applications

such as the Internet of things and e-commerce

demand higher requirements for the latency and performance of data storage and processing.Thus

big data is not only more “big” but also more “fast”

and there is an urgent need to combine new storage media to build large-scale

high-performance storage systems.Two storage system design schemes were started

namely

flash storage and persistent memory storage.Their respective challenges were illustrated in detail

and the existing solutions were summarized.Finally

the future trends in the construction of data centers and storage systems were looked into.

关键词

Keywords

references

SHARMA S . Trends in server efficiency and power usage in data centers [R ] . Auckland:SPEC 2016 ASIA SUMMIT , 2016 .

AJOUX P , BRONSON N , KUMAR S , et al . Challenges to adopting stronger consistency at scale [C ] // The 15th Workshop on Hot Topics in Operating Systems (HotOS XV),May 18-20,2015,Switzerland . Berkeley:USENIX Association , 2015 .

AGRAWAL N , PRABHAKARAN V , WOBBER T , et al . Design tradeoffs for SSD performance [C ] // USENIX Annual Technical Conference,June 22-27,2008,Boston,USA . Berkeley:USENIX Association , 2008 :57.

LU Y Y , SHU J W , ZHENG W . Extending the lifetime of flash-based storage through reducing write amplification from file systems [C ] // The 11th USENIX Conference on File and Storage Technologies,February 12-15,San Jose,USA . Berkeley:USENIX Association , 2013 : 257 - 270 .

LEE S , LIU M , JUN S , et al . Applicationmanaged flash [C ] // The 14th USENIX Conference on File and Storage Technologies,February 22-25,2016,San Jose,USA . Berkeley:USENIX Association , 2016 : 339 - 353 .

OUYANG J , LIN S , JIANG S , et al . SDF:software-defined flash for webscale internet storage systems [J ] . ACM SIGPLAN Notices , 2014 , 49 ( 4 ): 471 - 484 .

ZHANG J , LU Y , SHU J , et al . FlashKV:accelerating KV performance with openchannel SSDs [J ] . ACM Transactions on Embedded Computing Systems , 2017 , 16 ( 5s ):139.

BJØRLING M , GONZÁLEZ J , BONNET P . LightNVM:the Linux open-channel{SSD} subsystem [C ] // The 15th USENIX Conference on File and Storage Technologies,February 27-March 2,2017,Santa Clara,USA . Berkeley:USENIX Association , 2017 : 359 - 374 .

HUANG F B , . Achieving both performance isolation and uniform lifetime for virtualized SSDs [C ] // The 15th USENIX Conference on File and Storage Technologies,February 27-March 2,2017,Santa Clara,USA . Berkeley:USENIX Association , 2017 .

ZHANG J , SHU J , LU Y . ParaFS:a log-structured file system to exploit the internal parallelism of flash devices [C ] // 2016 USENIX Annual Technical Conference,June 22-24,2016,Denver,USA . Berkeley:USENIX Association , 2016 : 87 - 100 .

CARLSON M . Storage for hyperscalers [R ] . Santa Clara:SNIA 2016 , 2016 .

YANG J , PLASSON N , GILLIS G , et al . Don’t stack your log on my log [C ] // The 2nd Workshop on Interactions of NVM/Flash with Operating Systems and Workloads,October 5,2014,Broomfield,USA.[S.l.:s.n . ] , 2014 .

IZRAELEVITZ J , YANG J , ZHANG L , et al . Basic performance measurements of the Intel Optane DC persistent memory module [J ] . Computer Science , 2019 ,arXiv:1903.05714.

SWANSON S . Redrawing the boundary between software and storage for fast non-volatile memories [R ] . San Diego:University of California , 2015 .

CONDIT J , NIGHTINGALE E B , FROST C , et al . Better I/O through byteaddressable,persistent memory [C ] // The ACM SIGOPS 22nd Symposium on Operating Systems Principles,October 11-14,2009,Big Sky,USA . New York:ACM Press , 2009 : 133 - 146 .

NARAYANAN D , HODSON O . Wholesystem persistence [J ] . ACM SIGARCH Computer Architecture News , 2012 , 40 ( 1 ): 401 - 410 .

LU Y Y , SHU J W , SUN L . Blurred persistence in transactional persistent memory [C ] // The 31st Symposium on Mass Storage Systems and Technologies (MSST),May 30-June 5,2015,Santa Clara,USA . Piscataway:IEEE Press , 2015 : 1 - 13 .

DULLOOR S R , KUMAR S , KESHAVAMURTHY A , et al . System software for persistent memory [C ] // The 9th European Conference on Computer Systems,April 14-16,2014,Amsterdam,the Netherlands . New York:ACM Press , 2014 :15.

XU J , SWANSON S . NOVA:a logstructured file system for hybrid volatile/non-volatile main memories [C ] // The 14th USENIX Conference on File and Storage Technologies,February 22-25,2016,Santa Clara,USA . Berkeley:USENIX Association , 2016 : 323 - 338 .

VOLOS H , NALLI S , PANNEERSELVAM S , et al . Aerie:flexible file-system interfaces to storage-class memory [C ] // The 9th European Conference on Computer Systems,April 14-16,2014,Amsterdam,the Netherlands . New York:ACM Press , 2014 :14.

KWON Y , FINGLER H , HUNT T , et al . Strata:a cross media file system [C ] // The 26th Symposium on Operating Systems Principles,October 28-31,2017,Shanghai,China . New York:ACM Press , 2017 : 460 - 477 .

陈游旻 , 陆游游 , 罗圣美 , 等 . 基于RDMA的分布式存储系统研究综述 [J ] . 计算机研究与发展 , 2019 , 56 ( 2 ): 227 - 239 .

CHEN Y M , LU Y Y , LUO S M , et al . Survey on RDMA-based distributed storage systems [J ] . Journal of Computer Research and Development , 2019 , 56 ( 2 ): 227 - 239 .

LU Y Y , SHU J W , CHEN Y M , et al . Octopus:an RDMA-enabled distributed persistent memory file system [C ] // 2017 USENIX Annual Technical Conference (USENIX ATC 17),July 12-14,2017,Santa Clara,USA . Berkeley:USENIX Association , 2017 : 773 - 785 .

浏览量

1043

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

大模型时代下的存储系统挑战与技术发展

面向NVM的IoT时序数据多态协作压缩策略

面向非易失性内存的持久索引数据结构研究综述

面向非易失内存的MPI-IO接口优化