Building storage systems in big data era:challenges,methods and trends

Youmin CHEN; Fei LI; Jiwu SHU

doi:10.11959/j.issn.2096-0271.2019030

您当前的位置：

首页 >

文章列表页 >

Building storage systems in big data era:challenges,methods and trends

TOPIC:SYSTEMS ARCHITECTURE FOR BIG DATA | 更新时间：2024-06-03

- Building storage systems in big data era:challenges,methods and trends
- Big Data Research Vol. 5, Issue 4, Pages: 27-40(2019)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-0271.2019030
  CLC： TP316.4
- Online First：2019-07，
  
  Published：15 July 2019
- 稿件说明：
移动端阅览
Youmin CHEN, Fei LI, Jiwu SHU. Building storage systems in big data era:challenges,methods and trends[J]. Big Data Research, 2019, 5(4): 27-40.
DOI：

Youmin CHEN, Fei LI, Jiwu SHU. Building storage systems in big data era:challenges,methods and trends[J]. Big Data Research, 2019, 5(4): 27-40. DOI： 10.11959/j.issn.2096-0271.2019030.

摘要

互联网规模的迅速扩展促使全球数据总量呈现爆炸式的增长。物联网、电子商务等新的应用对数据存储及处理的实时性提出了更高的要求，迫切需要结合新型存储介质，以构建大规模、高性能存储系统。分别从闪存存储、持久性内存存储两种存储系统构建方案出发，详细阐述了其各自面临的挑战，并总结了现有的解决方案。最后，展望了未来数据中心及存储系统构建的若干发展趋势。

Abstract

The rapid expansion of the Internet has led to an explosive growth in global data.New applications

such as the Internet of things and e-commerce

demand higher requirements for the latency and performance of data storage and processing.Thus

big data is not only more “big” but also more “fast”

and there is an urgent need to combine new storage media to build large-scale

high-performance storage systems.Two storage system design schemes were started

namely

flash storage and persistent memory storage.Their respective challenges were illustrated in detail

and the existing solutions were summarized.Finally

the future trends in the construction of data centers and storage systems were looked into.

关键词

Keywords

references

SHARMA S . Trends in server efficiency and power usage in data centers [R ] . Auckland:SPEC 2016 ASIA SUMMIT , 2016 .

AJOUX P , BRONSON N , KUMAR S , et al . Challenges to adopting stronger consistency at scale [C ] // The 15th Workshop on Hot Topics in Operating Systems (HotOS XV),May 18-20,2015,Switzerland . Berkeley:USENIX Association , 2015 .

AGRAWAL N , PRABHAKARAN V , WOBBER T , et al . Design tradeoffs for SSD performance [C ] // USENIX Annual Technical Conference,June 22-27,2008,Boston,USA . Berkeley:USENIX Association , 2008 :57.

LU Y Y , SHU J W , ZHENG W . Extending the lifetime of flash-based storage through reducing write amplification from file systems [C ] // The 11th USENIX Conference on File and Storage Technologies,February 12-15,San Jose,USA . Berkeley:USENIX Association , 2013 : 257 - 270 .

LEE S , LIU M , JUN S , et al . Applicationmanaged flash [C ] // The 14th USENIX Conference on File and Storage Technologies,February 22-25,2016,San Jose,USA . Berkeley:USENIX Association , 2016 : 339 - 353 .

OUYANG J , LIN S , JIANG S , et al . SDF:software-defined flash for webscale internet storage systems [J ] . ACM SIGPLAN Notices , 2014 , 49 ( 4 ): 471 - 484 .

ZHANG J , LU Y , SHU J , et al . FlashKV:accelerating KV performance with openchannel SSDs [J ] . ACM Transactions on Embedded Computing Systems , 2017 , 16 ( 5s ):139.

BJØRLING M , GONZÁLEZ J , BONNET P . LightNVM:the Linux open-channel{SSD} subsystem [C ] // The 15th USENIX Conference on File and Storage Technologies,February 27-March 2,2017,Santa Clara,USA . Berkeley:USENIX Association , 2017 : 359 - 374 .

HUANG F B , . Achieving both performance isolation and uniform lifetime for virtualized SSDs [C ] // The 15th USENIX Conference on File and Storage Technologies,February 27-March 2,2017,Santa Clara,USA . Berkeley:USENIX Association , 2017 .

ZHANG J , SHU J , LU Y . ParaFS:a log-structured file system to exploit the internal parallelism of flash devices [C ] // 2016 USENIX Annual Technical Conference,June 22-24,2016,Denver,USA . Berkeley:USENIX Association , 2016 : 87 - 100 .

CARLSON M . Storage for hyperscalers [R ] . Santa Clara:SNIA 2016 , 2016 .

YANG J , PLASSON N , GILLIS G , et al . Don’t stack your log on my log [C ] // The 2nd Workshop on Interactions of NVM/Flash with Operating Systems and Workloads,October 5,2014,Broomfield,USA.[S.l.:s.n . ] , 2014 .

IZRAELEVITZ J , YANG J , ZHANG L , et al . Basic performance measurements of the Intel Optane DC persistent memory module [J ] . Computer Science , 2019 ,arXiv:1903.05714.

SWANSON S . Redrawing the boundary between software and storage for fast non-volatile memories [R ] . San Diego:University of California , 2015 .

CONDIT J , NIGHTINGALE E B , FROST C , et al . Better I/O through byteaddressable,persistent memory [C ] // The ACM SIGOPS 22nd Symposium on Operating Systems Principles,October 11-14,2009,Big Sky,USA . New York:ACM Press , 2009 : 133 - 146 .

NARAYANAN D , HODSON O . Wholesystem persistence [J ] . ACM SIGARCH Computer Architecture News , 2012 , 40 ( 1 ): 401 - 410 .

LU Y Y , SHU J W , SUN L . Blurred persistence in transactional persistent memory [C ] // The 31st Symposium on Mass Storage Systems and Technologies (MSST),May 30-June 5,2015,Santa Clara,USA . Piscataway:IEEE Press , 2015 : 1 - 13 .

DULLOOR S R , KUMAR S , KESHAVAMURTHY A , et al . System software for persistent memory [C ] // The 9th European Conference on Computer Systems,April 14-16,2014,Amsterdam,the Netherlands . New York:ACM Press , 2014 :15.

XU J , SWANSON S . NOVA:a logstructured file system for hybrid volatile/non-volatile main memories [C ] // The 14th USENIX Conference on File and Storage Technologies,February 22-25,2016,Santa Clara,USA . Berkeley:USENIX Association , 2016 : 323 - 338 .

VOLOS H , NALLI S , PANNEERSELVAM S , et al . Aerie:flexible file-system interfaces to storage-class memory [C ] // The 9th European Conference on Computer Systems,April 14-16,2014,Amsterdam,the Netherlands . New York:ACM Press , 2014 :14.

KWON Y , FINGLER H , HUNT T , et al . Strata:a cross media file system [C ] // The 26th Symposium on Operating Systems Principles,October 28-31,2017,Shanghai,China . New York:ACM Press , 2017 : 460 - 477 .

陈游旻 , 陆游游 , 罗圣美 , 等 . 基于RDMA的分布式存储系统研究综述 [J ] . 计算机研究与发展 , 2019 , 56 ( 2 ): 227 - 239 .

CHEN Y M , LU Y Y , LUO S M , et al . Survey on RDMA-based distributed storage systems [J ] . Journal of Computer Research and Development , 2019 , 56 ( 2 ): 227 - 239 .

LU Y Y , SHU J W , CHEN Y M , et al . Octopus:an RDMA-enabled distributed persistent memory file system [C ] // 2017 USENIX Annual Technical Conference (USENIX ATC 17),July 12-14,2017,Santa Clara,USA . Berkeley:USENIX Association , 2017 : 773 - 785 .

Views

1043

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Challenges and technical development of storage systems in the era of large language models

A polymorphic cooperative compression strategy for IoT time series data based on NVM

A survey of persistent index data structures on non-volatile memory

An optimization of MPI-IO interface for non-volatile memory

Related Author

FENG Yangyang

WANG Qing

SHU Jiwu

Tao CAI

Tianle LEI

Dejiao NIU

Jianfei DAI

Zeyu HUANG

Related Institution

Department of Computer Science and Technology, Tsinghua University

Postal code：100079
Tel：（010）53879208、53859533 Email：bdr@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：273380 Visits Today：286

⁰