基于新型存储的大数据存储管理

金培权

doi:10.11959/j.issn.2096-0271.2017053

您当前的位置：

首页 >

文章列表页 >

基于新型存储的大数据存储管理

研究 | 更新时间：2024-06-03

- 基于新型存储的大数据存储管理
- Big data storage management based on new storage
- 大数据 2017年3卷第5期页码：2017053
- 作者机构：
  
  1. 中国科学技术大学计算机科学与技术学院，安徽合肥 230027
  2. 中国科学院电磁空间信息重点实验室，安徽合肥 230027
- 作者简介：
  
  [ "金培权（1975-），男，博士，中国科学技术大学计算机科学与技术学院和中国科学院电磁空间信息重点实验室副教授、硕士生导师，目前主要从事大数据与数据库领域的研究工作，近年来主持了20余项科研项目，包括5项国家自然科学基金项目和2项“863”计划项目，在VLDB Journal、TKDE、TPDS、ICDE、WWW等本领域著名期刊和会议上发表论文20余篇，曾获中国科学院院长奖、NPC 2014最佳论文奖、DASFAA 2015最佳海报奖、NDBC 2012最佳论文提名奖以及NDBC 2011最佳系统演示奖。" ]
- 基金信息：
  
  国家自然科学基金资助项目;The National Natural Science Foundation of China(61672479)
- DOI：10.11959/j.issn.2096-0271.2017053
  中图分类号： TP311
- 网络首发：2017-09，
  
  纸质出版：2017-09-20
- 稿件说明：
移动端阅览
金培权. 基于新型存储的大数据存储管理[J]. 大数据, 2017,3(5):2017053.

Peiquan JIN. Big data storage management based on new storage[J]. Big data research, 2017, 3(5): 2017053.
金培权. 基于新型存储的大数据存储管理[J]. 大数据, 2017,3(5):2017053. DOI： 10.11959/j.issn.2096-0271.2017053.

Peiquan JIN. Big data storage management based on new storage[J]. Big data research, 2017, 3(5): 2017053. DOI： 10.11959/j.issn.2096-0271.2017053.

摘要

如何高效地存储大数据并支持实时大数据处理与分析是大数据技术发展面临的首要问题。近年来，以相变存储器、闪存等为代表的新型存储为实现高效的大数据存储和管理提供了新思路。以相变存储器为代表的存储级主存技术为切入点，针对大数据存储与管理中的高效存储、实时处理等存在的挑战，讨论了面向新型存储的大数据存储管理研究现状，并对未来基于新型存储的大数据研究进行了展望。

Abstract

How to efficiently store big data and support real-time big data processing and analysis has been the most critical issue in the development of big data technologies.Recently

new storage media such as phase change memory and flash memory provides new opportunities for developing an efficient framework for big data storage and management.Based on the challenges of efficient storage and real-time processing in big data storage and management

storage class memories were focused

which were represented by phase change memory

and the state of the art of new-storage-based big data storage management was discussed.Finally

some future research directions for new-storage-based big data storage management were proposed.

关键词

Keywords

references

孟小峰 , 慈祥 . 大数据管理：概念、技术与挑战 [J ] . 计算机研究与发展 , 2013 , 50 ( 1 ): 146 - 169 .

MENG X F , CI X . Big data management:concepts,techniques and challenges [J ] . Journal of Computer Research and Development , 2013 , 50 ( 1 ): 146 - 169 .

LU C , SHI J P , JIA J Y . Abnormal event detection at 150 FPS in MATLAB [C ] // 2013 IEEE International Conference on Computer Vision(ICCV),December1-8,2013,Sydney,Australia . New Jersey:IEEE Press , 2013 : 2720 - 2727 .

CHENG M M , ZHANG Z M , LIN W Y , et al . BING:binarized normed gradients for objectness estimation at 300fps [C ] // 2014 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),June 23-28,2014,Columbus,USA . New Jersey:IEEE Press , 2014 : 3286 - 3293 .

YAN J J , LEI Z , WEN L Y , et al . The fastest deformable part model for objectdetection [C ] // 2014 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),June 23-28,2014,Columbus,USA . New Jersey:IEEE Press , 2014 : 2497 - 2504

ROSS B , . Girshick:fast R-CNN [C ] // International Conference on Computer Vision(ICCV),December 13-16,2015,Santiago,Chile.[S.l.:s.n.] , 2015 : 1440 - 1448

申德荣 , 于戈 , 王习特 , 等 . 支持大数据管理的NoSQL系统研究综述 [J ] . 软件学报 , 2013 , 24 ( 8 ): 1786 - 1803 .

SHEN D R , YU G , WANG X T , et al . Survey on NoSQL for management of big data [J ] . Journal of Software , 2013 , 24 ( 8 ): 1786 - 1803 .

哈索·普拉特纳 , 亚历山大·蔡尔 . 内存数据管理（第2版） [M ] . SAP,译.北京:清华大学出版社 , 2012 .

PLATTNER H , ZEIER A . In-memory data management(2nd edition) [M ] . Translated by SAP . Beijing : Tsinghua University PressPress , 2012 .

王江涛 , 赖文豫 , 孟小峰 . 闪存数据库：现状、技术与展望 [J ] . 计算机学报 , 2013 , 36 ( 8 ): 1549 - 1567 .

WANG W T , LAI W Y , MENG X F . Flashbased database:studies,techniques and forecasts [J ] . Chinese Journal of Computers , 2013 , 36 ( 8 ): 1549 - 1567 .

吴章玲 , 金培权 , 岳丽华 , 等 . 基于PCM的大数据存储与管理研究综述 [J ] . 计算机研究与发展 , 2015 , 52 ( 2 ): 343 - 361 .

WU Z L , JIN P Q , YUE L H , et al . A survey on PCM-based big data storage and management [J ] . Journal of Computer Research and Development , 2015 , 52 ( 2 ): 343 - 361 .

FREITAS R F , WILCKE W W . Storageclass memory:the next storage system technology [J ] . IBM Journal of Research and Development , 2008 , 52 ( 4 ): 439 - 447

BURR G W , KURDI B N , SCOTT J C , et al . Overview of candidate device technologies for storage-class memory [J ] . IBM Journal of Research and Development , 2008 , 52 ( 4 ): 449 - 464 .

HAO X J , JIN P Q , YUE L H . Efficient storage of multi-sensor object-tracking data [J ] . IEEE Transactions on Parallel and Distributed Systems , 2016 , 27 ( 10 ): 2881 - 2894 .

HARTER T , BORTHAKUR D , DONG S Y , et al . Analysis of HDFS under HBase:a Facebook messages case study [C ] // The 12th USENIX Conference on File and Storage Technologies(FAST),February 17-20,2014,Santa Clara,USA . Berkeley:USENIX Association Berkeley , 2014 : 199 - 212 .

O'NEIL P E , CHENG E , GAWLICK D , et al . The log-structured merge-tree (LSMTree) [J ] . Acta Informatica , 1996 , 33 ( 4 ): 351 - 385 .

OUSTERHOUT J K , GOPALAN A , GUPTA A , et al . The RAMCloud storage system [J ] . ACM Transactions on Computer Systems , 2015 , 33 ( 3 ):7.

金培权 , 郝行军 , 岳丽华 . 面向新型存储的大数据存储架构与核心算法综述 [J ] . 计算机工程与科学 , 2013 , 35 ( 10 ): 12 - 24 .

JIN P Q , HAO X J , YUE L H . A survey on storage architectures and core algorithms for big data management on new storages [J ] . Computer Engineering and Science , 2013 , 35 ( 10 ): 12 - 24 .

张鸿斌 , 范捷 , 舒继武 , 等 . 基于相变存储器的存储系统与技术综述 [J ] . 计算机研究与发展 , 2014 , 51 ( 8 ): 1647 - 1662 .

ZHANG H B , FAN J , SHU J W , et al . Summary of storage system and technology based on phase change memory [J ] . Journal of Computer Research and Development , 2014 , 51 ( 8 ): 1647 - 1662 .

VIGLAS S D . Write-limited sorts and joins for persistent memory [J ] . Proceedings of the VLDB Endowment , 2014 , 7 ( 5 ): 413 - 424 .

XIA F , JIANG D J , XIONG J , et al . A survey of phase change memory systems [J ] . Journal of Computer Science and Technology , 2015 , 30 ( 1 ): 121 - 144 .

冒伟 , 刘景宁 , 童薇 , 等 . 基于相变存储器的存储技术研究综述 [J ] . 计算机学报 , 2015 , 38 ( 5 ): 944 - 960 .

MAO W , LIU J N , TONG W , et al . A review of storage technology research based on phase change memory [J ] . Chinese Journal of Computers , 2015 , 38 ( 5 ): 944 - 960 .

陈小刚 , 许林海 , 陈一峰 , 等 . 一种非易失性随机存储器及其操作方法:CN201110202834.7 [P ] .2013-01-23.

CHEN X G , XU L H , CHEN Y F , et al . A non-volatile random memory and its operations:CN201110202834.7 [P ] .201301-23.

缪向水 , 余念念 , 童浩 , 等 . 相变存储器:CN105247677A [J ] .2015-11-05.-- >

MIAO X S , YU N N , DONG H , et al . Phase change memory:CN105247677A [J ] .2015-11-05.

QURESHI M K , SRINIVASAN V , RIVERS J A . Scalable high performance main memory system using phase-change memory technology [C ] // The 36th Annual International Symposium on Computer Architecture(ISCA),June 20-24,2009,Austin,USA . New York:ACM Press , 2009 : 24 - 33 .

QURESHI M K , KARIDIS J P , FRANCESCHINI M , et al . Enhancing lifetime and security of PCM-based main memory with start-gap wear leveling [C ] // The 42nd Annual IEEE/ACM International Symposium on Microarchitecture(MICRO),December 12-16,2009,New York,USA . New Jersey:IEEE Press , 2009 : 14 - 23 .

RAMOS L E , BIANCHINI R . Exploiting phase-change memory in cooperative caches [C ] // 2012 IEEE 24th International Symposium on Computer Architecture and High Performance Computing(SBACPAD),October 24-26,2012,New York,USA . New Jersey:IEEE Press , 2012 : 227 - 234 .

BADAM A , PAI V S . SSDAlloc:hybrid SSD/RAM memory management made easy [C ] // The 8th USENIX Conference on Networked Systems Design and Implementation(NSDI),March 30-April 1,2011,Boston,USA . New York:ACM Press , 2011 : 16 - 30 .

韩旭 , 曹巍 , 孟小峰 . 使用固态硬盘管理主存KV数据库的虚拟内存 [J ] . 计算机科学与探索 , 2011 , 5 ( 8 ): 686 - 694 .

HAN X , CAO W , MENG X F . Virtual memory management for main-memory KV database using solid state disk [J ] . Journal of Frontiers of Computer Science＆ Technology , 2011 , 5 ( 8 ): 686 - 694 .

OUYANG X Y , ISLAMN S , RAJACHANDRASEKAR R , et al . SSDassisted hybrid memory to accelerate memcached over high performance networks [C ] // The 41st International Conference on Parallel Processing(ICPP),September 10-13,2012,Pittsburgh,USA . New Jersey:IEEE Press , 2012 : 470 - 479 .

陈卓 , 熊劲 , 马灿 , 等 . 基于SSD的机群文件系统元数据存储系统 [J ] . 计算机研究与发展 , 2012 , 49 ( z1 ): 269 - 275 .

CHEN Z , XIONG J , MA C , et al . SSDbased metadata storage system for cluster file system [J ] . Journal of Computer Research and Development , 2012 , 49 ( z1 ): 269 - 275 .

RAMOS L E , GORBATOV E , BIANCHINI R . Page placement in hybrid memory systems [C ] // The International Conference on Supercomputing(ICS),May 31-June 4,2011,Tucson,USA . New York:ACM Press , 2011 : 85 - 95 .

WU Z L , JIN P Q , YUE L H . Efficient space management and wear leveling for pcm-based storage systems [C ] // The 15th International Conference on Algorithms and Architectures for Parallel Processing(ICA3PP),November 18-20,2015,Zhangjiajie,China . Berlin:Springer , 2015 : 784 - 798 .

WU Z L , JIN P Q , YANG C C , et al . Efficient memory management for NVM-based hybrid memory systems [J ] . International Journal of Control and Automation , 2016 , 9 ( 1 ): 445 - 458 .

WU S , JIANG D W , OOI B C , et al . Efficient B-tree based indexing for cloud data processing [J ] . Proceedings of the VLDB Endowment , 2010 , 3 ( 1 ): 1207 - 1218 .

WANG J B , WU S , GAO H , et al . Indexing multi-dimensional data in a cloud system [C ] // The 2010 ACM SIGMOD International Conference on Management of Data(SIGMOD),June 6-10,2010,Indianapolis,USA . New York:ACM Press , 2010 : 591 - 602 .

DING L L , QIAO B Y , WANG G R , et al . An efficient quad-tree based index structure for cloud data management [C ] // International Conference on WebAge Information Management (WAIM),September 14-16,2011,Wuhan,China . Berlin:Springer Berlin Heidelberg , 2011 : 238 - 250 .

ZOU Y Q , LIU J , WANG S C , et al . CCIndex:a complemental clustering index on distributed ordered tables for multidimensional range queries [C ] // IFIP International Conference on Network and Parallel Computing(NPC),April 28,2011,Changsha,China . Berlin:Springer , 2010 : 247 - 261 .

NISHIMURA S , DAS S , AGRAWAL D , et al . MD-HBase:a scalable multidimensional data infrastructure for location aware services [C ] // The 12th IEEE International Conference on Mobile Data Management(MDM),June 6-9,2011,Lulea,Sweden . New Jersey:IEEE Press , 2011 : 7 - 16

MA Y Z , RAO J , HU W S , et al . An efficient index for massive IOT data in cloud environment [C ] // The 21st ACM International Conference on Information and Knowledge Management(CIKM),October 29-November 2,2012,Maui,USA . New York:ACM Press , 2012 : 2129 - 2133 .

AHN J S , KANG D W , JUNG D , et al . μ*-Tree:an ordered index structure for NAND flash memory with adaptive page layout scheme [J ] . IEEE Transactions on Computers , 2013 , 62 ( 4 ): 784 - 797 .

ATHANASSOULIS M , AILAMAKI A . BF-Tree:approximate tree indexing [J ] . Proceedings of the VLDB Endowment , 2014 , 7 ( 14 ): 1881 - 1892 .

ON S T , HU H B , LI Y , et al . Flashoptimized B+-tree [J ] . Journal of Computer Science and Technology (JCST) , 2010 , 25 ( 3 ): 509 - 522 .

CUI K , JIN P Q , YUE L H . HashTree:a new hybrid index for flash disks [C ] // The 12th International Asia-Pacific Web Conference(APWeb),April 6-8,2010,Busan,Korea . New Jersey:IEEE Press , 2010 : 45 - 51 .

JIN P Q , YANG C C , JENSEN C S , et al . Read/Write-optimized tree indexing for solid state drives [J ] . VLDB Journal , 2016 , 25 ( 5 ): 695 - 717 .

CHEN S M , GIBBONS P B . SumanNath:rethinking database algorithms for phase change memory [C ] // The 5th Biennial Conference on Innovative Data Systems Research(CIDR),January 9-12,2011,Asilomar,USA.[S.l.:s.n . ] , 2011 : 21 - 31 .

CHEN S M , JIN Q . Persistent B+-Trees in non-volatile main memory [J ] . Proceedings of the VLDB Endowment , 2015 , 8 ( 7 ): 786 - 797 .

HU W W , LI G L , NI J C , et al . Bptree:a predictive B+-tree for reducing writes on phase change memory [J ] . IEEE Transactions on Knowledge and Data Engineering , 2014 , 26 ( 10 ): 2368 - 2381 .

CHI P , LEE W C , XIE Y . Making B+-tree efficient in PCM-based main memory [C ] // International Symposium on Low Power Electronics and Design(ISLPED),August 11-13,2014,La Jolla,USA.[S.l.:s.n . ] , 2014 : 69 - 74 .

LI L , JIN P Q , YANG C C , et al . Optimizing B+-Tree for PCM-based hybrid memory [C ] // The 19th International Conference on Extending Database Technology(EDBT),March 15-18,2016,Bordeaux,France.[S.l.:s.n . ] , 2016 : 662 - 663 .

ZHANG D Z , JIN P Q , WANG X L , et al . DPHSim:a flexible simulator for DRAM/PCM-based hybrid memory [C ] // AsiaPacific Web (APWeb) and Web-Age Information Management (WAIM) JointConference on Web and Big Data(APWeb/WAIM),July 7-9,2017,Beijing,China . Berlin:Springer , 2017 : 319 - 323 .

浏览量

1039

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

大数据技术前瞻

面向大数据应用的混合内存架构特征分析