GVDS: a global virtual data space for wide-area high-performance computing environments

Limin XIAO; Yao SONG; Guangjun QIN; Hanjie ZHOU; Chaobo WANG; Bing WEI; Wei WEI; Zhisheng HUO

doi:10.11959/j.issn.2096-0271.2021017

您当前的位置：

首页 >

文章列表页 >

GVDS: a global virtual data space for wide-area high-performance computing environments

TOPIC：VIRTUAL DATA SPACE FOR HIGH-PERFORMANCE COMPUTING | 更新时间：2024-06-03

- GVDS: a global virtual data space for wide-area high-performance computing environments
- Big data research Vol. 7, Issue 2, Pages: 2021017(2021)
- 作者机构：
  
  1. 北京航空航天大学计算机学院，北京 100191
  2. 软件开发环境国家重点实验室，北京 100191
  3. 北京联合大学智慧城市学院，北京 100101
  4. 西安理工大学计算机科学与工程学院，陕西西安 710048
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-0271.2021017
  CLC： TP316
- Online First：2021-03，
  
  Published：15 March 2021
- 稿件说明：
移动端阅览
Limin XIAO, Yao SONG, Guangjun QIN, et al. GVDS: a global virtual data space for wide-area high-performance computing environments[J]. Big data research, 2021, 7(2): 2021017.
DOI：

Limin XIAO, Yao SONG, Guangjun QIN, et al. GVDS: a global virtual data space for wide-area high-performance computing environments[J]. Big data research, 2021, 7(2): 2021017. DOI： 10.11959/j.issn.2096-0271.2021017.

摘要

广域高性能计算环境是支撑科技创新和社会经济发展的核心信息基础设施。然而，在广域高性能计算环境中，异构存储资源在地理位置上的分散分布导致无法发挥广域存储资源的聚合效应，难以满足对广域分布数据的统一管理和高效访问需求。为此，提出了虚拟数据空间构建方法及数据访问性能优化方法，并实现了一个面向广域高性能计算环境的全局虚拟数据空间（GVDS）。GVDS可聚合广域分布的异构存储资源，形成统一的虚拟数据空间，有效支撑用户以统一访问模式高效访问广域分散的异构存储资源，实现广域环境中分布数据的跨域共享和协同处理。测试结果表明，与国际领先的面向广域高性能计算环境的OneData、GFFS等存储系统相比，GVDS实现了相当的功能，且数据访问性能明显提升。

Abstract

The wide-area high-performance computing environment is the core information infrastructure to support technology innovation

economic development

and national defense.However

heterogeneous storage resources are geographically distributed in wide-area high-performance computing environments

resulting in the barriers between applications and data.The requirements of unified data management and efficient data access cannot be met.A method of establishing virtual data space and a data access optimization method was presented

and a global virtual data space (GVDS) for wide-area high-performance computing environments to satisfy the requirements was implemented.GVDS aggregates geographically distributed and heterogeneous storage resources

creating a unified virtual data space to provide unified and efficient data access.Sharing and collaborative processing of geographically distributed data were achieved in widearea environments.The experimental results indicate that compared with the state-of-the-art wide-area storage system in the field of high-performance computing

such as OneData and GFFS

GVDS has similar functions and improves the read bandwidth significantly.

关键词

Keywords

references

SLOTA R , DUTKA L , KRYZA B , et al . Storage systems for organizationally distributed environments-PLGrid plus case study [C ] // International Conference on Parallel Processing and Applied Mathematics . Heidelberg:Springer , 2014 : 724 - 733 .

XIE X , XIAO N , XU Z W , et al . CNGrid software 2:service oriented approach to grid computing [C ] // The UK e-Science All Hands Meeting .[S.l.:s.n. ] , 2005 : 701 - 708 .

DEPEI Q , . CNGrid:a test-bed for grid technologies in China [C ] // The 10th IEEE International Workshop on Future Trends of Distributed Computing Systems . Piscataway:IEEE Press , 2004 : 135 - 139 .

Cluster File Systems,Inc . Lustre:a scalable,high-performance file system [Z ] . 2002 .

WEIL S A , BRANDT S A , MILLER E L , et al . Ceph:a scalable,high-performance distributed file system [C ] // The 7th Symposium on Operating Systems Design and Implementation . New York:ACM Press , 2006 : 307 - 320 .

SCHMUCK F B , HASKIN R L . GPFS:a shared-disk file system for large computing clusters [C ] // The Conference on File and Storage Technologies . New York:ACM Press , 2002 : 231 - 244 .

DABEK F , KAASHOEK M F , KARGER D , et al . Wide-area cooperative storage with CFS [C ] // The 18th ACM Symposium on Operating Systems Principles . New York:ACM Press , 2001 : 202 - 215 .

DABEK F , BRUNSKILL E , KAASHOEK M F , et al . Building peer-to-peer systems with chord,a distributed lookup service [C ] // The 8th Workshop on Hot Topics in Operating Systems . Piscataway:IEEE Press , 2001 : 81 - 86 .

GRAFFI K , GROSS C , STINGL D , et al . Lifesocial.KOM:a secure and p2p-based solution for online social networks [C ] // 2011 IEEE Consumer Communications and Networking Conference . Piscataway:IEEE Press , 2011 : 554 - 558 .

CHAN Y W , HO T H , SHIH P C , et al . Malugo:a peer-to-peer storage system [J ] . International Journal of Ad Hoc and Ubiquitous Computing , 2010 , 5 ( 4 ): 209 .

TOKA L , DELL’AMICO M , MICHIARDI P . Data transfer scheduling for P2P storage [C ] // 2011 IEEE International Conference on Peer-to-Peer Computing . Piscataway:IEEE Press , 2011 : 132 - 141 .

SHEN H Y , LI Z , LI J . A DHT-aided chunkdriven overlay for scalable and efficient peer-to-peer live streaming [J ] . IEEE Transactions on Parallel and Distributed Systems , 2012 , 24 ( 11 ): 2125 - 2137 .

KARGER D , LEHMAN E , LEIGHTON T , et al . Consistent hashing and random trees:distributed caching protocols for relieving hot spots on the World Wide Web [C ] // The 29th Annual ACM Symposium on Theory of Computing . New York:ACM Press , 1997 : 65 - 663 .

CALDER B , WANG J , OGUS A , et al . Windows Azure storage:a highly available cloud storage service with strong consistency [C ] // The 23rd ACM Symposium on Operating Systems Principles . New York:ACM Press , 2011 : 143 - 157 .

SIMMS C K , PIKE G G , BALOG D . Wide area filesystem performance using Lustre on the TeraGrid [R ] . 2007 .

TATEBE O , HIRAGA K , SODA N . Gfarm grid file system [J ] . New Generation Computing , 2010 , 28 : 257 - 275 .

THOMSON A , ABADI D J . CalvinFS:consistent WAN replication and scalable metadata management for distributed file systems [C ] // The 13th USENIX Conference on File and Storage Technologies . New York:ACM Press , 2015 : 1 - 14 .

WRZESZCZ M , TRZEPLA K , SOTA R , et al . Metadata organization and management for globalization of data access with OneData [C ] // International Conference on Parallel Processing and Applied Mathematics . Cham:Springer , 2015 : 312 - 321 .

DUTKA U , SOTA R , WRZESZCZ M , et al . Uniform and efficient access to data in organizationally distributed environments [M ] // eScience on Distributed Computing Infrastructure . Cham : Springer , 2014 : 178 - 194 .

GRIMSHAW A , MORGAN M , KALYANARAMAN A . GFFS-the XSEDE global federated file system [J ] . Parallel Processing Letters , 2013 , 23 ( 2 ): 1340005 .

胡正丁 , 薛巍 . 面向异构众核超级计算机的大规模稀疏计算性能优化研究 [J ] . 大数据 , 2020 , 6 ( 4 ): 40 - 55 .

HU Z D , XUE W . Research on performance optimization for largescale sparse computation over many-core heterogenous supercomputer [J ] . Big Data Research , 2020 , 6 ( 4 ): 40 - 55 .

KUNSZT P , BADINO P , FROHNER A , et al . Data storage,access and catalogs in gLite [C ] // 2005 IEEE International Symposium on Mass Storage Systems and Technology . Piscataway:IEEE Press , 2005 : 166 - 170 .

FOSTER I , CZAJKOWSKI K , FERGUSON D E , et al . Modeling and managing state in distributed systems:The role of OGSI and WSRF [J ] . Proceedings of the IEEE , 2005 , 93 ( 3 ): 604 - 612 .

CHERVENAK A , FOSTER I , KESSELMAN C , et al . The data grid:towards an architecture for the distributed management and analysis of large scientific datasets [J ] . Journal of Network and Computer Applications , 2000 , 23 ( 3 ): 187 - 200 .

ALLCOCK W , BRESNAHAN J , BESTER J . GridFTP protocol specification [Z ] . 2002 .

FITZGERALD S , FOSTER I , KESSELMAN C , et al . A directory service for configuring high-performance distributed computations [C ] // The 6th IEEE International Symposium on High Performance Distributed Computing . Piscataway:IEEE Press , 1997 : 365 - 375 .

CHEN M , BANGERA G B , HILDEBRAND D , et al . vNFS:maximizing NFS performance with compounds and vectorized I/O [J ] . ACM Transactions on Storage , 2017 , 13 ( 3 ): 1 - 24 .

RYU J , LEE D , SHIN K G , et al . ClusterFetch:a lightweight prefetcher for intensive disk reads [J ] . IEEE Transactions on Computers , 2017 , 67 ( 2 ): 284 - 290 .

WU Z , BUTKIEWICZ M , PERKINS D , et al . SPANStore:cost-effective georeplicated storage spanning multiple cloud services [C ] // The 24th ACM Symposium on Operating Systems Principles . New York:ACM Press , 2013 : 292 - 308 .

SHAO Y , LI C , TANG H . A data replica placement strategy for IoT workflows in collaborative edge and cloud environm ents [J ] . Computer Networks , 2019 , 148 : 46 - 59 .

CHANG W C , WANG P C . Write-aware replica placement for cloud computing [J ] . IEEE Journal on Selected Areas in Commun ications , 2019 , 37 ( 3 ): 656 - 667 .

The reference implementation of the linux FUSE (filesystem in userspace) interface [Z ] . 2019 .

CHOPADE R , DHAVASE N S . MongoDB,Couchbase:performance comparison for image dataset [C ] // 2017 2nd International Conference for Convergence in Technology . Piscataway:IEEE Press , 2017 : 255 - 258 .

A network filesystem client to connect to SSH servers [Z ] . 2019 .

SHEPLER S , NOVECK D , EISLER M . Network file system (NFS) version 4 minor version 1 protocol [Z ] . 2010 .

Fio-flexible I/O tester rev.3.16 [Z ] . 2019 .

Views

754

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Cross-domain data management for computing power networks

Related Author

Weizheng LU

Qizhi DAI

Ce ZHANG

Related Institution

Office of Research Infrastructure, Renmin University of China

Lianjingzhineng Technology (Shanghai) Co., Ltd.

Network and Computing Center, Huazhong University of Science ＆Technology

Postal code：100079
Tel：（010）53879208、53859533 Email：bdr@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：273198 Visits Today：104

⁰