Cross-domain data management for computing power networks

Weizheng LU; Qizhi DAI; Ce ZHANG

doi:10.11959/j.issn.2096-0271.2023068

您当前的位置：

首页 >

文章列表页 >

Cross-domain data management for computing power networks

STUDY | 更新时间：2024-06-03

- Cross-domain data management for computing power networks
- Big data research Vol. 10, Issue 2, Pages: 94-108(2024)
- 作者机构：
  
  1. 中国人民大学大型科学仪器共享平台，北京 100083
  2. 联旌智能科技（上海）有限公司，上海 200051
  3. 华中科技大学网络与计算中心，湖北武汉 430074
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-0271.2023068
  CLC： TP316
- Online First：2024-03，
  
  Published：15 March 2024
- 稿件说明：
移动端阅览
Weizheng LU, Qizhi DAI, Ce ZHANG. Cross-domain data management for computing power networks[J]. Big data research, 2024, 10(2): 94-108.
DOI：

Weizheng LU, Qizhi DAI, Ce ZHANG. Cross-domain data management for computing power networks[J]. Big data research, 2024, 10(2): 94-108. DOI： 10.11959/j.issn.2096-0271.2023068.

摘要

跨域算力网络希望整合多个算力中心的计算和数据资源，但现有的方案对跨域文件和数据管理关注不够。提出了一种轻量级的跨域算力网络数据管理方案：通过文件系统协议转换，接入远程算力中心的并行文件系统存储资源；算力中心内部的存储资源作为一种补充，应对高IOPS应用；通过容器绑定技术，将远程存储挂载并绑定到指定目录。基于该方案的原型系统已经在高校校级计算平台部署运行。实测数据和用户体验显示，该方案能够满足常见高性能计算应用需求。

Abstract

Cross-domain computing power networks wish to integrate computational and data resources from multiple computing centers

but existing methods do not pay enough attention to cross-domain file and data management.In this paper

a lightweight data access scheme for cross-domain computing power networks was proposed: (1) accessing parallel file system storage resources of remote computing centers through file system protocol conversion; (2) local caching as a supplement to cope with high IOPS applications; and (3) mounting remote or local storage to specified directories through container binding technology.The prototype system based on this scheme had been deployed on highperformance computing centers in multiple universities.The measured data and user experience showed that the scheme in this paper could meet the requirements of common high-performance computing applications.

关键词

Keywords

references

FOSTER I T , KESSELMAN C . The grid:blueprint for a new computing infrastructure [M ] . San Francisco : Morgan Kaufman Publishers , 1998 .

XU Z W , CHI X B , XIAO N . Highperformance computing environment:a review of twenty years of experiments in China [J ] . National Science Review , 2016 , 3 ( 1 ): 36 - 48 .

钱德沛 , 栾钟治 , 刘轶 . 从网格到“东数西算”：构建国家算力基础设施 [J ] . 北京航空航天大学学报 , 2022 , 48 ( 9 ): 1561 - 1574 .

QIAN D P , LUAN Z Z , LIU Y . From grid to “East-west Computing Transfer”:constructing national computing infrastructure [J ] . Journal of Beijing University of Aeronautics and Astronautics , 2022 , 48 ( 9 ): 1561 - 1574 .

秦广军 , 肖利民 , 张广艳 , 等 . 面向国家高性能计算环境的虚拟数据空间系统 [J ] . 大数据 , 2021 , 7 ( 2 ): 101 - 122 .

QIN G J , XIAO L M , ZHANG G Y , et al . Virtual data space system for national high-performance computing environment [J ] . Big Data Research , 2021 , 7 ( 2 ): 101 - 122 .

何小雨 , 邓笋根 , 栾海晶 , 等 . 国家高性能计算环境的虚拟数据空间运行支撑技术研究 [J ] . 大数据 , 2021 , 7 ( 2 ): 158 - 171 .

HE X Y , DENG S G , LUAN H J , et al . Study of technique support on the operation of virtual data space in national highperformance computing environment [J ] . Big Data Research , 2021 , 7 ( 2 ): 158 - 171 .

肖利民 , 宋尧 , 秦广军 , 等 . GVDS：面向广域高性能计算环境的虚拟数据空间 [J ] . 大数据 , 2021 , 7 ( 2 ): 123 - 146 .

XIAO L M , SONG Y , QING J , et al . GVDS:a global virtual data space for wide-area high-performance computing environments [J ] . Big Data Research , 2021 , 7 ( 2 ): 123 - 146 .

YOO A B , JETTE M A , GRONDONA M . SLURM:simple linux utility for resource management [C ] // Proceedings of Job Scheduling Strategies for Parallel Processing . Seattle:Springer , 2003 : 44 - 60 .

YIN F , SHI F . A comparative survey of big data computing and hpc:from a parallel programming model to a cluster architecture [J ] . International Journal of Parallel Programming , 2022 , 50 ( 1 ): 27 - 64 .

BURROWS M . Lustre:building a file system for 1000-node clusters [C ] // Proceedings of the 2003 Linux Symposium .[S.l.:s.n. ] , 2003 : 380 - 386 .

SCHMUCK F B , HASKIN R L . GPFS:a shared-disk file system for largecomputing clusters [C ] // The FAST 2002 Conferenceon File and Storage Technologies . Monterey:USENIX , 2002 : 231 - 244 .

CATLETT C , ALLCOCK W E , ANDREWS P , et al . TeraGrid:analysis of organization,system architecture,and middleware enabling new types of applications [C ] // Proceedings of the 2006 International Advanced Research Workshop on High Performance Computing and Grids . Amsterdam:IOS Press , 2006 : 225 - 249 .

TOWN J , BOISSEAU J , ROSKIES J , et al . XSEDE:extreme science and engineering discovery environment (OAC 15-48562) [R ] . 2020 .

NEWHOUSE S . Seeking new horizons:EGI’s role in 2020 (EGI-1098-D230-V3) [R ] . 2021 .

HENSCHEL R , SIMMS S , HANCOCK D , et al . Demonstrating lustre over a 100Gbps wide area network of 3500km [C ] // Proceedings ofthe International Conference on High Performance Computing,Networking,Storage and Analysis . Piscataway:IEEE Press , 2012 : 1 - 8 .

THOMSON A , ABADI D J . CalvinFS:consistent WAN replication and scalable metadata management for distributed file systems [C ] // Proceedings of the 13th USENIX Conference on File and Storage Technologies . Berkeley:USENIX , 2015 : 1 - 14 .

CORBETT J C , DEAN J , EPSTEIN M , et al . Spanner:Google’s globally-distributed database [J ] . ACM Transactions on Computer Systems , 2013 , 31 ( 3 ): 8 .

CAO W , LIU Z J , WANG P , et al . PolarFS:an ultra-low latency and failure resilient distributed file system for shared storage cloud database [J ] . Proceedings of the VLDB Endowment , 2018 , 11 ( 12 ): 1849 - 1862 .

A network filesystem client to connect to SSH servers [Z ] . 2023 .

DUTKA U , SOTA R , WRZESZCZ M , et al . Uniform and efficient access to data in organizationally distributed environments [C ] // Proceeding of eScience on Distributed Computing Infrastructure . Cham:Springer , 2014 : 178 - 194 .

DONENFELD J A . Wireguard:next generation kernel network tunnel [C ] // Proceeding of 24th Annual Network and Distributed System Security Symposium . San Diego:The Internet Society , 2017 : 1 - 12 .

DONGARRA J , LUSZCZEK P , PETITET A . The LINPACK benchmark:past,present,and future [J ] . Concurrency and Computation:Practice and Experience , 2003 , 15 ( 9 ): 803 - 820 .

Fio - flexible I/O tester rev.3.25 [Z ] . 2020 .

Views

296

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Challenges and technical development of storage systems in the era of large language models

Cloud-edge-end collaborative big data management for metaverse

Classification of big data in metrology

HEPS scientific data management policy research and applications

GVDS: a global virtual data space for wide-area high-performance computing environments

Related Author

SHU Jiwu

WANG Qing

FENG Yangyang

Rui ZHU

Hongzhi WANG

Shuangshuang CUI

Kaixin ZHANG

Yu YAN

Related Institution

Department of Computer Science and Technology, Tsinghua University

Institute of High Energy Physics, Chinese Academy of Sciences

National High Energy Physics Science Data Center

University of Science and Technology of China

School of Computer Science and Engineering, Beihang University

Postal code：100079
Tel：（010）53879208、53859533 Email：bdr@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：273110 Visits Today：16

⁰