面向算力网络的跨域数据管理方法

鲁蔚征; 戴奇志; 张策

doi:10.11959/j.issn.2096-0271.2023068

您当前的位置：

首页 >

文章列表页 >

面向算力网络的跨域数据管理方法

研究 | 更新时间：2024-06-03

- 面向算力网络的跨域数据管理方法
- Cross-domain data management for computing power networks
- 大数据 2024年10卷第2期页码：94-108
- 作者机构：
  
  1. 中国人民大学大型科学仪器共享平台，北京 100083
  2. 联旌智能科技（上海）有限公司，上海 200051
  3. 华中科技大学网络与计算中心，湖北武汉 430074
- 作者简介：
  
  [ "鲁蔚征（1990- ），男，中国人民大学大型科学仪器共享平台实验师，主要研究方向为高性能计算、数据科学。" ]
  [ "戴奇志（1984- ），男，联旌智能科技（上海）有限公司首席技术官，主要研究方向为高性能计算、云计算。" ]
  [ "张策（1992- ），男，华中科技大学网络与计算中心工程师，主要研究方向为高性能计算、数据中心信息化。" ]
- 基金信息：
  
  国家重点研发计划资助项目;The National Key Researchand Development Program of China(2020YFB1710004）)
- DOI：10.11959/j.issn.2096-0271.2023068
  中图分类号： TP316
- 网络首发：2024-03，
  
  纸质出版：2024-03-15
- 稿件说明：
移动端阅览
鲁蔚征, 戴奇志, 张策. 面向算力网络的跨域数据管理方法[J]. 大数据, 2024,10(2):94-108.

Weizheng LU, Qizhi DAI, Ce ZHANG. Cross-domain data management for computing power networks[J]. Big data research, 2024, 10(2): 94-108.
鲁蔚征, 戴奇志, 张策. 面向算力网络的跨域数据管理方法[J]. 大数据, 2024,10(2):94-108. DOI： 10.11959/j.issn.2096-0271.2023068.

Weizheng LU, Qizhi DAI, Ce ZHANG. Cross-domain data management for computing power networks[J]. Big data research, 2024, 10(2): 94-108. DOI： 10.11959/j.issn.2096-0271.2023068.

摘要

跨域算力网络希望整合多个算力中心的计算和数据资源，但现有的方案对跨域文件和数据管理关注不够。提出了一种轻量级的跨域算力网络数据管理方案：通过文件系统协议转换，接入远程算力中心的并行文件系统存储资源；算力中心内部的存储资源作为一种补充，应对高IOPS应用；通过容器绑定技术，将远程存储挂载并绑定到指定目录。基于该方案的原型系统已经在高校校级计算平台部署运行。实测数据和用户体验显示，该方案能够满足常见高性能计算应用需求。

Abstract

Cross-domain computing power networks wish to integrate computational and data resources from multiple computing centers

but existing methods do not pay enough attention to cross-domain file and data management.In this paper

a lightweight data access scheme for cross-domain computing power networks was proposed: (1) accessing parallel file system storage resources of remote computing centers through file system protocol conversion; (2) local caching as a supplement to cope with high IOPS applications; and (3) mounting remote or local storage to specified directories through container binding technology.The prototype system based on this scheme had been deployed on highperformance computing centers in multiple universities.The measured data and user experience showed that the scheme in this paper could meet the requirements of common high-performance computing applications.

关键词

Keywords

references

FOSTER I T , KESSELMAN C . The grid:blueprint for a new computing infrastructure [M ] . San Francisco : Morgan Kaufman Publishers , 1998 .

XU Z W , CHI X B , XIAO N . Highperformance computing environment:a review of twenty years of experiments in China [J ] . National Science Review , 2016 , 3 ( 1 ): 36 - 48 .

钱德沛 , 栾钟治 , 刘轶 . 从网格到“东数西算”：构建国家算力基础设施 [J ] . 北京航空航天大学学报 , 2022 , 48 ( 9 ): 1561 - 1574 .

QIAN D P , LUAN Z Z , LIU Y . From grid to “East-west Computing Transfer”:constructing national computing infrastructure [J ] . Journal of Beijing University of Aeronautics and Astronautics , 2022 , 48 ( 9 ): 1561 - 1574 .

秦广军 , 肖利民 , 张广艳 , 等 . 面向国家高性能计算环境的虚拟数据空间系统 [J ] . 大数据 , 2021 , 7 ( 2 ): 101 - 122 .

QIN G J , XIAO L M , ZHANG G Y , et al . Virtual data space system for national high-performance computing environment [J ] . Big Data Research , 2021 , 7 ( 2 ): 101 - 122 .

何小雨 , 邓笋根 , 栾海晶 , 等 . 国家高性能计算环境的虚拟数据空间运行支撑技术研究 [J ] . 大数据 , 2021 , 7 ( 2 ): 158 - 171 .

HE X Y , DENG S G , LUAN H J , et al . Study of technique support on the operation of virtual data space in national highperformance computing environment [J ] . Big Data Research , 2021 , 7 ( 2 ): 158 - 171 .

肖利民 , 宋尧 , 秦广军 , 等 . GVDS：面向广域高性能计算环境的虚拟数据空间 [J ] . 大数据 , 2021 , 7 ( 2 ): 123 - 146 .

XIAO L M , SONG Y , QING J , et al . GVDS:a global virtual data space for wide-area high-performance computing environments [J ] . Big Data Research , 2021 , 7 ( 2 ): 123 - 146 .

YOO A B , JETTE M A , GRONDONA M . SLURM:simple linux utility for resource management [C ] // Proceedings of Job Scheduling Strategies for Parallel Processing . Seattle:Springer , 2003 : 44 - 60 .

YIN F , SHI F . A comparative survey of big data computing and hpc:from a parallel programming model to a cluster architecture [J ] . International Journal of Parallel Programming , 2022 , 50 ( 1 ): 27 - 64 .

BURROWS M . Lustre:building a file system for 1000-node clusters [C ] // Proceedings of the 2003 Linux Symposium .[S.l.:s.n. ] , 2003 : 380 - 386 .

SCHMUCK F B , HASKIN R L . GPFS:a shared-disk file system for largecomputing clusters [C ] // The FAST 2002 Conferenceon File and Storage Technologies . Monterey:USENIX , 2002 : 231 - 244 .

CATLETT C , ALLCOCK W E , ANDREWS P , et al . TeraGrid:analysis of organization,system architecture,and middleware enabling new types of applications [C ] // Proceedings of the 2006 International Advanced Research Workshop on High Performance Computing and Grids . Amsterdam:IOS Press , 2006 : 225 - 249 .

TOWN J , BOISSEAU J , ROSKIES J , et al . XSEDE:extreme science and engineering discovery environment (OAC 15-48562) [R ] . 2020 .

NEWHOUSE S . Seeking new horizons:EGI’s role in 2020 (EGI-1098-D230-V3) [R ] . 2021 .

HENSCHEL R , SIMMS S , HANCOCK D , et al . Demonstrating lustre over a 100Gbps wide area network of 3500km [C ] // Proceedings ofthe International Conference on High Performance Computing,Networking,Storage and Analysis . Piscataway:IEEE Press , 2012 : 1 - 8 .

THOMSON A , ABADI D J . CalvinFS:consistent WAN replication and scalable metadata management for distributed file systems [C ] // Proceedings of the 13th USENIX Conference on File and Storage Technologies . Berkeley:USENIX , 2015 : 1 - 14 .

CORBETT J C , DEAN J , EPSTEIN M , et al . Spanner:Google’s globally-distributed database [J ] . ACM Transactions on Computer Systems , 2013 , 31 ( 3 ): 8 .

CAO W , LIU Z J , WANG P , et al . PolarFS:an ultra-low latency and failure resilient distributed file system for shared storage cloud database [J ] . Proceedings of the VLDB Endowment , 2018 , 11 ( 12 ): 1849 - 1862 .

A network filesystem client to connect to SSH servers [Z ] . 2023 .

DUTKA U , SOTA R , WRZESZCZ M , et al . Uniform and efficient access to data in organizationally distributed environments [C ] // Proceeding of eScience on Distributed Computing Infrastructure . Cham:Springer , 2014 : 178 - 194 .

DONENFELD J A . Wireguard:next generation kernel network tunnel [C ] // Proceeding of 24th Annual Network and Distributed System Security Symposium . San Diego:The Internet Society , 2017 : 1 - 12 .

DONGARRA J , LUSZCZEK P , PETITET A . The LINPACK benchmark:past,present,and future [J ] . Concurrency and Computation:Practice and Experience , 2003 , 15 ( 9 ): 803 - 820 .

Fio - flexible I/O tester rev.3.25 [Z ] . 2020 .

浏览量

296

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

大模型时代下的存储系统挑战与技术发展

面向元宇宙的云边端协同大数据管理

计量科学大数据分级分类

高能同步辐射光源科学数据管理策略研究与应用

GVDS：面向广域高性能计算环境的虚拟数据空间