面向广域分布式计算环境的任务与资源动态双向匹配方法

尚晶; 肖利民; 肖智文; 王锦权; 武智晖; 李辉阳; 张逸飞; 宋尧; 王冀彬

doi:10.11959/j.issn.2096-0271.2024050

您当前的位置：

首页 >

文章列表页 >

面向广域分布式计算环境的任务与资源动态双向匹配方法

专题：大数据与云存储 | 更新时间：2024-08-07

- 面向广域分布式计算环境的任务与资源动态双向匹配方法
- A dynamic bidirectional matching method of tasks and resources oriented to wide-area distributed computing
- 大数据 2024年10卷第4期页码：51-65
- 作者机构：
  
  1. 中移动信息技术有限公司，北京 100033
  2. 复杂关键软件环境全国重点实验室，北京 100191
  3. 北京航空航天大学计算机学院，北京 100191
  4. 中国信息通信研究院，北京 100191
- 作者简介：
  
  [ "尚晶（1978- ），女，博士，中国移动信息技术中心高级工程师，中国移动首席专家，主要研究方向为大数据、云计算、数据库。" ]
  [ "肖利民（1970- ），男，博士，北京航空航天大学教授、博士生导师，计算机科学技术系主任，系统结构研究所所长，中国计算机学会大数据专委会委员、高性能计算专委会委员、容错计算专委会委员，中国电子学会云计算专委会委员，主要研究方向为计算机体系结构、大数据存储、高性能计算等。曾获国家科技进步二等奖4项、省部级科技一等奖4项及其他省部级奖5项。" ]
  [ "肖智文（1993- ），男，博士，中国移动信息技术中心中级工程师，主要研究方向为大数据、机器学习、云计算。" ]
  [ "王锦权（1998- ），男，北京航空航天大学博士生，主要研究方向为分布式存储系统、分布式调度系统、高性能计算等。" ]
  [ "武智晖（1978- ），男，中国移动信息技术中心高级工程师、架构师，主要研究方向为大数据平台、数据处理、数据库。" ]
  [ "李辉阳（2000- ），男，北京航空航天大学硕士生，主要研究方向为分布式调度系统、大数据存储等。" ]
  [ "张逸飞（1996- ），男，博士，中国移动信息技术中心中级工程师、项目经理，主要研究方向为大数据、机器学习、云计算。" ]
  [ "宋尧（1994- ），男，博士，中国信息通信研究院技术与标准研究所中级工程师，主要研究方向为高性能计算、边缘计算、算网融合、分布式存储、分布式调度系统、存算联动调度等。" ]
  [ "王冀彬（1980- ），男，中国移动信息技术中心高级工程师、大数据事业部总经理，主要研究方向为大数据、数据分析。" ]
- 基金信息：
  
  国家重点研发计划项目;The National Key Research and Development Program of China(2023YFB4503100);国家自然科学基金项目;The National Natural Science Foundation of China(U23B2027);中国移动“联创+”资助项目;China Mobile Joint R＆D Project(R23103E4)
- DOI：10.11959/j.issn.2096-0271.2024050
  中图分类号： TP316
- 网络首发：2024-07，
  
  纸质出版：2024-07-15
- 稿件说明：
移动端阅览
尚晶, 肖利民, 肖智文, 等. 面向广域分布式计算环境的任务与资源动态双向匹配方法[J]. 大数据, 2024,10(4):51-65.

Jing SHANG, Limin XIAO, Zhiwen XIAO, et al. A dynamic bidirectional matching method of tasks and resources oriented to wide-area distributed computing[J]. Big data research, 2024, 10(4): 51-65.
尚晶, 肖利民, 肖智文, 等. 面向广域分布式计算环境的任务与资源动态双向匹配方法[J]. 大数据, 2024,10(4):51-65. DOI： 10.11959/j.issn.2096-0271.2024050.

Jing SHANG, Limin XIAO, Zhiwen XIAO, et al. A dynamic bidirectional matching method of tasks and resources oriented to wide-area distributed computing[J]. Big data research, 2024, 10(4): 51-65. DOI： 10.11959/j.issn.2096-0271.2024050.

摘要

广域分布式计算环境可提供大规模的计算和存储资源，是支持算力互联和数据流转的重要基础设施。在广域分布式计算环境中，任务与资源的匹配对于提高系统性能具有重要意义。然而，任务与资源的多样性、地理位置分散的资源会增加二者匹配的复杂性。针对响应延迟高、匹配效率低等问题，提出了面向广域分布式计算环境的任务与资源动态匹配方法，通过建立统一的任务需求模型和资源能力模型来简化匹配过程，降低响应延迟。此外，定义了任务向匹配度和资源向匹配度以刻画任务视角和资源视角的偏好，并权衡二者；定义了任务和资源的双向综合匹配度以量化任务需求和资源能力的适配程度。最后通过动态计算每一组任务与资源间的双向综合匹配度以优化匹配效果。实验结果表明，与现有的方法相比，该方法可提升匹配效果，并大幅降低平均响应延迟。

Abstract

Due to the huge capacities of computing and storage resources

wide-area distributed computing environment has become important infrastructures supporting computing power and data interconnection.In wide-area distributed computing environment

matching of tasks and resources is important to improve system performance.However

the diversity of tasks and resources and the geographical dispersion of resources increase the complexity of matching problems.To solve the problems of high response delay and low matching efficiency

a dynamic bidirectional matching method of tasks and resources oriented to wide-area distributed computing environments is proposed.The matching process is simplified and the response delay is mitigated by building a unified task requirement model and resource capability model.Moreover

the task-oriented and resource-oriented matching degrees are defined to express the preference of task-perspective and resource-perspective; the two-side comprehensive matching degree of tasks and resources is defined by the trade-off of the task-oriented and resource-oriented matching degree.The two-side comprehensive matching degrees are dynamically calculated for each task group and the resources to improve the matching quality.The experimental results show that the proposed method can effectively promoting the matching quality and significantly reduce the response delay compared with the existing methods.

关键词

Keywords

references

CHEN Q , ZHENG Z M , HU C , et al . On-edge multi-task transfer learning:model and practice with data-driven task allocation [J ] . IEEE Transactions on Parallel and Distributed Systems , 2020 , 31 ( 6 ): 1357 - 1371 .

TOWNS J , COCKERILL T , DAHAN M , et al . XSEDE:accelerating scientific discovery [J ] . Computing in Science ＆Engineering , 2014 , 16 ( 5 ): 62 - 74 .

GAGLIARDI F . The EGEE European grid infrastructure project [M ] // High Performance Computing for Computational Science - VECPAR 2004 . Heidelberg : Springer , 2005 : 194 - 203 .

DEPEI Q . CNGrid:a test-bed for Grid technologies in China [C ] // Proceedings of 10th IEEE International Workshop on Future Trends of Distributed Computing Systems . Piscataway:IEEE Press , 2004 : 135 - 139 .

KANG S , VEERAVALLI B , AUNG K M M . Dynamic scheduling strategy with efficient node availability prediction for handling divisible loads in multicloud systems [J ] . Journal of Parallel and Distributed Computing , 2018 , 113 : 1 - 16 .

CHOUDHARY A . A walkthrough of Amazon elastic compute cloud (amazon EC2):a review [J ] . International Journal for Research in Applied Science and Engineering Technology , 2021 , 9 ( 11 ): 93 - 97 .

王建冬 , 于施洋 , 窦悦 . 东数西算：我国数据跨域流通的总体框架和实施路径研究 [J ] . 电子政务 , 2020 ( 3 ): 13 - 21 .

WANG J D , YU S Y , DOU Y . East digital computing and west computing:research on the overall framework and implementation path of cross-domain data circulation in China [J ] . E-Government , 2020 ( 3 ): 13 - 21 .

高文 . 中国算力网的机遇与挑战 [J ] . 中国计算机学会通讯 , 2023 , 1 : 1 - 6 .

GAO W . The opportunities and challenges of China’s computing power network [J ] . Communictions of the CCF , 2023 ( 2 ): 1 - 6 .

钱德沛 , 栾钟治 , 刘轶 . 从网格到“东数西算”:构建国家算力基础设施 [J ] . 北京航空航天大学学报 , 2022 , 48 ( 9 ): 1561 - 1574 .

QIAN D P , LUAN Z Z , LIU Y . From grid to “East-west Computing Transfer”:constructing national computing infrastructure [J ] . Journal of Beijing University of Aeronautics and Astronautics , 2022 , 48 ( 9 ): 1561 - 1574 .

XU K , LYU L , LI T , et al . Minimizing tardiness for data-intensive applications in heterogeneous systems:a matching theory perspective [J ] . IEEE Transactions on Parallel and Distributed Systems , 2020 , 31 ( 1 ): 144 - 158 .

毕娅 , 原惠群 , 初叶萍 , 等 . 大数据环境下基于公共服务平台的资源多级智能寻租与匹配策略和价值创造 [J ] . 计算机科学 , 2019 , 46 ( 2 ): 42 - 49 .

BI Y , YUAN H Q , CHU Y P , et al . Multilevel and intelligent rent-seeking and matching resource strategy and value creation of public service platform in big data environment [J ] . Computer Science , 2019 , 46 ( 2 ): 42 - 49 .

ZHAO L P , YANG Y N , MUNIR A , et al . Optimizing geo-distributed data analytics with coordinated task scheduling and routing [J ] . IEEE Transactions on Parallel and Distributed Systems , 2020 , 31 ( 2 ): 279 - 293 .

HE L , QIAN Z C . Intent-based resource matching strategy in cloud [J ] . Information Sciences , 2020 , 538 : 1 - 18 .

RAVEENDRAN N , ZHANG H Q , SONG L Y , et al . Pricing and resource allocation optimization for IoT fog computing and NFV:an EPEC and matching based perspective [J ] . IEEE Transactions on Mobile Computing , 2022 , 21 ( 4 ): 1349 - 1361 .

FENG W J , ZHENG J L , JIANG W H . Joint pilot and data transmission power control and computing resource allocation algorithm for massive MIMO-MEC networks [J ] . IEEE Access , 2020 , 8 : 80801 - 80811 .

CHEN Y F , LI Z Y , YANG B , et al . A Stackelberg game approach to multiple resources allocation and pricing in mobile edge computing [J ] . Future Generation Computer Systems , 2020 , 108 : 273 - 287 .

LI B D , YANG Y , SU J F , et al . Two-sided matching decision-making model with hesitant fuzzy preference information for configuring cloud manufacturing tasks and resources [J ] . Journal of Intelligent Manufacturing , 2020 , 31 ( 8 ): 2033 - 2047 .

LI C L , BAI J P , TANG J H . Joint optimization of data placement and scheduling for improving user experience in edge computing [J ] . Journal of Parallel and Distributed Computing , 2019 , 125 : 93 - 105 .

CHEN L T , CHEN S Q . Volunteer multi-person multi-task optimization dispatch method considering two-sided matching [J ] . Soft Computing , 2022 , 26 ( 8 ): 3837 - 3861 .

WANG W , LI B C , LIANG B , et al . Multiresource fair sharing for datacenter jobs with placement constraints [C ] // Proceedings of the International Conference for High Performance Computing,Networking,Storage and Analysis . Piscataway:IEEE Press , 2016 : 1003 - 1014 .

LEONG S H , PARODI A , KRANZLMÜLLER D . A robust reliable energy-aware urgent computing resource allocation for flash-flood ensemble forecasting on HPC infrastructures for decision support [J ] . Future Generation Computer Systems , 2017 , 68 : 136 - 149 .

KAMECKE U , ROTH A E , SOTOMAYOR M A O . Two sided matching:a study in game-theoretic modeling and analysis [J ] . Economica , 1992 , 59 ( 236 ): 487 .

MA Y T , WANG H J , XIONG J , et al . Joint allocation on communication and computing resources for fog radio access networks [J ] . IEEE Access , 2020 , 8 : 108310 - 108323 .

YUAN Y L , YANG T , HU Y L , et al . Two-timescale resource allocation for cooperative D2D communication:a matching game approach [J ] . IEEE Transactions on Vehicular Technology , 2021 , 70 ( 1 ): 543 - 557 .

DING D , FAN X C , LUO S W . Useroriented cloud resource scheduling with feedback integration [J ] . The Journal of Supercomputing , 2016 , 72 ( 8 ): 3114 - 3135 .

WANG Y F , LIU J , TONG Y , et al . Resource scheduling in mobile edge computing using improved ant colony algorithm for space information network [J ] . International Journal of Satellite Communications and Networking , 2023 , 41 ( 4 ): 331 - 356 .

SHEN H X , LI S G , LIANG Y Y . Faster algorithms for bicriteria scheduling of identical jobs on uniform machines [J ] . Journal of Industrial and Management Optimization , 2023 , 19 ( 7 ): 5398 - 5406 .

SONG Y , WANG L , XIAO L M , et al . Dynamic two-side matching of tasks and resources in wide-area distributed computing environments [J ] . The Journal of Supercomputing , 2023 , 79 ( 9 ): 10208 - 10231 .

LI C Y , TANG X Y . On fault-tolerant Bin packing for online resource allocation [J ] . IEEE Transactions on Parallel and Distributed Systems , 2020 , 31 ( 4 ): 817 - 829 .

KHIYAITA A , EL BAKKALI H , ZBAKH M , et al . Load balancing cloud computing:state of art [C ] // Proceedings of the 2012 National Days of Network Security and Systems . Piscataway:IEEE Press , 2012 : 106 - 109 .

浏览量

153

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

暂无数据