Design of big data processing system supporting multi-satellites and multi-tasks

Fuli MA; Tao SHI; Ling CHEN; Yan ZHENG; Senlin XIONG

doi:10.11959/j.issn.2096-0271.2021045

您当前的位置：

首页 >

文章列表页 >

Design of big data processing system supporting multi-satellites and multi-tasks

TOPIC: BIG DATA PROCESSING SYSTEM IN CHINA’S HOMEMADE COMPUTING ENVIRONMENT | 更新时间：2024-06-03

- Design of big data processing system supporting multi-satellites and multi-tasks
- Big data research Vol. 7, Issue 5, Pages: 2021045(2021)
- 作者机构：
  
  1. 中国科学院国家空间科学中心，北京 100190
  2. 中国科学院空天信息创新研究院，北京 100094
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-0271.2021045
  CLC： V557
- Online First：2021-09，
  
  Published：15 September 2021
- 稿件说明：
移动端阅览
Fuli MA, Tao SHI, Ling CHEN, et al. Design of big data processing system supporting multi-satellites and multi-tasks[J]. Big data research, 2021, 7(5): 2021045.
DOI：

Fuli MA, Tao SHI, Ling CHEN, et al. Design of big data processing system supporting multi-satellites and multi-tasks[J]. Big data research, 2021, 7(5): 2021045. DOI： 10.11959/j.issn.2096-0271.2021045.

摘要

随着我国发射的空间科学卫星越来越多，科学数据量爆发式增长，空间科学卫星大数据处理逐渐成为空间科学创新发展过程中被重点关注的环节，建设一套高性能的地面数据处理系统是建设我国自主可控的空间科学大数据生态、助力科学成果产出的重要推手。面向多星多任务并行处理的任务需求，针对空间科学卫星大数据处理中的多分级分类、多源产品融合组织以及高时效性要求等特点，提出了适用于大数据处理业务场景的高可靠硬件环境设计方案，并针对科学卫星处理任务类型多的特点，提出了基于任务类型感知的统一资源调度系统。完成了面向多星多任务的可扩展地面大数据处理系统的研制，支撑了中国科学院空间科学先导专项中的科学卫星数据处理任务。

Abstract

With more and more space science satellites were launched in China

the scientific data amount has grown explosively

and the resulting space science satellite big data processing has gradually become a key link in the development of space science innovation.A high-performance ground data processing system is an important driver for promoting the construction of Chinese controllable space science big data ecology and boosting the output of scientific results.Aiming at the characteristics of multi-level classification

multi-source product integration organization and high timeliness requirements in space science satellite data processing

and for the task requirements of multi-satellite and multi-task parallel processing

a high-reliability hardware system design suitable for big data processing business scenarios was proposed

and a unified resource scheduling system based on task type perception was proposed according to the characteristics of scientific satellite processing tasks.Based on these designs

the development of a space science big data scalable ground big data processing system was completed

which had successfully supported the data processing tasks for space science missions from Chinese Academy of Sciences.

关键词

Keywords

references

陈世荣 , 严立 , 申佩佩 . 高分遥感卫星影像预处理系统的设计和实现 [J ] . 测绘与空间地理信息 , 2021 , 44 ( 3 ): 1 - 3 .

CHEN S R , YAN L , SHEN P P . Design and implements of high-resolution satellite image preprocessing system [J ] . Geomatics ＆Spatial Information Technology , 2021 , 44 ( 3 ): 1 - 3 .

刘定生 , 陈元伟 , 李景山 . 遥感卫星地面预处理系统技术发展模式探讨 [J ] . 遥感信息 , 2008 , 23 ( 5 ): 87 - 91 .

LIU D S , CHEN Y W , LI J S . The study on the development patterns of remote sensing satellite ground pre-processing systems [J ] . Remote Sensing Information , 2008 , 23 ( 5 ): 87 - 91 .

MOSES J F , . Architectures toward reusable science data systems [C ] // Proceedings of the AGU Fall Meeting .[S.l:s.n. ] , 2014 .

钱建梅 , 孙安来 , 徐喆 , 等 . 风云气象卫星数据存档与服务系统 [J ] . 应用气象学报 , 2012 , 23 ( 3 ): 369 - 376 .

QIAN J M , SUN A L , XU Z , et al . Fengyun series meteorological satellite data archiving and service system [J ] . Journal of Applied Meteorological Science , 2012 , 23 ( 3 ): 369 - 376 .

赵现纲 , 林曼筠 , 谢利子 , 等 . 风云卫星地面应用系统计算机网络平台架构 [J ] . 气象科技进展 , 2020 , 10 ( 3 ): 43 - 48 .

ZHAO X G , LIN M J , XIE L Z , et al . Computer network system of FY satellite ground segment [J ] . Advances in Meteorological Science and Technology , 2020 , 10 ( 3 ): 43 - 48 .

李盛阳 , 刘志文 , 张万峰 , 等 . 天宫二号应用任务地面数据处理与服务系统 [J ] . 中国科学:技术科学 , 2020 , 50 ( 8 ): 1066 - 1080 .

LI S Y , LIU Z W , ZHANG W F , et al . Tiangong-2 application mission ground data processing and service system [J ] . Scientia Sinica Technologica , 2020 , 50 ( 8 ): 1066 - 1080 .

BENKHOFF J , CASTEREN J V , HAYAKAWA H , et al . BepiColombocomprehensive exploration of mercury:mission overview and science goals [J ] . Planetary and Space Science , 2010 , 58 ( 1-2 ): 2 - 20 .

D’AMORE M , JÖRN H , MATURILLI A , et al . BepiColombo/MERTIS data processing pipeline at DLR [C ] // Proceedings of the EGU General Assembly 2016 .[S.l:s.n. ] , 2016 .

DAMPE C . Direct detection of a break in the teraelectronvolt cosmic-ray spectrum of electrons and positrons [J ] . Nature , 2017 , 552 ( 7683 ): 63 - 66 .

LIAO S K , CAI W Q , LIU W Y , et al . Satelliteto-ground quantum key distribution [J ] . Nature , 2017 , 549 ( 7670 ): 43 - 47 .

LIAO S K , YONG H L , LIU C , et al . Long-distance free-space quantum key distribution in daylight towards intersatellite communication [J ] . Nature Photonics , 2017 , 11 ( 8 ): 509 - 513 .

YIN J , CAO Y , LI Y H , et al . Satellitebased entanglement distribution over 1200 kilometers [J ] . Science , 2017 , 356 ( 6343 ): 1140 - 1144 .

LI T P , XIONG S L , ZHANG S N , et al . Insight-HXMT observations of the first binary neutron star merger GW170817 [J ] . Science China Physics,Mechanics ＆Astronomy , 2017 , 61 ( 3 ): 1 - 8 .

Views

548

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

A scalable parallel sorting algorithm by regular sampling for big data

Gansu smart tourism system based on big data

Big data and computing models

Analysis and research on the strategy and problems of “Channel Computing Resources from the East to the West”

Digital transformation in higher education:a systematic review

Related Author

Ying WANG

Zhiguang CHEN

Yutong LU

Liang GUO

Yi YANG

Bingfeng QIN

Jianwen CAO

Min LI

Related Institution

School of Information Science and Engineering, Lanzhou University

Key Laboratory of Tourism Information Fusion Processing and Data Ownership Protection, Ministry of Culture and Tourism, Lanzhou University

Technology Information Department, Gansu Provincial Department of Culture and Tourism

School of Computer and Artificial Intelligence, Lanzhou Polytechnical College

Key Laboratory of Artificial Intelligence and Computing Power Technology

Postal code：100079
Tel：（010）53879208、53859533 Email：bdr@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：273162 Visits Today：68

⁰