水环境模型与大数据技术融合研究

马金锋; 饶凯锋; 李若男; 张京; 郑华

doi:10.11959/j.issn.2096-0271.2021064

您当前的位置：

首页 >

文章列表页 >

水环境模型与大数据技术融合研究

研究 | 更新时间：2024-06-03

- 水环境模型与大数据技术融合研究
- Research on the integration of water environment model and big data technology
- 大数据 2021年7卷第6期页码：103-119
- 作者机构：
  
  1. 中国科学院生态环境研究中心城市与区域生态国家重点实验室，北京 100085
  2. 中国科学院大学，北京 100049
- 作者简介：
  
  [ "马金锋（1978- ），男，中国科学院生态环境研究中心助理研究员，主要研究方向为水环境数值模拟" ]
  [ "饶凯锋（1976- ），男，中国科学院生态环境研究中心助理研究员，主要研究方向为水生态毒理学、环境预警监测与物联网" ]
  [ "李若男（1982- ），女，中国科学院生态环境研究中心副研究员，主要研究方向为流域生态系统过程与模拟" ]
  [ "张京（1996- ），女，中国科学院生态环境研究中心硕士生，主要研究方向为水环境数值模拟" ]
  [ "郑华（1974- ），男，中国科学院生态环境研究中心研究员，主要研究方向为生态系统过程与生态系统服务" ]
- 基金信息：
  
  国家重点研发计划资助项目;The National Key Research and Development Program of China(2019YFD0901105)
- DOI：10.11959/j.issn.2096-0271.2021064
  中图分类号： TP311
- 网络首发：2021-11，
  
  纸质出版：2021-11-15
- 稿件说明：
移动端阅览
马金锋, 饶凯锋, 李若男, 等. 水环境模型与大数据技术融合研究[J]. 大数据, 2021,7(6):103-119.

Jinfeng MA, Kaifeng RAO, Ruonan LI, et al. Research on the integration of water environment model and big data technology[J]. Big data research, 2021, 7(6): 103-119.
马金锋, 饶凯锋, 李若男, 等. 水环境模型与大数据技术融合研究[J]. 大数据, 2021,7(6):103-119. DOI： 10.11959/j.issn.2096-0271.2021064.

Jinfeng MA, Kaifeng RAO, Ruonan LI, et al. Research on the integration of water environment model and big data technology[J]. Big data research, 2021, 7(6): 103-119. DOI： 10.11959/j.issn.2096-0271.2021064.

摘要

水环境模型内部结构复杂且计算耗时，造成参数率定、多情景分析及决策优化过程中面临高负荷计算难题，这极大地限制了其应用价值的发挥。如何融合水环境模型和大数据技术，深入挖掘模型应用潜力和充分发挥其应用价值是一个研究热点。总结了水环境模型在实际应用过程中面临的瓶颈，分析了大数据技术在解决这些问题上具有的潜力。基于现有成熟的大数据技术，提出了水环境模型与大数据技术融合框架，解决了水环境模型规模计算、规模存储和应用分析问题。阐述了模型与大数据技术融合过程中面临的问题，提出了具体的实现技术思路。通过SWAT模型率定应用案例，证明融合框架的可行性。最后探讨了大数据背景下水环境模型的未来研究方向，指出开展复杂水环境模型的代理模型研究和水环境模拟优化框架研究是未来的发展趋势。

Abstract

Applications of water environment models are greatly limited by complex internal structure of the model and timeconsuming calculations

significant computation burdens arise during the process of parameter calibration

multi-scenario analysis

and decision-making optimization.How to integrate water environment model and big data technology

deeply explore the potential of model application and give full play to its application value is a research hotspot.The bottlenecks faced by the water environment model in the process of practical application were summarized

and the potential of big data technology in solving these problems was analyzed.Based on the existing big data technology

a framework for the integration of water environment model and big data technology was proposed to solve the problem of large-scale calculation

large-scale storage and application analysis of water environment model.The problems faced in the integration of model and big data technologies were described

and specific technical ways of implementation were proposed.A case study for calibration of SWAT model was used to demonstrate feasibility of the proposed framework.Finally

the future research direction of water environment modeling in the context of big data was discussed

and the conclusion was pointed out that the research on surrogate modeling of complex water environment model and on water environment simulation and optimization framework is the future development trend.

关键词

Keywords

references

National Academies of Sciences,Engineering , Engineering , Engineering,and Medicine , Division on Earth and Life Studies , Water Science and Technology Board ,, et al . Future water priorities for the nation:directions for the U.S.geological survey water mission area [M ] . Washington D.C. : National Academies Press , 2018 .

REICHSTEIN M , CAMPS-VALLS G ,, STEVENS B , et al . Deep learning and process understanding for data-driven earth system science [J ] . Nature , 2019 , 566 ( 7743 ): 195 - 204 .

陈永灿 , 刘昭伟 , 朱德军 . 水动力及水环境模拟方法与应用 [M ] . 北京 : 科学出版社 , 2012 .

CHEN Y C , LIU Z W , ZHU D J . Hydrodynamic and water environment simulation methods and applications [M ] . Beijing : Science Press , 2012 .

季振刚 . 水动力学和水质——河流、湖泊及河口数值模拟 [M ] . 李建平,冯立成,赵万星 ,译. 北京 : 海洋出版社 , 2017 .

JI Z G . Hydrodynamics and water quality:modeling rivers,lakes,and estuaries [M ] . Translated by LI J P,FENG L C,ZHAO W X . Beijing : China Ocean Press , 2017 .

RAZAVI S , TOLSON B A , MATOTT L S , et al . Reducing the computational cost of automatic calibration through model preemption [J ] . Water Resources Research , 2010 , 46 ( 11 ).

ZHANG X S , SRINIVASAN R , BOSCH D . Calibration and uncertainty analysis of the SWAT model using genetic algorithms and bayesian model averaging [J ] . Journal of Hydrology , 2009 , 374 ( 3-4 ): 307 - 317 .

LIN F R , WU N J , TU C H , et al . Automatic calibration of an unsteady river flow model by using dynamically dimensioned search algorithm [J ] . Mathematical Problems in Engineering , 2017 ( 7 ): 1 - 19 .

马金锋 , 唐力 , 饶凯锋 , 等 . Hadoop下水环境模拟集群运算模式 [J ] . 大数据 , 2019 , 5 ( 6 ): 73 - 84 .

MA J F , TANG L , RAO K F , et al . Cluster computing mode for water environment simulation based on Hadoop [J ] . Big Data Research , 2019 , 5 ( 6 ): 73 - 84 .

ZHANG D J , CHEN X W , YAO H X , et al . Moving SWAT model calibration and uncertainty analysis to an enterprise Hadoop-based cloud [J ] . Environmental Modelling ＆ Software , 2016 , 84 : 140 - 148 .

GEORGE L . HBase:the definitive guide(fisrt edition) [M ] . Sebastopol : O’Reilly Media , 2011 .

KARAU H , KONWINSKI A , WENDELL P , et al . Spark快速大数据分析 [M ] . 王道远 ,译. 北京 : 人民邮电出版社 , 2015 .

KARAU H , KONWINSKI A , WENDELL P , et al . Learning Spark:lightning-fast data analysis [M ] . Translated by WANG D Y . Beijing : Posts ＆ Telecom Press , 2015 .

WHITE T . Hadoop:the definitive guide(second edition) [M ] . Sebastopol : O’Reilly Media , 2010 .

吴信东 , 嵇圣硙 . MapReduce与Spark用于大数据分析之比较 [J ] . 软件学报 , 2018 , 29 ( 6 ): 1770 - 1791 .

WU X D , JI S W . Comparative study on MapReduce and Spark for big data analytics [J ] . Journal of Software , 2018 , 29 ( 6 ): 1770 - 1791 .

SHIVHARE N , DIKSHIT P K S , DWIVEDI S B . A comparison of SWAT model calibration techniques for hydrological modeling in the Ganga river watershed [J ] . Engineering , 2018 , 4 ( 5 ): 643 - 652 .

MURTHY A C , VAVILAPALLI V K . Apache Hadoop YARN:moving beyond MapReduce and batch processing with Apache Hadoop 2 [M ] . Boston : AddisonWesley Professional , 2014 .

YI X , ZOU R , GUO H C . Global sensitivity analysis of a three-dimensional nutrients-algae dynamic model for a large shallow lake [J ] . Ecological Modelling , 2016 , 327 : 74 - 84 .

SONG X M , ZHANG J Y , ZHAN C S , et al . Global sensitivity analysis in hydrological modeling:review of concepts,methods,theoretical framework,and applications [J ] . Journal of Hydrology , 2015 , 523 : 739 - 757 .

JIANG L , LI Y P , ZHAO X , et al . Parameter uncertainty and sensitivity analysis of water quality model in Lake Taihu,China [J ] . Ecological Modelling , 2018 , 375 : 1 - 12 .

CAMPOLONGO F , SALTELLI A . Sensitivity analysis of an environmental model:an application of different analysis methods [J ] . Reliability Engineering ＆System Safety , 1997 , 57 ( 1 ): 49 - 69 .

刘松 , 佘敦先 , 张利平 , 等 . 基于Morris和Sobol的水文模型参数敏感性分析 [J ] . 长江流域资源与环境 , 2019 , 28 ( 6 ): 1296 - 1303 .

LIU S , SHE D X , ZHANG L P , et al . Global sensitivity analysis of hydrological model parameters based on Morris and Sobol methods [J ] . Resources and Environment in the Yangtze Basin , 2019 , 28 ( 6 ): 1296 - 1303 .

BENNETT N D , CROKE B F W , GUARISO G , et al . Characterising performance of environmental models [J ] . Environmental Modelling ＆ Software , 2013 , 40 : 1 - 20 .

BAE S , SEO D . Analysis and modeling of algal blooms in the Nakdong River,Korea [J ] . Ecological Modelling , 2018 , 372 : 53 - 63 .

NASH J E , SUTCLIFFE J V . River flow forecasting through conceptual models part I—a discussion of principles [J ] . Journal of Hydrology , 1970 , 10 ( 3 ): 282 - 290 .

GHAITH M , LI Z . Propagation of parameter uncertainty in SWAT:a probabilistic forecasting method based on polynomial chaos expansion and machine learning [J ] . Journal of Hydrology , 2020 , 10 : 124854

任婷玉 , 梁中耀 , 刘永 , 等 . 基于贝叶斯优化的三维水动力-水质模型参数估值方法 [J ] . 环境科学学报 , 2019 , 39 ( 6 ): 2024 - 2032 .

REN T Y , LIANG Z Y , LIU Y , et al . The parameters estimation method based on Bayesian optimization for complex water quality models [J ] . Acta Scientiae Circumstantiae , 2019 , 39 ( 6 ): 2024 - 2032 .

RAZAVI S , TOLSON B A , BURN D H . Review of surrogate modeling in water resources [J ] . Water Resources Research , 2012 , 48 ( 7 ).

SIMPSON T W , MAUERY T M , KORTE J J , et al . Kriging models for global approximation in simulation-based multidisciplinary design optimization [J ] . AIAA Journal , 2001 , 39 ( 12 ): 2233 - 2241 .

DOWLA F U , ROGERS L L . Solving problems in environmental engineering and geosciences with artificial neural networks [M ] . Cambridge : MIT Press , 1995 .

REGIS R G , SHOEMAKER C A . A stochastic radial basis function method for the global optimization of expensive functions [J ] . INFORMS Journal on Computing , 2007 , 19 ( 4 ): 497 - 509 .

FEN C S , CHAN C C , CHENG H C . Assessing a response surface-based optimization approach for soil vapor extraction system design [J ] . Journal of Water Resources Planning and Management , 2009 , 135 ( 3 ): 198 - 207 .

ZHANG X S , SRINIVASAN R , LIEW M V . Approximating SWAT model using artificial neural network and support vector machine [J ] . JAWRA Journal of the American Water Resources Association , 2009 , 45 ( 2 ): 460 - 474 .

ZENG L Z , SHI L S , ZHANG D X , et al . A sparse grid based Bayesian method for contaminant source identification [J ] . Advances in Water Resources , 2012 , 37 : 1 - 9 .

BOYLE J S , KLEIN S A , LUCAS D D , et al . The parametric sensitivity of CAM5’s MJO [J ] . Journal of Geophysical Research:Atmospheres , 2015 , 120 ( 4 ): 1424 - 1444 .

LIANG Z Y , ZOU R , CHEN X , et al . Simulate the forecast capacity of a complicated water quality model using the long short-term memory approach [J ] . Journal of Hydrology , 2020 , 581 : 124432

浏览量

544

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

Hadoop下水环境模拟集群运算模式

面向大数据的可扩展正则采样并行排序算法

面向大数据场景的系统性能优化实践

基于大数据技术的甘肃智慧旅游系统

大数据与计算模型