Research on the integration of water environment model and big data technology

Jinfeng MA; Kaifeng RAO; Ruonan LI; Jing ZHANG; Hua ZHENG

doi:10.11959/j.issn.2096-0271.2021064

您当前的位置：

首页 >

文章列表页 >

Research on the integration of water environment model and big data technology

STUDY | 更新时间：2024-06-03

- Research on the integration of water environment model and big data technology
- Big data research Vol. 7, Issue 6, Pages: 103-119(2021)
- 作者机构：
  
  1. 中国科学院生态环境研究中心城市与区域生态国家重点实验室，北京 100085
  2. 中国科学院大学，北京 100049
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-0271.2021064
  CLC： TP311
- Online First：2021-11，
  
  Published：15 November 2021
- 稿件说明：
移动端阅览
Jinfeng MA, Kaifeng RAO, Ruonan LI, et al. Research on the integration of water environment model and big data technology[J]. Big data research, 2021, 7(6): 103-119.
DOI：

Jinfeng MA, Kaifeng RAO, Ruonan LI, et al. Research on the integration of water environment model and big data technology[J]. Big data research, 2021, 7(6): 103-119. DOI： 10.11959/j.issn.2096-0271.2021064.

摘要

水环境模型内部结构复杂且计算耗时，造成参数率定、多情景分析及决策优化过程中面临高负荷计算难题，这极大地限制了其应用价值的发挥。如何融合水环境模型和大数据技术，深入挖掘模型应用潜力和充分发挥其应用价值是一个研究热点。总结了水环境模型在实际应用过程中面临的瓶颈，分析了大数据技术在解决这些问题上具有的潜力。基于现有成熟的大数据技术，提出了水环境模型与大数据技术融合框架，解决了水环境模型规模计算、规模存储和应用分析问题。阐述了模型与大数据技术融合过程中面临的问题，提出了具体的实现技术思路。通过SWAT模型率定应用案例，证明融合框架的可行性。最后探讨了大数据背景下水环境模型的未来研究方向，指出开展复杂水环境模型的代理模型研究和水环境模拟优化框架研究是未来的发展趋势。

Abstract

Applications of water environment models are greatly limited by complex internal structure of the model and timeconsuming calculations

significant computation burdens arise during the process of parameter calibration

multi-scenario analysis

and decision-making optimization.How to integrate water environment model and big data technology

deeply explore the potential of model application and give full play to its application value is a research hotspot.The bottlenecks faced by the water environment model in the process of practical application were summarized

and the potential of big data technology in solving these problems was analyzed.Based on the existing big data technology

a framework for the integration of water environment model and big data technology was proposed to solve the problem of large-scale calculation

large-scale storage and application analysis of water environment model.The problems faced in the integration of model and big data technologies were described

and specific technical ways of implementation were proposed.A case study for calibration of SWAT model was used to demonstrate feasibility of the proposed framework.Finally

the future research direction of water environment modeling in the context of big data was discussed

and the conclusion was pointed out that the research on surrogate modeling of complex water environment model and on water environment simulation and optimization framework is the future development trend.

关键词

Keywords

references

National Academies of Sciences,Engineering , Engineering , Engineering,and Medicine , Division on Earth and Life Studies , Water Science and Technology Board ,, et al . Future water priorities for the nation:directions for the U.S.geological survey water mission area [M ] . Washington D.C. : National Academies Press , 2018 .

REICHSTEIN M , CAMPS-VALLS G ,, STEVENS B , et al . Deep learning and process understanding for data-driven earth system science [J ] . Nature , 2019 , 566 ( 7743 ): 195 - 204 .

陈永灿 , 刘昭伟 , 朱德军 . 水动力及水环境模拟方法与应用 [M ] . 北京 : 科学出版社 , 2012 .

CHEN Y C , LIU Z W , ZHU D J . Hydrodynamic and water environment simulation methods and applications [M ] . Beijing : Science Press , 2012 .

季振刚 . 水动力学和水质——河流、湖泊及河口数值模拟 [M ] . 李建平,冯立成,赵万星 ,译. 北京 : 海洋出版社 , 2017 .

JI Z G . Hydrodynamics and water quality:modeling rivers,lakes,and estuaries [M ] . Translated by LI J P,FENG L C,ZHAO W X . Beijing : China Ocean Press , 2017 .

RAZAVI S , TOLSON B A , MATOTT L S , et al . Reducing the computational cost of automatic calibration through model preemption [J ] . Water Resources Research , 2010 , 46 ( 11 ).

ZHANG X S , SRINIVASAN R , BOSCH D . Calibration and uncertainty analysis of the SWAT model using genetic algorithms and bayesian model averaging [J ] . Journal of Hydrology , 2009 , 374 ( 3-4 ): 307 - 317 .

LIN F R , WU N J , TU C H , et al . Automatic calibration of an unsteady river flow model by using dynamically dimensioned search algorithm [J ] . Mathematical Problems in Engineering , 2017 ( 7 ): 1 - 19 .

马金锋 , 唐力 , 饶凯锋 , 等 . Hadoop下水环境模拟集群运算模式 [J ] . 大数据 , 2019 , 5 ( 6 ): 73 - 84 .

MA J F , TANG L , RAO K F , et al . Cluster computing mode for water environment simulation based on Hadoop [J ] . Big Data Research , 2019 , 5 ( 6 ): 73 - 84 .

ZHANG D J , CHEN X W , YAO H X , et al . Moving SWAT model calibration and uncertainty analysis to an enterprise Hadoop-based cloud [J ] . Environmental Modelling ＆ Software , 2016 , 84 : 140 - 148 .

GEORGE L . HBase:the definitive guide(fisrt edition) [M ] . Sebastopol : O’Reilly Media , 2011 .

KARAU H , KONWINSKI A , WENDELL P , et al . Spark快速大数据分析 [M ] . 王道远 ,译. 北京 : 人民邮电出版社 , 2015 .

KARAU H , KONWINSKI A , WENDELL P , et al . Learning Spark:lightning-fast data analysis [M ] . Translated by WANG D Y . Beijing : Posts ＆ Telecom Press , 2015 .

WHITE T . Hadoop:the definitive guide(second edition) [M ] . Sebastopol : O’Reilly Media , 2010 .

吴信东 , 嵇圣硙 . MapReduce与Spark用于大数据分析之比较 [J ] . 软件学报 , 2018 , 29 ( 6 ): 1770 - 1791 .

WU X D , JI S W . Comparative study on MapReduce and Spark for big data analytics [J ] . Journal of Software , 2018 , 29 ( 6 ): 1770 - 1791 .

SHIVHARE N , DIKSHIT P K S , DWIVEDI S B . A comparison of SWAT model calibration techniques for hydrological modeling in the Ganga river watershed [J ] . Engineering , 2018 , 4 ( 5 ): 643 - 652 .

MURTHY A C , VAVILAPALLI V K . Apache Hadoop YARN:moving beyond MapReduce and batch processing with Apache Hadoop 2 [M ] . Boston : AddisonWesley Professional , 2014 .

YI X , ZOU R , GUO H C . Global sensitivity analysis of a three-dimensional nutrients-algae dynamic model for a large shallow lake [J ] . Ecological Modelling , 2016 , 327 : 74 - 84 .

SONG X M , ZHANG J Y , ZHAN C S , et al . Global sensitivity analysis in hydrological modeling:review of concepts,methods,theoretical framework,and applications [J ] . Journal of Hydrology , 2015 , 523 : 739 - 757 .

JIANG L , LI Y P , ZHAO X , et al . Parameter uncertainty and sensitivity analysis of water quality model in Lake Taihu,China [J ] . Ecological Modelling , 2018 , 375 : 1 - 12 .

CAMPOLONGO F , SALTELLI A . Sensitivity analysis of an environmental model:an application of different analysis methods [J ] . Reliability Engineering ＆System Safety , 1997 , 57 ( 1 ): 49 - 69 .

刘松 , 佘敦先 , 张利平 , 等 . 基于Morris和Sobol的水文模型参数敏感性分析 [J ] . 长江流域资源与环境 , 2019 , 28 ( 6 ): 1296 - 1303 .

LIU S , SHE D X , ZHANG L P , et al . Global sensitivity analysis of hydrological model parameters based on Morris and Sobol methods [J ] . Resources and Environment in the Yangtze Basin , 2019 , 28 ( 6 ): 1296 - 1303 .

BENNETT N D , CROKE B F W , GUARISO G , et al . Characterising performance of environmental models [J ] . Environmental Modelling ＆ Software , 2013 , 40 : 1 - 20 .

BAE S , SEO D . Analysis and modeling of algal blooms in the Nakdong River,Korea [J ] . Ecological Modelling , 2018 , 372 : 53 - 63 .

NASH J E , SUTCLIFFE J V . River flow forecasting through conceptual models part I—a discussion of principles [J ] . Journal of Hydrology , 1970 , 10 ( 3 ): 282 - 290 .

GHAITH M , LI Z . Propagation of parameter uncertainty in SWAT:a probabilistic forecasting method based on polynomial chaos expansion and machine learning [J ] . Journal of Hydrology , 2020 , 10 : 124854

任婷玉 , 梁中耀 , 刘永 , 等 . 基于贝叶斯优化的三维水动力-水质模型参数估值方法 [J ] . 环境科学学报 , 2019 , 39 ( 6 ): 2024 - 2032 .

REN T Y , LIANG Z Y , LIU Y , et al . The parameters estimation method based on Bayesian optimization for complex water quality models [J ] . Acta Scientiae Circumstantiae , 2019 , 39 ( 6 ): 2024 - 2032 .

RAZAVI S , TOLSON B A , BURN D H . Review of surrogate modeling in water resources [J ] . Water Resources Research , 2012 , 48 ( 7 ).

SIMPSON T W , MAUERY T M , KORTE J J , et al . Kriging models for global approximation in simulation-based multidisciplinary design optimization [J ] . AIAA Journal , 2001 , 39 ( 12 ): 2233 - 2241 .

DOWLA F U , ROGERS L L . Solving problems in environmental engineering and geosciences with artificial neural networks [M ] . Cambridge : MIT Press , 1995 .

REGIS R G , SHOEMAKER C A . A stochastic radial basis function method for the global optimization of expensive functions [J ] . INFORMS Journal on Computing , 2007 , 19 ( 4 ): 497 - 509 .

FEN C S , CHAN C C , CHENG H C . Assessing a response surface-based optimization approach for soil vapor extraction system design [J ] . Journal of Water Resources Planning and Management , 2009 , 135 ( 3 ): 198 - 207 .

ZHANG X S , SRINIVASAN R , LIEW M V . Approximating SWAT model using artificial neural network and support vector machine [J ] . JAWRA Journal of the American Water Resources Association , 2009 , 45 ( 2 ): 460 - 474 .

ZENG L Z , SHI L S , ZHANG D X , et al . A sparse grid based Bayesian method for contaminant source identification [J ] . Advances in Water Resources , 2012 , 37 : 1 - 9 .

BOYLE J S , KLEIN S A , LUCAS D D , et al . The parametric sensitivity of CAM5’s MJO [J ] . Journal of Geophysical Research:Atmospheres , 2015 , 120 ( 4 ): 1424 - 1444 .

LIANG Z Y , ZOU R , CHEN X , et al . Simulate the forecast capacity of a complicated water quality model using the long short-term memory approach [J ] . Journal of Hydrology , 2020 , 581 : 124432

Views

544

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Cluster computing mode for water environment simulation based on Hadoop

A scalable parallel sorting algorithm by regular sampling for big data

System performance optimization practice for big data scenarios

Gansu smart tourism system based on big data

Big data and computing models

Related Author

Jinfeng MA

Li TANG

Kaifeng RAO

Gang HONG

Mei MA

Ying WANG

Zhiguang CHEN

Yutong LU

Related Institution

Key Laboratory of Drinking Water Science and Technology,Research Center for Eco-Environmental Sciences,Chinese Academy of Sciences

Shenzhen Environment Monitoring Center

Shijiazhuang Environmental Information Center

College of Resources and Environment University of Chinese Academy of Sciences

China Mobile Information Technology Center

Postal code：100079
Tel：（010）53879208、53859533 Email：bdr@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：273162 Visits Today：68

⁰