diff --git a/docs/build/doctrees/environment.pickle b/docs/build/doctrees/environment.pickle index a96f5822..c78ed9ae 100644 Binary files a/docs/build/doctrees/environment.pickle and b/docs/build/doctrees/environment.pickle differ diff --git a/docs/build/doctrees/index.doctree b/docs/build/doctrees/index.doctree index f02603bf..7d232f7e 100644 Binary files a/docs/build/doctrees/index.doctree and b/docs/build/doctrees/index.doctree differ diff --git a/docs/build/html/_sources/index.rst.txt b/docs/build/html/_sources/index.rst.txt index 16b264b4..689e6249 100644 --- a/docs/build/html/_sources/index.rst.txt +++ b/docs/build/html/_sources/index.rst.txt @@ -14,6 +14,12 @@ Addressing Uncertainty in MultiSector Dynamics Research 1_introduction 2_diagnostic_modeling_overview_and_perspectives 3_sensitivity_analysis_the_basics + 4_sensitivity_analysis_diagnostic_and_exploratory_modeling + 5_uncertainty_quantification_the_basics + 6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes + 7_conclusion + 8_references + 9_glossary Indices and tables diff --git a/docs/build/html/genindex.html b/docs/build/html/genindex.html index d4d9823f..a5969186 100644 --- a/docs/build/html/genindex.html +++ b/docs/build/html/genindex.html @@ -137,6 +137,12 @@

Navigation

  • Introduction
  • Diagnostic Modeling Overview and Perspectives
  • Sensitivity Analysis: The Basics
  • +
  • Sensitivity Analysis: Diagnostic & Exploratory Modeling
  • +
  • Uncertainty Quantification: The Basics
  • +
  • Uncertainty Quantification: A Tool For Capturing Risks & Extremes
  • +
  • Conclusion
  • +
  • References
  • +
  • Glossary
  • diff --git a/docs/build/html/index.html b/docs/build/html/index.html index dca389aa..9cc81942 100644 --- a/docs/build/html/index.html +++ b/docs/build/html/index.html @@ -70,6 +70,49 @@

    Addressing Uncertainty in MultiSector Dynamics ResearchSoftware Toolkits +
  • Sensitivity Analysis: Diagnostic & Exploratory Modeling +
  • +
  • Uncertainty Quantification: The Basics +
  • +
  • Uncertainty Quantification: A Tool For Capturing Risks & Extremes +
  • +
  • Conclusion
  • +
  • References
  • +
  • Glossary
  • @@ -103,6 +146,12 @@

    Navigation

  • Introduction
  • Diagnostic Modeling Overview and Perspectives
  • Sensitivity Analysis: The Basics
  • +
  • Sensitivity Analysis: Diagnostic & Exploratory Modeling
  • +
  • Uncertainty Quantification: The Basics
  • +
  • Uncertainty Quantification: A Tool For Capturing Risks & Extremes
  • +
  • Conclusion
  • +
  • References
  • +
  • Glossary
  • diff --git a/docs/build/html/objects.inv b/docs/build/html/objects.inv index 9924fc1e..9b824ac9 100644 Binary files a/docs/build/html/objects.inv and b/docs/build/html/objects.inv differ diff --git a/docs/build/html/py-modindex.html b/docs/build/html/py-modindex.html index 8cbe9f87..e77907b4 100644 --- a/docs/build/html/py-modindex.html +++ b/docs/build/html/py-modindex.html @@ -88,6 +88,12 @@

    Navigation

  • Introduction
  • Diagnostic Modeling Overview and Perspectives
  • Sensitivity Analysis: The Basics
  • +
  • Sensitivity Analysis: Diagnostic & Exploratory Modeling
  • +
  • Uncertainty Quantification: The Basics
  • +
  • Uncertainty Quantification: A Tool For Capturing Risks & Extremes
  • +
  • Conclusion
  • +
  • References
  • +
  • Glossary
  • diff --git a/docs/build/html/search.html b/docs/build/html/search.html index fc5d27cf..0186f26f 100644 --- a/docs/build/html/search.html +++ b/docs/build/html/search.html @@ -86,6 +86,12 @@

    Navigation

  • Introduction
  • Diagnostic Modeling Overview and Perspectives
  • Sensitivity Analysis: The Basics
  • +
  • Sensitivity Analysis: Diagnostic & Exploratory Modeling
  • +
  • Uncertainty Quantification: The Basics
  • +
  • Uncertainty Quantification: A Tool For Capturing Risks & Extremes
  • +
  • Conclusion
  • +
  • References
  • +
  • Glossary
  • diff --git a/docs/build/html/searchindex.js b/docs/build/html/searchindex.js index 5b825655..b33cfded 100644 --- a/docs/build/html/searchindex.js +++ b/docs/build/html/searchindex.js @@ -1 +1 @@ -Search.setIndex({docnames:["1_introduction","2.1_overview_of_model_diagnostics","2.2_perspectives_on_diagnostic_model_evaluation","2_diagnostic_modeling_overview_and_perspectives","3.1_global_versus_local_sensitivity","3.2_why_perform_sensitivity_analysis","3.3_sensitivity_analysis_applications_for_model_evaluation_and_fidelity_testing","3.4_sensitivity_analysis_applications_for_exploratory_modeling_and_scenario_discovery","3.5.1_one_at_a_time_oat","3.5.2_full_fractional_factorial_sampling","3.5.3_latin_hypercube_sampling","3.5.4_low_discrepancy_sequences","3.5.5_other_types_of_sampling","3.5.6_synthetic_generation_of_input_time_series","3.5_design_of_experiments","3.6.1_derivative_based_methods","3.6.2_elementary_effect_methods","3.6.3_regression_based_methods","3.6.4_regional_sensitivity_analysis","3.6.5_variance_based_methods","3.6.6_analysis_of_variance_anova","3.6.7_moment_independent_density_based_methods","3.6_sensitivity_analysis_methods","3.7_how_to_choose_a_sensitivity_analysis_method_model_traits_and_dimensionality","3.8_software_toolkits","3_sensitivity_analysis_the_basics","examples","index","modules","nanites","nanites.tests","preface"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":3,"sphinx.domains.rst":2,"sphinx.domains.std":2,sphinx:56},filenames:["1_introduction.rst","2.1_overview_of_model_diagnostics.rst","2.2_perspectives_on_diagnostic_model_evaluation.rst","2_diagnostic_modeling_overview_and_perspectives.rst","3.1_global_versus_local_sensitivity.rst","3.2_why_perform_sensitivity_analysis.rst","3.3_sensitivity_analysis_applications_for_model_evaluation_and_fidelity_testing.rst","3.4_sensitivity_analysis_applications_for_exploratory_modeling_and_scenario_discovery.rst","3.5.1_one_at_a_time_oat.rst","3.5.2_full_fractional_factorial_sampling.rst","3.5.3_latin_hypercube_sampling.rst","3.5.4_low_discrepancy_sequences.rst","3.5.5_other_types_of_sampling.rst","3.5.6_synthetic_generation_of_input_time_series.rst","3.5_design_of_experiments.rst","3.6.1_derivative_based_methods.rst","3.6.2_elementary_effect_methods.rst","3.6.3_regression_based_methods.rst","3.6.4_regional_sensitivity_analysis.rst","3.6.5_variance_based_methods.rst","3.6.6_analysis_of_variance_anova.rst","3.6.7_moment_independent_density_based_methods.rst","3.6_sensitivity_analysis_methods.rst","3.7_how_to_choose_a_sensitivity_analysis_method_model_traits_and_dimensionality.rst","3.8_software_toolkits.rst","3_sensitivity_analysis_the_basics.rst","examples.rst","index.rst","modules.rst","nanites.rst","nanites.tests.rst","preface.rst"],objects:{"":{nanites:[29,0,0,"-"]},"nanites.model":{sum_ints:[29,1,1,""]},"nanites.tests":{test_model:[30,0,0,"-"]},"nanites.tests.test_model":{TestModel:[30,2,1,""]},"nanites.tests.test_model.TestModel":{test_sum_ints:[30,3,1,""]},nanites:{model:[29,0,0,"-"],tests:[30,0,0,"-"]}},objnames:{"0":["py","module","Python module"],"1":["py","function","Python function"],"2":["py","class","Python class"],"3":["py","method","Python method"]},objtypes:{"0":"py:module","1":"py:function","2":"py:class","3":"py:method"},terms:{"1":[0,2,3,12,14,25,26],"1000":[23,25],"1949":[14,25],"1960":[2,3,14,25],"1961":[9,14,25],"1967":[2,3,11,14,25],"1968":[11,14,25],"1973":[12,14,25],"1976":[11,14,25],"1977":[2,3],"1979":[10,14,25],"1980":[6,25],"1988":[13,14,25],"1990":[2,3],"1991":[0,12,14,25],"1992":[2,3],"1993":[2,3,5,10,14,25],"1994":[2,3,11,14,25],"1995":[11,14,25],"1996":[2,3],"1998":[2,3],"1999":[12,13,14,25],"1km":0,"2":[0,1,3,23,25],"2001":26,"2002":[2,3,6,12,14,25],"2002a":[23,25,26],"2003":[0,2,3],"2004":[4,6,25],"2006":[4,14,25],"2007":[4,25],"2008":[1,2,3,6,9,10,11,13,14,15,22,25],"2010":[2,3,4,5,6,8,11,14,25],"2011":[2,3,6,23,25],"2012":[2,3],"2013":[0,2,3,13,14,25],"2014":[0,2,3,4,6,14,25],"2015":[0,11,13,14,25],"2016":[0,4,6,12,13,14,15,22,25],"2017":[0,9,13,14,25],"2018":[0,2,3,13,14,25],"2019":[0,1,3,4,13,14,25],"2020":[0,2,3],"2020a":0,"25":[7,25],"2d":[23,25],"2k":[9,14,25],"3":[0,4,7,25],"32":[9,14,25],"4":[0,2,3,6,25],"43":[9,14,25],"5":[0,23,25],"6":[0,9,10,14,25],"64":[9,14,25],"7":0,"8":[23,25],"abstract":[2,3],"case":[0,2,3,4,6,9,14,23,25,30],"class":30,"do":[0,1,3,23,25],"final":[2,3,6,25],"function":[0,2,3,14,25,29],"import":[0,2,3,4,5,8,9,14,23,25,26],"int":29,"long":0,"new":[0,2,3,5,25],"public":[2,3,13,14,25],"return":[15,22,25,29,30],"short":0,"true":[2,3,6,25],"while":[0,5,6,8,10,12,13,14,25],A:[2,3,9,11,13,14,27],And:27,As:[0,1,2,3,6,7,9,14,25],At:[0,12,27],Be:26,For:[2,3,4,5,6,7,9,12,13,14,15,22,23,25],If:[2,3,4,9,14,25],In:[2,3,4,5,6,8,9,10,14,22,23,25],It:[0,5,8,13,14,25],Its:[14,25],Nearing:[2,3],No:[14,25],On:[2,3,5,25],One:[5,12,27],Such:[23,25],The:[0,1,2,3,6,7,8,9,10,11,12,13,14,23,26,27],There:[0,2,3,12,14,25],These:[0,2,3,11,14,15,22,25],To:[10,14,27],With:[10,14,25],_:[15,22,25],_i:[15,22,25],_static:26,abil:[0,2,3],about:[5,6,9,14,25,31],abov:[0,7,13,14,25],absenc:0,accept:[2,3],account:[4,9,14,25],accur:[0,5,7,25],accuraci:[2,3,7,25],achiev:[2,3],acknowledg:0,across:[0,2,3,8,9,10,14,25],activ:[14,25],ad:[23,25],adapt:[1,2,3],add:26,addit:[0,2,3,4,5,7,14,25],address:[0,1,2,3,10,14,23,25],advanc:[0,2,3,5,23,25],advantag:[11,14,25],affect:[0,14,15,22,25],afford:[9,14,23,25],aforement:[2,3],agenc:[2,3],agre:0,agreement:[2,3],agricultur:[2,3],aid:0,aim:[0,2,3,6,25],al:[0,1,2,3,4,6,9,10,11,12,13,14,15,22,23,25],all:[0,6,7,8,9,10,12,14,25],allow:[0,1,3,9,14,25],alon:26,along:[12,14,22,25],alreadi:[2,3],also:[0,2,3,5,6,9,10,12,14,22,23,25],altern:[0,1,2,3,4,5,9,14,25],alwai:[2,3],ambiti:0,among:[2,3],amount:[12,14,25],amplitud:[12,14,25],an:[0,1,2,3,4,5,9,10,14,25],analys:[0,1,2,3,4,6,23,25],analysi:[0,1,2,3,4,10,12,13,14,27],analyst:[7,9,14,23,25],analyt:[14,25],analyz:[8,14,25],anderson:[6,25],ani:[0,1,2,3,4,5,10,11,14,23,25,29],annoni:[4,8,14,25],anoth:[11,14,25],anova:27,answer:[0,1,3,14,25],appear:[2,3,5,25],appendic:0,appli:[4,5,6,8,9,14,22,23,25],applic:[0,2,3,4,5,12,14,23,27],apport:[4,25],approach:[0,2,3,4,8,11,12,13,14,25],appropri:[2,3,4,6,9,10,14,23,25],approxim:[11,14,25],ar:[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15,22,23,25],area:0,argu:[2,3],argument:[2,3],around:[0,4,15,22,25],arrai:[10,14,25],art:[0,3],ask:[23,25],aspect:[2,3],assess:[2,3,5,14,23,25],assum:[8,9,14,15,22,25],assumpt:[1,2,3,5,7,14,23,25],attain:[0,2,3,11,14,25],attract:[10,14,25],audienc:[],author:[2,3],avail:[0,2,3,13,14,25],awar:[6,23,25],axi:0,b:[4,6,9,14,25,29],back:[6,25],bank:[2,3,5,25],bar:[23,25],barla:[2,3],base:[0,2,3,4,5,6,14,23,27,30],baselin:[2,3],basi:[1,3,23,25],basic:[0,27],becaus:[2,3,10,14,23,25],becom:[2,3,6,14,25],beek:[13,14,25],been:[0,1,2,3,4,5,6,7,9,12,14,25],befor:[0,14,25],begin:0,behavior:[0,1,2,3,5,6,14,25],being:[2,3,9,10,14,23,25],below:[0,6,23,25],besid:[2,3],best:[2,3],better:[2,3,13,14,23,25],between:[0,1,2,3,4,9,10,11,14,23,25],beven:[2,3],beyond:[2,3,12,13,14,25],bhar:[13,14,25],bia:[9,14,25],bias:[4,10,14,25],binlei:[2,3],blog:26,bodi:[0,13,14,25],book:[],borgomeo:[13,14,25],borgonovo:[4,6,15,22,25],bottom:[7,25],boundari:[0,2,3],box:[9,14,25],bridg:0,briefli:0,broad:[4,25],broader:[2,3],broadli:0,bryant:[5,25],build:[2,3,12,14,25],burden:[6,9,14,25],burdensom:[9,14,25],c:[4,9,10,14,15,22,25],calcul:[14,25],calibr:[2,3,5,6,25],call:[23,25],camp:[2,3],campaign:[6,25],can:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,15,22,23,25],cancel:[2,3],candid:[9,14,25],cannot:[0,2,3,4,9,10,14,23,25],capabl:0,captur:[0,13,14,25],care:0,carefulli:0,carlo:[1,3,6,11,14,25],carpent:[2,3],cash:[2,3],categori:[4,5,25],caus:[2,3,5,6,14,25],caveat:[23,25],centenni:[2,3],central:0,centuri:[13,14,25],challeng:[0,2,3,8,14,25],chang:[0,2,3,5,6,7,8,13,14,25],chapter:0,character:[0,1,3,11,13,14,25],characterist:[2,3,14,25],chart:[1,3],cheap:[23,25],choic:[0,1,2,3,14,25],choos:27,chosen:[4,14,25],circl:0,cite:[13,14,25],citi:[14,25],claim:[0,2,3],clarifi:[14,25],classifi:[1,2,3],clear:[2,3,23,25],climat:[2,3,13,14,25],close:[2,3,14,25],cluster:[11,14,25],co:[0,2,3],collect:0,color:0,combin:[0,3,9,14,25],come:[1,3,13,14,25],commod:[13,14,25],common:[0,2,3,6,10,14,25],commonli:[2,3,5,9,14,25],commun:[0,2,3],compar:[2,3,9,10,14,25],comparison:[13,14,22,25],compensatori:[2,3],complet:0,complex:[0,2,3,4,5,6,14,23,25],complic:[2,3],compon:[0,5,6,25],compos:[12,14,25],compound:[0,13,14,25],comprehens:[9,14,25],comput:[0,2,3,5,6,8,9,14,23,25],computation:[9,14,23,25],concept:[0,2,3],conceptu:[0,1,3],conclud:0,conclus:[6,23,25],condit:[0,2,3,14,25],conduct:[2,3,4,14,25],confid:[2,3],confound:[9,14,25],confront:0,consequ:[2,3],consequenti:[1,2,3,5,7,13,14,25],consid:[0,2,3,4,5,6,7,9,13,14,23,25],consider:[4,25],consist:[0,1,3,14,25],constrain:[10,14,23,25],constraint:[14,25],consult:0,consum:[5,25],contain:[2,3],content:28,context:[2,3,5,6,13,14,25],contigu:0,continent:0,continu:[2,3,9,14,25],continuum:0,contour:[4,25],contrast:[2,3,4,25],contribut:[0,6,25,26],control:[1,3,5,10,14,25],conveni:[4,25],converg:[23,25],convers:[6,25],cook:0,core:[2,3,11,14,25],correctli:[2,3],correl:[10,13,14,25],correspond:[9,14,25],cost:[5,8,9,14,25],costli:[4,25],could:[0,6,14,25],council:0,coupl:[0,1,2,3,14,25],cover:0,coverag:[10,14,25],credenc:[5,25],credibl:[2,3],criteria:[11,14,25],criterion:[2,3],critic:[2,3,8,14,25],crucial:[6,25],cukier:[12,14,25],cut:[6,25],cutoff:[6,25],d:[9,10,14,23,25],dai:[2,3],dalal:[11,14,25],data:[0,1,2,3,5,14,25],deal:[9,14,25],decad:[2,3],decid:[9,14,25],decis:[0,1,2,3,5,7,25],decompos:[1,3],deem:[2,3],deep:[0,13,14,25],defens:[6,25],defici:[2,3],defin:[0,1,2,3,14,25],definit:[4,22,25],degre:[6,23,25],delib:3,deliv:[23,25],delv:0,demand:[6,7,13,14,25],denot:[6,12,14,25,26],densiti:27,departur:[2,3],depend:[0,2,3,4,6,10,23],deriv:[10,14,27],describ:[0,22,25],design:[0,6,9,10,11,27],desir:[2,3],despit:[14,25],detail:[0,5,6,12,14,22,25],determin:0,determinist:[0,2,3],develop:[0,2,3,5,12,13,14,25],deviat:[11,14,25],diagnos:[2,3],diagnost:[0,6,25,27],dietari:[7,25],differ:[0,2,3,4,5,8,12,14,25],difficult:0,dimens:[2,3,11,14,25],dimension:[2,3,4,10,14,27],direct:[1,3,5,12,14,25],disadvantag:[14,25],disciplin:0,disciplinari:0,discontinu:[23,25],discov:[0,2,3,5,7,14,25],discoveri:[0,5,27],discredit:[5,25],discrep:27,discret:[9,10,14,25],discuss:[0,4,13,14,22,25],dismiss:[2,3],distinguish:[0,2,3,10,14,25],distribut:[0,1,3,9,11,12,14,25],diverg:[2,3],divers:[0,10,14,25],divid:[10,14,25],doc:26,doe:[2,3,4,25],domain:[0,14,25],dominantli:[1,3],drawn:[5,14,25],driven:0,drought:[0,13,14,25],dubiou:[5,25],due:[4,25],dure:[14,25],dynam:[0,2,3],e:[0,1,2,3,4,5,6,7,8,9,11,13,14,25,26],e_:26,each:[0,2,3,4,6,8,9,10,12,14,22,23,25],earli:[2,3],eas:[8,14,25],ebook:26,econom:0,edward:[6,25],effect:[0,2,3,4,5,6,8,9,10,14,26,27],effici:[14,25],effort:[5,25],either:[1,3,5,14,25],eker:[2,3],elabor:[2,3,6,7,14,25],element:[0,14,25],elementari:27,elicit:[14,25],elimin:[6,11,14,25],elsawah:0,elsewher:[1,3],elucid:[6,25],embark:[14,25],embrac:[2,3],emerg:0,emphas:[0,2,3],empir:[1,2,3,4,25],empiric:[2,3],emploi:[0,2,3,11,14,25],enabl:0,encompass:0,encourag:[13,14,25],end:[14,25],energi:[0,2,3,7,13,14,25],engag:0,engin:0,enhanc:[1,2,3],enough:[7,25],ensembl:[0,1,2,3],ensur:[11,14,25],entir:[0,4,14,25],epistem:[9,14,25],equal:[2,3,10,14,25],equifin:[2,3],equival:[7,25],error:[0,2,3,7,11,14,25],especi:[0,2,3,13,14,25],establish:[0,2,3,5,7,25],estim:[9,11,13,14,23,25],et:[0,1,2,3,4,6,9,10,11,12,13,14,15,22,23,25],etc:[13,14,25],evalu:[0,5,7,14,15,22,23,27],even:[2,3,11,14,25],evenli:[11,14,25],event:[13,14,25],ever:[14,25],everi:[10,14,25],evid:[6,25],evolut:[0,2,3],evolutionari:[0,2,3],evolv:[0,2,3],exact:0,examin:[10,13,14,25],exampl:[0,2,3,6,7,9,10,12,13,14,23,25],execut:[6,25],exert:[14,25],exhibit:[0,2,3],exist:[0,12,13,14,25],expans:[2,3],expect:[26,30],expens:[23,25],experi:[1,2,3,4,6,9,27],experiment:[3,10,14,25],expert:[0,14,25],explain:[1,3,6,25],explanatori:[2,3],explicitli:[2,3],exploit:0,explor:[0,2,3,4,8,13,14,15,22,25],exploratori:[0,1,2,3,5,13,14,27],exponenti:[9,14,25],extend:[2,3,12,14,25],extens:[5,25],extract:[14,25],extrapol:[5,25],extrem:[0,9,13,14,25],face:[0,2,3,6,25],fact:[2,3],factor:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,25,26],factori:27,faith:[7,25],famili:[2,3],far:[11,12,14,25],fashion:[2,3],fast:[12,14,25],favor:[2,3],feasibl:[0,4,14,25],featur:[22,25],fed:[14,25],feedback:0,few:[0,14,25],fewer:[6,11,14,25],fidel:[0,2,3,5,7,27],field:[14,25],fig:[4,6,7,9,10,14,23,25],figur:[0,1,3,23,25],fill:[10,11,14,25],filter:[6,25],find:[2,3],finer:[2,3],finger:[2,3],first:[1,2,3,12,14,23,25,26],fisher:[14,25],fit:[2,3],fix:[6,7,8,12,14,25],flexibl:0,focu:[0,1,2,3,6,7,8,14,25],focus:[0,13,14,25],follow:[5,13,14,15,22,25,26],form:[0,1,2,3,5,14,25],formal:[0,1,2,3,14,25],formul:[2,3],forward:3,found:[2,3,5,7,14,25],foundat:[1,3,5,25],four:[9,14,25],fourier:[12,14,25],frac:[15,22,25,26],fraction:27,frame:[0,2,3,5,25],framework:[1,3,14,25],freeli:[6,25],from:[0,1,2,3,6,10,11,14,25],full:[0,27],fulli:[0,2,3],fundament:[14,25],funtowicz:[2,3],further:[0,2,3,6,13,14,25],furthermor:[2,3,23,25],futur:[1,2,3,6,25],g:[0,1,2,3,4,6,7,9,11,13,14,15,22,25],gap:[0,11,14,25],gaug:[5,6,25],gave:[2,3],gener:[0,1,3,4,9,10,11,12,27],get:[1,3],given:[0,6,8,14,23,25],global:[0,1,3,6,27],glossari:0,goal:[0,2,3,4,25],graphic:[23,25],greatest:[6,25],grei:0,group:[5,25],grow:[0,7,9,14,25],growth:[0,7,9,14,25],gsa:[1,3],guarante:[2,3,10,11,14,25],guid:[0,2,3],guidanc:0,gupta:[2,3],ha:[0,1,3,4,5,6,7,9,14,25],hadjimichael:3,haim:0,hamm:[4,25],hand:[5,14,23,25],hardli:[14,25],have:[0,2,3,4,5,6,8,9,12,13,14,23,25],heat:0,heavili:[0,4,25],helb:0,held:[2,3],help:[2,3,5,6,25],helton:[14,25],here:[0,1,2,3,23,25],herman:[13,14,25],herrera:[13,14,25],heterogen:[2,3],heurist:[2,3,5,25],high:[2,3,7,13,14,25],higher:[0,2,3,9,14,25],highli:[0,1,2,3,5,8,10,14,25],highlight:0,hing:[0,7,25],histor:[0,2,3,4,25],histori:[2,3,13,14,25],hoc:[23,25],hole:[14,25],horizont:0,hornberg:[6,25],host:[5,25],hot:[13,14,25],how:[0,1,2,3,4,8,14,15,22,27],howev:[2,3,4,7,12,14,25],human:[0,1,2,3,14,25],hunter:[9,14,25],hydrolog:[2,3],hypercub:[9,11,27],hypothes:[0,2,3],hypothesi:[0,1,3],i:[2,3,5,6,8,11,14,15,22,25,26],ideal:[0,1,3,14,25],identifi:[1,2,3,5,6,25],ignor:[2,3],illumin:[6,25],illustr:[0,1,3,4,25],im3:[0,2,3],immedi:[13,14,25],impact:[0,13,14,25],implement:[2,3,8,14,25],impli:0,implic:0,imposs:[4,6,14,25],improv:[0,1,3,6,7,11,14,25],includ:[0,2,3,4,12,14,25],inconsequenti:[7,25],incorpor:[13,14,25],increas:[0,6,7,8,9,14,23,25],increment:[2,3],inde:[9,14,25],independ:[1,3,8,14,23,27],index:[15,22,23,25,26,27],indic:[4,6,14,23,25,26],individu:[0,1,3,6,8,13,14,25,26],infer:[0,1,2,3,5,7,14,25],influenc:[0,1,2,3,4,8,14,25],influenti:[5,6,25],inform:[0,1,2,3,4,5,6,8,9,10,13,14,23,25,31],inher:[2,3],initi:[1,3],innov:[0,1,3],input:[0,1,2,3,4,6,12,15,22,27],inquiri:[2,3,5,25],insight:[0,2,3],inspir:[2,3],instal:[],instanc:[5,6,7,25],instead:[2,3,4,5,9,14,25],institution:[2,3],insuffici:[2,3,13,14,25],integ:29,integr:[0,1,3],intens:[23,25],interact:[0,1,2,3,4,6,8,9,14,23,25],interdepend:0,interest:[0,1,3,5,6,8,13,14,25],interpret:[2,3],intersect:[2,3],interv:[9,14,25],introduc:[14,25],introduct:27,invari:[23,25],invest:[5,25],investig:[5,25],iooss:[11,14,25],issu:[2,3],iter:[3,8,14,25],its:[0,1,2,3,4,5,6,8,9,14,25],itself:[6,25],jin:[11,14,25],joint:0,judg:[2,3],justif:[2,3],k:[9,14,25],keep:26,kei:[0,14,25],kept:[8,14,25],kind:[14,25],kirsch:[13,14,25],kleindorf:[2,3],know:[0,2,3,5,25],knowledg:[2,3],known:[0,2,3,6,25],kucherenko:[11,14,25],kumar:[2,3],kwakkel:0,laboratori:[14,25],lamontagn:[13,14,25],land:0,landscap:0,languag:[7,25],larg:[0,2,3,4,13,14,25],larger:[2,3,10,14,25],largest:[6,25],later:[0,5,14,25],latest:26,latin:[9,27],lead:[2,3,6,7,25],learn:[2,3],least:[5,23,25],leav:[14,25],left:[6,25],legitimaci:[2,3],lempert:[5,25],lend:[5,25],length:[14,25],less:[10,14,25],let:[15,22,25],level:[0,2,3,9,10,11,14,23,25],lh:[11,27],like:[5,6,23,25],likelihood:0,limit:[0,2,3,4,5,8,9,11,13,14,22,23,25],linear:[0,4,14,23,25],linearli:[8,14,25],literatur:[0,2,3,4,6,9,12,13,14,25],local:[0,6,8,14,15,22,27],locat:[8,14,25],longer:[5,25],look:3,loss:[2,3],lost:[2,3],lot:0,louck:[13,14,25],low:[2,3,27],lower:[9,11,14,25],lumpi:[11,14,25],m:[4,25],made:[2,3,6,14,25],magnitud:[23,25],mai:[0,2,3],main:[8,9,14,25],major:[5,25],make:[0,2,3,6,7,9,14,23,25,30],mandat:[23,25],mani:[0,2,3,5,14,25],manner:[5,6,8,10,14,25],map:[0,4,6,7,14,25],massmann:[14,25],mathemat:[11,14,15,22,25],maxim:[10,14,25],mckai:[10,14,25],mean:[1,3,11,14,25],meant:0,measur:[2,3,5,6,9,11,14,15,22,23,25],mechan:[2,3],medda:[13,14,25],membership:0,mention:[7,12,14,25],met:[2,3],method:[0,2,3,4,6,8,10,11,12,13,14,27],methodnam:30,methodolog:[0,2,3,14,25],metric:[1,2,3,5,6,7,8,14,25],metropoli:[14,25],might:[5,13,14,25],milli:[13,14,25],mimick:[2,3],mind:[7,13,14,25],mine:[14,25],minim:[11,14,25],minimum:[0,6,14,23,25],mirror:[2,3],miscalcul:[4,25],miss:[8,14,23,25],mitchel:[11,14,25],mitig:[14,25],moallemi:0,mode:[0,2,3,6,25],model:[0,4,5,8,9,11,13,14,15,22,26,27,28],modif:[10,14,25],modifi:[10,14,25],modul:[27,28],moment:27,monetari:[5,25],mont:[1,3,6,11,14,25],montgomeri:[9,14,25],more:[0,1,2,3,4,6,7,8,9,12,14,23,25],morri:[11,12,14,25],most:[1,2,3,4,5,6,7,10,11,14,22,23,25],motiv:[5,14,23,25],move:[12,14,25],msd:0,msd_uncertainty_ebook:26,much:[10,11,14,25],multi:[0,13,14,25],multidimension:[11,14,25],multipl:[2,3,8,14,23,25],multiscal:[0,2,3],multisector:[0,2,3],multitud:[0,2,3],must:[2,3],my:26,n:[4,9,10,12,14,23,25],narr:0,nation:0,natur:[0,1,2,3,14,25],naylor:[2,3],nb:[6,25],necessari:[0,1,2,3,6,9,14,25],necessarili:[2,3],need:[0,1,2,3,7,14,25],neg:[14,25],negat:[8,14,25],neglect:[2,3],neglig:[6,9,14,25],negoti:[2,3],never:[2,3,14,25],next:[4,25],nexu:0,nk:[9,14,25],nomin:[5,6,14,15,22,25],non:[0,6,13,14,25],nonlinear:[4,8,14,23,25],nor:[0,14,25],norton:[14,25],nossent:[23,25],notat:[15,22,25],note:0,novel:0,now:[7,25],nowak:0,number:[0,2,3,6,9,10,11,12,14,23,25],numer:[0,2,3,4,5,6,14,25],oat:[15,22,27],object:[0,6,25],observ:[0,2,3,6,13,14,23,25],obtain:[9,14,25,26],occur:[2,3],often:[2,3,4,13,14,25],onc:[12,14,25],one:[1,2,3,5,6,8,9,10,11,12,14,15,22,25],onli:[2,3,4,8,9,14,15,22,23,25],open:0,opinion:[0,14,25],opportun:0,oppos:0,optim:[11,14,25],order:[6,9,14,23,25,26],oresk:[2,3],organ:[5,25],orient:[2,3],origin:[1,3],orthogon:[10,14,25],other:[2,3,4,5,6,8,9,23,27],otherwis:[4,25],our:[0,1,2,3,5,25],out:[2,3,4,25],outcom:[0,1,3,5,6,25],output:[0,1,2,3,4,5,6,7,8,11,14,15,22,25,26],over:[14,25],overcom:[8,14,25],overlap:[10,14,25],overview:[0,23,27],p:[9,14,25],packag:28,page:27,pair:[5,25],panel:[4,7,25],paramet:[0,1,2,3,4,5,8,9,10,14,25,29],parameter:[1,2,3,5,25],parametr:[1,3,4,25],park:[11,14,25],partial:[4,15,22,25],particular:[1,3,14,15,22,25],particularli:0,past:[13,14,25],path:[0,2,3],per:[2,3,9,14,25],percent:26,perfectli:[2,3],perform:[0,1,2,3,4,7,9,14,15,22,23,27],persist:[13,14,25],perspect:[0,7,25,27],pertin:[5,25],perturb:[4,12,14,15,22,25],physic:[2,3,9,14,25],pianosi:[6,12,14,15,22,25],pie:[1,3],pinpoint:[6,25],pip:31,place:[0,2,3,7,25],placehold:29,platform:[14,25],plausibl:[14,25],plischk:[4,25],plot:[0,4,25],plot_experimental_design:26,plural:[2,3],point:[2,3,4,11,12,14,25],popul:[0,7,14,25],popular:[8,14,25],pose:0,positivist:[2,3],possibl:[2,3,6,9,14,25],potenti:[0,2,3,9,11,13,14,23,25],power:[0,2,3],practic:[2,3,4,25],practition:[2,3],pre:[6,25],precis:[2,3],predict:[2,3],prefac:[],prefer:[2,3,4,25],premis:[2,3],prescrib:3,presenc:0,present:[0,1,2,3,4,5,6,9,14,23,25],preval:[2,3],previous:[7,25],price:[13,14,25],primari:0,primarili:[2,3,4,8,14,25],prior:0,priori:[14,25],priorit:[6,7,25],probabilist:0,probabl:0,problem:[6,9,14,25],procedur:[2,3,14,25],process:[0,2,3,5,6,8,9,13,14,25],produc:[2,3,4,5,7,9,14,23,25],program:26,progress:[2,3],project:[0,2,3],promot:0,propag:[1,3],properli:[4,25],properti:[0,5,11,14,25],propos:[2,3,4,6,9,14,25],proven:[4,23,25],provid:[0,1,2,3,4,13,14,22,25],prudent:[23,25],pseudo:[11,14,25],pure:[2,3],purpos:[2,3,5,13,14,23,25],put:0,python:[],qualit:[2,3],qualiti:[2,3],quantif:0,quantifi:[1,3],quantit:[0,2,3,11,14,25],quasi:[11,14,25],question:[0,1,2,3,5,14,23,25],r:[12,14,25],rais:[1,3],rakovec:[4,25],random:[11,14,25],randomli:[10,12,14,25],rang:[0,2,3,4,6,9,14,25],rank:[6,25],rapid:0,rapidli:0,rare:[13,14,25],rather:[2,3,7,25],reader:[2,3,6,12,13,14,23,25],readili:[14,25],real:[0,5,7,14,25],realiti:[2,3],reason:[2,3,5,8,9,13,14,25],recal:[23,25],recent:[9,14,25],recogn:[2,3],record:[13,14,25],reduc:[1,2,3,5,6,9,14,25],reduct:[6,25],refer:[0,4,6,9,12,14,25],refin:[5,25],reflect:[0,2,3,7,14,23,25],regard:[23,25],regardless:[5,25],region:[6,7,27],regress:[23,27],rel:[0,2,3,5,25],relat:[0,2,3,14,25],relationship:[0,1,3,4,14,23,25],relativist:[2,3,7,25],relev:[0,2,3,5,7,25],remain:[6,25],remark:0,repeat:[8,14,25],repetit:[12,14,25],report:[0,14,25],repres:[0,2,3,5,6,13,14,25],represent:[0,1,2,3,5,7,25],requir:[0,1,2,3,23,25],research:[0,5,14,23,25],resili:[0,1,2,3],resolut:[0,2,3,14,25],resourc:[5,25],respect:26,respond:0,respons:[0,1,3,4,7,14,25],rest:[14,25],result:[0,1,2,3,4,5,6,9,10,11,14,25],reveal:[4,5,25],review:[0,13,14,25],rich:[1,3,13,14,25],right:[2,3,7,25],rigor:0,rise:[2,3],risk:[0,6,25],run:[1,3,6,7,9,14,25],runtest:30,s:[0,1,2,3,4,5,6,7,8,14,15,22,25],s_i:[15,22,25,26],sa:[5,25],salienc:[2,3],saltelli:[0,1,2,3,4,6,8,9,10,12,14,23,25,26],same:[2,3,5,9,10,11,14,23,25],sampl:[1,3,8,11,15,22,23,27],scale:[0,2,3,14,15,22,23,25],scenario:[0,2,3,5,13,14,27],scheme:[8,9,10,14,25],schepdael:[14,25],scienc:[0,2,3],scientif:[0,2,3,14,25],scope:[0,2,3,13,14,25],screen:[6,25],script:0,se:[2,3],search:27,second:[1,2,3,23,25],section:[0,2,3,6,7,13,22,23],seed:[14,25],seek:[0,2,3],seen:[2,3],select:[2,3,8,12,14,23,25],sensit:[0,1,2,3,8,10,12,13,14,15,26,27],sensor:[9,14,25],sequenc:[8,27],seri:27,set:[0,1,2,3,4,5,6,14,23,25],sever:[2,3,4,12,13,14,23,25],shape:[0,5,6,14,25],shift:[2,3],shock:0,should:[0,2,3,4,6,14,23,25],show:[6,25],showcas:0,shown:[0,1,3],side:[2,3],signatur:[2,3],signific:[0,2,3,5,6,25],significantli:[5,9,14,25],sim_i:26,similar:[2,3,8,14,25],simplic:[15,22,25],simplif:[1,3,5,25],simplifi:[2,3,5,23,25],simul:[1,2,3,9,11,14,25],singl:[2,3,15,22,25],singular:0,situat:[0,2,3,6,25],size:[11,23],small:[4,5,11,14,25],smallest:[6,25],so:[2,3,5,6,12,14,25],sobol:[11,12,14,23,25,26],softwar:27,solut:26,some:[0,5,8,14,22,25],someth:[11,14,25],sophist:[6,8,14,25],sourc:[0,1,3,4,25,26],space:[0,1,2,3,4,6,8,9,10,11,12,14,25],span:[0,2,3],spatial:[2,3,13,14,25],spear:[6,25],specif:[0,1,2,3,4,5,6,12,14,25],specifi:0,stabl:[5,23,25],stai:[2,3],stakehold:[2,3,7,25],start:[12,14,25],state:[0,2,3,7,14,25],statement:[23,25],stationar:[13,14,25],statist:[0,1,2,3,11,13,14,25],steding:[13,14,25],steinschneid:[13,14,25],stem:[2,3],step:[1,3],still:[2,3],stochast:[13,14,25],strategi:[9,12,14,25],stratifi:[14,25],streamflow:[13,14,25],stressor:[0,2,3,13,14,25],strict:[7,25],strong:[0,13,14,25],structur:[0,1,2,3,5,9,14,25],studi:[0,4,5,13,14,25],style:[11,14,25],styliz:[6,25],subinterv:[10,14,25],subject:[2,3],submodul:28,subpackag:28,subsequ:[6,7,10,12,14,23,25],subspac:[11,14,25],substanti:[14,25],suffici:[0,2,3,5,7,13,14,25],suggest:[6,25],suit:[13,14,25],sum:29,sum_int:[29,30],summar:[14,25],summari:0,support:[0,2,3],sure:[26,30],surowiec:[9,14,25],surround:[5,25],survei:0,synthes:0,synthesi:[0,23,25],synthet:27,system:[0,1,2,3,5,7,8,14,23,25],tabl:[14,25],take:[0,11,14,25],tang:[4,10,14,25],tarantola:[6,25],target:31,task:0,team:[0,9,14,25],technic:[0,2,3],techniqu:[3,4,5,6,8,10,11,12,14,23,25],technolog:0,temperatur:[7,9,14,25],tempor:[2,3,13,14,25],tend:[7,25],tenuou:[2,3],term:[0,2,3,4,7,25],test:[0,1,2,3,5,9,12,14,27,28,29],test_model:[28,29],test_sum_int:30,testcas:30,testmodel:30,text:[0,2,3,4,5,13,14,25],th:[15,22,25],than:[0,2,3,7,9,10,14,25],thei:[0,2,3,4,5,9,13,14,23,25],them:[5,6,7,14,25],themselv:[1,3],theoret:0,theori:[0,2,3],therefor:[2,3,4,5,6,14,25],thi:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,22,23,26],those:[0,1,2,3,4,25],though:[2,3,11,14,25],three:[6,9,14,25],threshold:[0,6,25],through:[1,2,3,10,11,14,25],throughout:0,time:[12,23,27],togeth:[0,1,3],toler:[6,25],tool:[0,1,2,3,6,13,14,25],toolkit:27,top:[7,25],topic:[0,13,14,25],total:[6,9,14,23,25],toulmin:[2,3],toward:[2,3],trace:[6,25],track:26,tradit:[0,2,3],tradition:[9,14,25],trait:27,trajectori:[12,14,25],transdisciplinari:0,transit:[0,2,3],translat:0,treat:[9,13,14,25],treatment:[0,2,3,4,14,25],trivial:0,truth:[2,3],turn:[14,23,25],two:[1,2,3,4,5,9,14,25,29],type:[0,2,3,6,27],typic:[0,9,14,23,25],uc:[0,1,3],ulam:[14,25],ultim:[8,14,25],unaccept:[7,25],uncertain:[0,1,2,3,4,5,6,8,9,10,12,14,23,25],uncertainti:[0,1,2,3,4,5,6,9,13,14,25],under:[0,2,3,7,14,25],underestim:[4,25],underli:[0,2,3],underlin:[15,22,25],undermin:[2,3],understand:[0,2,3,5,23,25],understood:0,undesir:[2,3,5,6,7,25],uniform:[9,11,14,25],uniformli:[11,12,14,25],unimport:[5,25],uniqu:[2,3],unit:0,unittest:30,unnecessari:[6,25],unpreced:[2,3],up:[2,3,14,25],uq:[0,14,25],urban:0,us:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,25],usual:[0,6,12,14,25],util:[0,2,3,11,14,25],v:[6,25,26],v_:26,valid:[2,3,4,25],valu:[0,2,3,4,5,6,7,9,14,15,22,23,25,26,30],valuabl:[13,14,25],van:[14,25],vari:[0,4,6,14,23,25,26],variabl:[2,3,4,6,9,10,11,14,25],varianc:[6,11,14,26,27],variat:[12,14,25],varieti:[2,3,6,12,13,14,25],variou:[0,1,2,3,11,14,25],vast:[2,3],vector:[15,22,25],veri:[2,3,10,13,14,25],verif:[2,3],versu:[6,13,14,27],vertic:0,view:[0,2,3,7,25],viewpoint:[2,3],vogel:[13,14,25],vs:[2,3],vulner:[0,1,2,3],w:0,wa:[14,25],wai:[2,3,5,6,14,25],walker:0,warrant:[5,25],water:[0,26],wave:0,we:[0,1,2,3,4,5,22,25,31],weather:[13,14,25],weaver:[2,3],well:[0,2,3,5,6,13,14,22,25],were:[2,3,14,25],what:[1,3,23,25],when:[0,1,2,3,5,6,7,9,10,11,13,14,23,25],whenev:[4,25],where:[0,1,2,3,4,9,12,14,15,22,23,25],wherea:[2,3],whether:[2,3],which:[0,1,2,3,4,5,6,7,8,11,14,23,25],white:[2,3],why:[1,3,27],wide:[0,2,3,4,9,11,14,22,25],wilbi:[13,14,25],wilk:[13,14,25],wirtz:0,within:[0,2,3,4,6,9,11,14,25],without:[4,10,14,25],word:[4,9,14,23,25],workflow:0,world:[0,5,14,25],would:[0,6,7,8,14,23,25],wrong:[2,3],x1:[4,6,9,14,25],x2:[4,6,9,14,25],x3:[6,9,14,25],x:[4,6,14,15,22,25,26],x_i:26,xi:[6,25],xn:[4,25],y1:[4,25],y2:[4,25],y:[4,6,14,25,26],ye:[14,25],year:[13,14,25],yield:[1,3,5,25],ym:[4,25],you:[1,3],your:[5,25,26],zaremba:[11,14,25]},titles:["Introduction","Overview of model diagnostics","Perspectives on diagnostic model evaluation","Diagnostic Modeling Overview and Perspectives","Global Versus Local Sensitivity","Why Perform Sensitivity Analysis","Sensitivity Analysis Applications for Model Evaluation and Fidelity Testing","Sensitivity Analysis Applications for Exploratory Modeling and Scenario Discovery","One-At-a-Time (OAT)","Full and Fractional Factorial Sampling","Latin Hypercube (LH) Sampling","Low-Discrepancy Sequences","Other types of sampling","Synthetic generation of input time series","Design of Experiments","Derivative-based Methods","Elementary Effect Methods","Regression-based Methods","Regional Sensitivity Analysis","Variance-based Methods","Analysis of Variance (ANOVA)","Moment-Independent (Density-Based) Methods","Sensitivity Analysis Methods","How To Choose A Sensitivity Analysis Method: Model Traits And Dimensionality","Software Toolkits","Sensitivity Analysis: The Basics","Examples","Addressing Uncertainty in MultiSector Dynamics Research","nanites","nanites package","nanites.tests package","Preface"],titleterms:{A:[23,25],And:[23,25],At:[8,14,25],One:[8,14,25],The:25,To:[23,25],acknowledg:31,address:27,analysi:[5,6,7,18,20,22,23,25],anova:[20,22,25],applic:[6,7,25],audienc:31,base:[15,17,19,21,22,25],basic:25,book:31,build:26,caption:26,choos:[23,25],codeblock:26,content:[29,30],densiti:[21,22,25],depend:[14,25],deriv:[15,22,25],design:[14,25],diagnost:[1,2,3],dimension:[23,25],discoveri:[7,25],discrep:[11,14,25],dynam:27,ebook:[],effect:[16,22,25],elementari:[16,22,25],evalu:[2,3,6,25],exampl:26,experi:[14,25],exploratori:[7,25],factori:[9,14,25],fidel:[6,25],figur:26,fraction:[9,14,25],full:[9,14,25],gener:[13,14,25],global:[4,25],how:[23,25,31],hypercub:[10,14,25],independ:[21,22,25],indic:27,input:[13,14,25],insert:26,instal:31,introduct:0,latin:[10,14,25],lh:[10,14,25],local:[4,25],low:[11,14,25],mathjax:26,method:[15,16,17,19,21,22,23,25],model:[1,2,3,6,7,23,25,29],modul:[29,30],moment:[21,22,25],msd_uncertainty_ebook:31,multisector:27,nanit:[28,29,30],note:26,oat:[8,14,25],other:[12,14,25],out:26,overview:[1,3,14,25],packag:[29,30,31],perform:[5,25],perspect:[2,3],prefac:31,python:31,region:[18,22,25],regress:[17,22,25],research:27,sampl:[9,10,12,14,25],scenario:[7,25],section:[14,25],sensit:[4,5,6,7,18,22,23,25],sequenc:[11,14,25],seri:[13,14,25],size:[14,25],softwar:[24,25],submodul:[29,30],subpackag:29,summari:[14,25],synthet:[13,14,25],tabl:27,test:[6,25,26,30],test_model:30,thi:[14,25,31],time:[8,13,14,25],toolkit:[24,25],trait:[23,25],type:[12,14,25],uncertainti:27,us:[26,31],varianc:[19,20,22,25],versu:[4,25],why:[5,25]}}) \ No newline at end of file +Search.setIndex({docnames:["1_introduction","2.1_overview_of_model_diagnostics","2.2_perspectives_on_diagnostic_model_evaluation","2_diagnostic_modeling_overview_and_perspectives","3.1_global_versus_local_sensitivity","3.2_why_perform_sensitivity_analysis","3.3_sensitivity_analysis_applications_for_model_evaluation_and_fidelity_testing","3.4_sensitivity_analysis_applications_for_exploratory_modeling_and_scenario_discovery","3.5.1_one_at_a_time_oat","3.5.2_full_fractional_factorial_sampling","3.5.3_latin_hypercube_sampling","3.5.4_low_discrepancy_sequences","3.5.5_other_types_of_sampling","3.5.6_synthetic_generation_of_input_time_series","3.5_design_of_experiments","3.6.1_derivative_based_methods","3.6.2_elementary_effect_methods","3.6.3_regression_based_methods","3.6.4_regional_sensitivity_analysis","3.6.5_variance_based_methods","3.6.6_analysis_of_variance_anova","3.6.7_moment_independent_density_based_methods","3.6_sensitivity_analysis_methods","3.7_how_to_choose_a_sensitivity_analysis_method_model_traits_and_dimensionality","3.8_software_toolkits","3_sensitivity_analysis_the_basics","4.1_understanding_errors_what_is_controlling_model_performance","4.2_consequential_dynamics_what_is_controlling_model_behaviors_of_interest","4.3_consequential_scenarios_what_is_controlling_consequential_outcomes","4_sensitivity_analysis_diagnostic_and_exploratory_modeling","5.1_why_is_uncertainty_quantification_important_for_understanding_multisector_system_dynamics","5.2_uncertainty_quantification_for_exploratory_modeling","5.3_bayesian_uncertainty_quantification","5.4_uncertainty_quantification_under_deep_uncertainty","5.5_integrating_model_diagnostics_and_uncertainty_quantification","5_uncertainty_quantification_the_basics","6.1_understanding_risk_how_probable_are_extreme_events","6.2_understanding_tails_statistical_modeling_of_extreme_events","6.3_how_to_choose_an_appropriate_method","6.4_how_to_select_a_prior_distribution","6.5_posterior_predictive_checking","6.6_model_selection_and_comparison","6.7.1_scenario_discovery","6.7.2_pre_calibration_glue","6.7.3.1_metropolis_hastings","6.7.3.2_gibbs_sampling","6.7.3.3_hamiltonian_monte_carlo","6.7.3_markov_chain_monte_carlo","6.7.4_particle_based_methods","6.7_what_are_common_methods","6.8.1_markov_chain_monte_carlo_with_the_true_model","6.8.2_markov_chain_monte_carlo_with_surrogate_models","6.8_what_are_example_software_implementations","6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes","7_conclusion","8_references","9_glossary","examples","index","modules","nanites","nanites.tests","preface"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":3,"sphinx.domains.rst":2,"sphinx.domains.std":2,sphinx:56},filenames:["1_introduction.rst","2.1_overview_of_model_diagnostics.rst","2.2_perspectives_on_diagnostic_model_evaluation.rst","2_diagnostic_modeling_overview_and_perspectives.rst","3.1_global_versus_local_sensitivity.rst","3.2_why_perform_sensitivity_analysis.rst","3.3_sensitivity_analysis_applications_for_model_evaluation_and_fidelity_testing.rst","3.4_sensitivity_analysis_applications_for_exploratory_modeling_and_scenario_discovery.rst","3.5.1_one_at_a_time_oat.rst","3.5.2_full_fractional_factorial_sampling.rst","3.5.3_latin_hypercube_sampling.rst","3.5.4_low_discrepancy_sequences.rst","3.5.5_other_types_of_sampling.rst","3.5.6_synthetic_generation_of_input_time_series.rst","3.5_design_of_experiments.rst","3.6.1_derivative_based_methods.rst","3.6.2_elementary_effect_methods.rst","3.6.3_regression_based_methods.rst","3.6.4_regional_sensitivity_analysis.rst","3.6.5_variance_based_methods.rst","3.6.6_analysis_of_variance_anova.rst","3.6.7_moment_independent_density_based_methods.rst","3.6_sensitivity_analysis_methods.rst","3.7_how_to_choose_a_sensitivity_analysis_method_model_traits_and_dimensionality.rst","3.8_software_toolkits.rst","3_sensitivity_analysis_the_basics.rst","4.1_understanding_errors_what_is_controlling_model_performance.rst","4.2_consequential_dynamics_what_is_controlling_model_behaviors_of_interest.rst","4.3_consequential_scenarios_what_is_controlling_consequential_outcomes.rst","4_sensitivity_analysis_diagnostic_and_exploratory_modeling.rst","5.1_why_is_uncertainty_quantification_important_for_understanding_multisector_system_dynamics.rst","5.2_uncertainty_quantification_for_exploratory_modeling.rst","5.3_bayesian_uncertainty_quantification.rst","5.4_uncertainty_quantification_under_deep_uncertainty.rst","5.5_integrating_model_diagnostics_and_uncertainty_quantification.rst","5_uncertainty_quantification_the_basics.rst","6.1_understanding_risk_how_probable_are_extreme_events.rst","6.2_understanding_tails_statistical_modeling_of_extreme_events.rst","6.3_how_to_choose_an_appropriate_method.rst","6.4_how_to_select_a_prior_distribution.rst","6.5_posterior_predictive_checking.rst","6.6_model_selection_and_comparison.rst","6.7.1_scenario_discovery.rst","6.7.2_pre_calibration_glue.rst","6.7.3.1_metropolis_hastings.rst","6.7.3.2_gibbs_sampling.rst","6.7.3.3_hamiltonian_monte_carlo.rst","6.7.3_markov_chain_monte_carlo.rst","6.7.4_particle_based_methods.rst","6.7_what_are_common_methods.rst","6.8.1_markov_chain_monte_carlo_with_the_true_model.rst","6.8.2_markov_chain_monte_carlo_with_surrogate_models.rst","6.8_what_are_example_software_implementations.rst","6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes.rst","7_conclusion.rst","8_references.rst","9_glossary.rst","examples.rst","index.rst","modules.rst","nanites.rst","nanites.tests.rst","preface.rst"],objects:{"":{nanites:[60,0,0,"-"]},"nanites.model":{sum_ints:[60,1,1,""]},"nanites.tests":{test_model:[61,0,0,"-"]},"nanites.tests.test_model":{TestModel:[61,2,1,""]},"nanites.tests.test_model.TestModel":{test_sum_ints:[61,3,1,""]},nanites:{model:[60,0,0,"-"],tests:[61,0,0,"-"]}},objnames:{"0":["py","module","Python module"],"1":["py","function","Python function"],"2":["py","class","Python class"],"3":["py","method","Python method"]},objtypes:{"0":"py:module","1":"py:function","2":"py:class","3":"py:method"},terms:{"0":[54,55],"00":55,"0000287":55,"0000626":55,"0000701":55,"0000915":55,"00035":55,"00073":55,"00082":55,"001":55,"00121":55,"0015":55,"002":55,"0023":55,"00270":55,"00294":55,"003":55,"00353":55,"00401706":55,"0041":55,"0043":55,"0049124104268644":55,"005":55,"00553":55,"006":55,"0060":55,"007":55,"008":55,"009":55,"01":55,"010":55,"011":55,"012":55,"013":55,"0130":55,"014":55,"0143624417705937":55,"0148":55,"0156574":55,"01621459":55,"0167":55,"017":55,"0170052":55,"018":55,"01802":55,"01816001":55,"019":55,"0190115":55,"0190641":55,"02":55,"020":55,"02037":55,"021":55,"0223":55,"0234":55,"02443":55,"02508060":55,"026":55,"02802":55,"0290":55,"03":55,"030":55,"030234210x497726":55,"0309":55,"032":55,"033":55,"0378":55,"04":55,"04016050":55,"04018007":55,"04134":55,"0469":55,"0493":55,"05":[47,49,53,55],"05252":55,"057":55,"06":55,"074019":55,"07811":55,"0796":55,"08":55,"08019":55,"082081699":55,"0855":55,"0870":55,"09":55,"0921":55,"0986":55,"1":[0,2,3,12,14,25,47,49,53,54,55,57],"10":55,"100":55,"1000":[23,25],"1002":55,"1007":55,"1008":55,"1008935410038":55,"10143":55,"1015":55,"1016":55,"10162":55,"102009":55,"1021":55,"102186":55,"1023":55,"1026":55,"1029":55,"1038":55,"10409":55,"104551":55,"10475138":55,"10476423":55,"10476572":55,"10483310":55,"10484804":55,"10485594":55,"1049":55,"1061":55,"1063":55,"107":55,"1071":55,"1073":55,"1076":55,"108":55,"1080":55,"1087":55,"1088":55,"109":55,"1092":55,"1098":55,"1099":55,"11":55,"1100705":55,"1109":55,"111":55,"1111":55,"1115":55,"1116":55,"112":55,"1126":55,"113":55,"11362":55,"1137":55,"11373":55,"114":55,"1151915":55,"1153":55,"117":55,"1175":55,"1176344136":55,"1177":55,"1177011136":55,"1177729694":55,"119":55,"1198":55,"12":55,"1209":55,"12117":55,"1214":55,"123":55,"12348":55,"125":55,"126":55,"1268522":55,"127":55,"1273":55,"1278":55,"12804":55,"1287":55,"12888":55,"129":55,"1295":55,"13":55,"130":55,"1337":55,"1339":55,"1347":55,"135":55,"1354":55,"1355819619840672":55,"1360":55,"1371":55,"138":55,"139":55,"1392":55,"1397":55,"14":55,"140":55,"1400":55,"141":55,"1413":55,"1414":55,"142":55,"143":55,"1432":55,"144":55,"14420":55,"145":55,"1467":55,"1469":55,"14693062":55,"148":55,"1482":55,"1483":55,"1490":55,"1493":55,"15":55,"1508":55,"1509":55,"1510":55,"1517":55,"1520":55,"1539":55,"154":55,"155":55,"1593":55,"16":55,"1600":55,"161":55,"1619048":55,"162":55,"16226":55,"1623":55,"16466":55,"1657":55,"1658":55,"166":55,"1667":55,"1672":55,"1680571":55,"1699114":55,"16m1090648":55,"17":55,"1708":55,"172":55,"174":55,"1748":55,"1763":55,"18":[54,55],"180":55,"18174":55,"183":55,"1884324":55,"189":55,"19":55,"190":55,"1907826116":55,"191":55,"19188":55,"192":55,"194":55,"1943":55,"1949":[14,25,55],"1950":55,"1951":55,"1953":55,"1960":[2,3,14,25,55],"1961":[9,14,25,55],"1963":55,"1964":[35,55],"1967":[2,3,11,14,25,55],"1968":[11,14,25,55],"197":55,"1970":55,"1972":55,"1973":[12,14,25,55],"1974":55,"1975":55,"1976":[11,14,25,55],"1977":[2,3,35,55],"1978":55,"1979":[10,14,25,55],"1980":[6,25,55],"1984":55,"1988":[13,14,25,55],"1989":55,"1990":[2,3,55],"1991":[0,12,14,25,55],"1992":[2,3,47,49,53,55],"1993":[2,3,5,10,14,25,55],"1994":[2,3,11,14,25,55],"1995":[11,14,25,55],"1996":[2,3,55],"1998":[2,3,55],"1999":[12,13,14,25,55],"1_metropolis_hast":[],"1km":0,"2":[0,1,3,23,25,54,55],"20":55,"200":55,"2000":55,"2001":[55,57],"2002":[2,3,6,12,14,25,55],"2002a":[23,25,57],"2003":[0,2,3,55],"2004":[4,6,25,55],"2005":55,"2006":[4,14,25,55],"2007":[4,25,55],"2008":[1,2,3,6,9,10,11,13,14,15,22,25,47,49,53,55],"2008wr006822":55,"2009":55,"2010":[2,3,4,5,6,8,11,14,25,55],"2010wr009769":55,"2011":[2,3,6,23,25,47,49,53,55],"2011wr011044":55,"2012":[2,3,55],"20124":55,"2013":[0,2,3,13,14,25,35,54,55],"2013wr014063":55,"2014":[0,2,3,4,6,14,25,55],"2014wr016827":55,"2015":[0,11,13,14,25,55],"2016":[0,4,6,12,13,14,15,22,25,55],"2017":[0,9,13,14,25,55],"2017ef000607":55,"2017ef000701":55,"2017gl074606":55,"2018":[0,2,3,13,14,25,55],"2018ef001082":55,"2019":[0,1,3,4,13,14,25,35,55],"201907826":55,"2019ef001340":55,"2019wr024918":55,"202":55,"2020":[0,2,3,55],"2020a":[0,55],"2020a16226":55,"2020b":[35,55],"2020wr028079":55,"203":55,"2044":55,"208":55,"21":55,"210":55,"2112":55,"2127":55,"21296":55,"214":55,"2140":55,"2142":55,"215":55,"217":55,"219":55,"22":55,"220":55,"223":55,"228":55,"2289594":55,"23":55,"2304394":55,"2307":55,"231":55,"232":55,"2334940":55,"2344614":55,"235":55,"2351":55,"236":55,"2372":55,"239":55,"24":55,"240":55,"242":55,"245":55,"2465":55,"248":55,"2484":55,"25":[7,25,55],"250":55,"253":55,"260":55,"261":55,"263":55,"268":55,"27":55,"271":55,"279":55,"28":55,"280":55,"2823":55,"2839":55,"289":55,"29":55,"293":55,"297":55,"298":55,"2988185":55,"2_gibbs_sampl":[],"2d":[23,25],"2k":[9,14,25],"2nd":55,"3":[0,4,7,25,55],"30":55,"303":55,"304":55,"305":55,"307":55,"31":55,"311":55,"314":55,"3152":55,"319":55,"32":[9,14,25,55],"321":55,"327":55,"329":55,"33":55,"332":55,"335":55,"3360060305":55,"337":55,"3390":55,"34":55,"3402":55,"341":55,"3437":55,"35":55,"351":55,"355":55,"357":55,"3571":55,"3594":55,"366":55,"37":55,"370":55,"3758":55,"38":55,"3802":55,"381":55,"3813":55,"384":55,"3873":55,"3878":55,"3880":55,"3894":55,"39":55,"393":55,"395":55,"396":55,"3_hamiltonian_monte_carlo":[],"4":[0,2,3,6,25,55],"402":55,"406":55,"409":55,"41":55,"410":55,"411":55,"418":55,"423":55,"425":55,"426":55,"4260060203":55,"43":[9,14,25,55],"435":55,"436":55,"438":55,"44":55,"4419":55,"443":55,"449":55,"455":55,"457":55,"458":55,"4580163":55,"4603":55,"461":55,"4620":55,"464":55,"47":55,"4754":55,"4757":55,"47587":55,"4767596":55,"478":55,"48":55,"485":55,"48760":55,"488":55,"49":55,"494":55,"496":55,"497":55,"5":[0,23,25,55],"50":55,"504":55,"51":55,"511":55,"5147":55,"519":55,"5194":55,"520":55,"526":55,"53":55,"531":55,"5382":55,"5405":55,"545":55,"5452":55,"548":55,"55":55,"554":55,"555":55,"5553":55,"559":55,"56":55,"57":55,"573":55,"574":55,"583":55,"585":55,"589":55,"59":55,"591":55,"597":55,"6":[0,9,10,14,25,55],"60":55,"602":55,"617":55,"619":55,"62":55,"623":55,"627":55,"63":55,"639":55,"64":[9,14,25,55],"641":55,"642":55,"643":55,"644":55,"646":55,"6491":55,"6497":55,"65":55,"66":55,"669":55,"67":55,"68":55,"681":55,"689":55,"69":55,"6924":55,"6_5":55,"7":[0,55],"702":55,"709":55,"70928":55,"71":55,"7152":55,"716":55,"721":55,"723":55,"73":55,"7309":55,"7313":55,"733":55,"737":55,"74":55,"741":55,"75":55,"750":55,"75007":55,"751":55,"76":55,"760":55,"763":55,"77":55,"773":55,"778":55,"787":55,"79":55,"793":55,"795":55,"796":55,"7_1140":55,"7b00506":55,"8":[23,25,55],"80":55,"803":55,"8086":55,"8091":55,"811":55,"812":55,"817":55,"8317":55,"84":55,"85":55,"86":55,"863":55,"869":55,"879":55,"88":55,"887":55,"89":55,"8_13":55,"9":55,"90":55,"90028":55,"90040":55,"90115":55,"90144":55,"90172":55,"91":55,"9137":55,"92":55,"9269":55,"93":55,"9326":55,"94":55,"9405":55,"9416":55,"9458":55,"94jc00572":55,"95":55,"953":55,"954":55,"955":55,"95rg00287":55,"963":55,"9685623":55,"97":55,"971":55,"972":55,"978":55,"9780784480717":55,"979":55,"97wr03495":55,"98":55,"985":55,"9868":55,"99":55,"997":55,"9_1":55,"9_10":55,"abstract":[2,3,54],"b\u00fcrkner":55,"bob\u00e9":55,"case":[0,2,3,4,6,9,14,23,25,47,49,53,54,55,61],"class":61,"do":[0,1,3,23,25,47,49,53],"extr\u00eam":55,"final":[2,3,6,25],"function":[0,2,3,14,25,35,54,55,60],"gonz\u00e1lez":55,"gr\u00fcbler":55,"harremo\u00eb":55,"import":[0,2,3,4,5,8,9,14,23,25,55,57,58],"int":[55,60],"j\u00e4ger":55,"j\u00e9z\u00e9quel":55,"long":[0,47,49,53,55],"new":[0,2,3,5,25,55],"pr\u00e9cipit":55,"public":[2,3,13,14,25,55],"return":[15,22,25,55,60,61],"short":0,"th\u00e9oriqu":55,"transient":[47,49,53],"true":[2,3,6,25,58],"vikstr\u00f6m":55,"while":[0,5,6,8,10,12,13,14,25,47,49,53],A:[2,3,9,11,13,14,55,58],And:58,As:[0,1,2,3,6,7,9,14,25,35,47,49,53,54],At:[0,12,58],Be:[55,57],For:[2,3,4,5,6,7,9,12,13,14,15,22,23,25,47,49,52,58],If:[2,3,4,9,14,25],In:[2,3,4,5,6,8,9,10,14,22,23,25,35,49,53,54],Is:58,It:[0,5,8,13,14,25,47,49,53],Its:[14,25],Nearing:[2,3,55],No:[14,25,55],OF:55,ONE:55,On:[2,3,5,25,55],One:[5,12,47,49,53,55,58],Such:[23,25],THE:55,The:[0,1,2,3,6,7,8,9,10,11,12,13,14,23,47,49,52,53,54,55,57,58],Their:55,There:[0,2,3,12,14,25,49,52,53],These:[0,2,3,11,14,15,22,25],To:[10,14,58],With:[10,14,25],_:[15,22,25],_i:[15,22,25],_static:57,aacb3d:55,aba2423:55,abil:[0,2,3],abl:[47,49,53],about:[5,6,9,14,25,35,47,49,53,55,62],abov:[0,7,13,14,25],abramowitz:55,absenc:0,abusorrah:55,ac:55,acad:55,academi:55,accept:[2,3,55],account:[4,9,14,25],accur:[0,5,7,25],accuraci:[2,3,7,25],achiev:[2,3],acknowledg:0,across:[0,2,3,8,9,10,14,25,47,49,53,54,55],act:35,activ:[14,25,55],ad:[23,25],adapt:[1,2,3,55],add:[54,57],addit:[0,2,3,4,5,7,14,25,54,55],address:[0,1,2,3,10,14,23,25,54,55],adequaci:55,adlouni:55,adrian:55,adv:55,advanc:[0,2,3,5,23,25,54,55],advantag:[11,14,25,49,53],aert:55,affect:[0,14,15,22,25],affin:55,afford:[9,14,23,25],aforement:[2,3],agenc:[2,3],agent:55,aghakouchak:55,agre:0,agreement:[2,3,55],agricultur:[2,3],agronomi:55,aguilar:55,aic:55,aid:[0,55],aim:[0,2,3,6,25],akaik:55,al:[0,1,2,3,4,6,9,10,11,12,13,14,15,22,23,25,35,47,49,53,54],alabdulwahab:55,albrecht:55,alcock:55,aleksankina:55,algorithm:[47,49,53,55],all:[0,6,7,8,9,10,12,14,25,35,47,49,53,55],allei:55,allow:[0,1,3,9,14,25,55],alon:57,along:[12,14,22,25,55],alreadi:[2,3],also:[0,2,3,5,6,9,10,12,14,22,23,25,54],altern:[0,1,2,3,4,5,9,14,25,35,54,55],although:54,alwai:[2,3],am:55,ambigu:55,ambiti:[0,54],american:55,among:[2,3],amount:[12,14,25,47,49,53],amplifi:55,amplitud:[12,14,25],an:[0,1,2,3,4,5,9,10,14,25,52,54,55,58],anal:55,analchem:55,analys:[0,1,2,3,4,6,23,25,49,53,54,55],analysi:[0,1,2,3,4,10,12,13,14,49,53,54,55,58],analyst:[7,9,14,23,25],analyt:[14,25,55],analyz:[8,14,25,55],anderson:[6,25,55],andr:55,andrieu:55,ani:[0,1,2,3,4,5,10,11,14,23,25,60],ann:55,annan:55,annoni:[4,8,14,25,55],anomali:55,anoth:[11,14,25,47,49,53],anova:58,answer:[0,1,3,14,25],antarct:55,antarctica:55,ao:55,aom:55,appear:[2,3,5,25],appendic:[0,54],appl:55,appli:[4,5,6,8,9,14,22,23,25],applic:[0,2,3,4,5,12,14,23,54,55,58],apport:[4,25],apprais:55,approach:[0,2,3,4,8,11,12,13,14,25,47,49,52,53,54,55],appropri:[2,3,4,6,9,10,14,23,25,47,49,54,58],approxim:[11,14,25,47,49,53,55],ar:[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15,22,23,25,35,47,54,55,58],architectur:55,area:[0,47,49,53,55],argu:[2,3],argument:[2,3],around:[0,4,15,22,25],arrai:[10,14,25,55],art:[0,3,47,49,53],artifici:55,arxiv150502350:55,arxiv:55,asap:55,asc:55,ash:55,ask:[23,25],askei:55,asm:55,aspect:[2,3,55],asselt:55,assess:[2,3,5,14,23,25,47,49,52,53,55],assimil:55,assist:55,assoc:55,associ:[49,53],assum:[8,9,14,15,22,25],assumpt:[1,2,3,5,7,14,23,25,35],astro:55,asymptot:[47,49,53,55],athanasiadi:55,atlant:55,atmo:55,attain:[0,2,3,11,14,25],attract:[10,14,25],audienc:[],auer:55,author:[2,3],autocorrel:[47,49,53,55],autom:55,avail:[0,2,3,13,14,25],averag:[49,53],avoid:55,awar:[6,23,25],axi:0,axiom:55,axtel:55,b:[4,6,9,14,25,55,60],back:[6,25],bader:55,bai:55,bakker:55,baldassarr:55,bam:55,bangladesh:55,bank:[2,3,5,25,55],bar:[23,25],barla:[2,3,55],barro:55,base:[0,2,3,4,5,6,14,23,52,54,55,58,61],baselin:[2,3],basi:[1,3,23,25,55],basic:[0,58],basin:55,basto:55,batcheld:55,bathymetri:55,batouli:55,bawa:55,bay:55,bayesian:[52,53,55,58],becaus:[2,3,10,14,23,25],becker:55,beckman:55,becom:[2,3,6,14,25],beek:[13,14,25,55],been:[0,1,2,3,4,5,6,7,9,12,14,25,47,49,53,54],befor:[0,14,25],begin:[0,47,49,53],behavior:[0,1,2,3,5,6,14,25,35,54,55,58],being:[2,3,9,10,14,23,25],belitz:55,below:[0,6,23,25,47,49,52,53],bennett:55,berlin:55,besid:[2,3],best:[2,3,55],betancourt:55,better:[2,3,13,14,23,25,52,53,54,55],between:[0,1,2,3,4,9,10,11,14,23,25],bevacqua:55,beven:[2,3,55],beyond:[2,3,12,13,14,25,49,53,55],bhar:[13,14,25,55],bia:[9,14,25,47,49,53,55],bias:[4,10,14,25,55],bic:55,bick:55,bilir:55,binlei:[2,3,55],biologi:55,biomateri:55,biometrika:55,bloemen:55,blog:57,boca:55,bodi:[0,13,14,25],bone:55,book:[],borgomeo:[13,14,25,55],borgonovo:[4,6,15,22,25,55],both:[49,53,54],bottom:[7,25,55],botzen:55,boukouvala:55,boundari:[0,2,3,55],boussouf:55,bouwer:55,bouziota:55,box:[9,14,25,55],boyd:55,br:55,brandimart:55,bridg:0,briefli:0,broad:[4,25],broader:[2,3],broadli:[0,49,53,54],brook:55,broomel:55,brown:55,bryan:55,bryant:[5,25,55],buchanan:55,budescu:55,build:[2,3,12,14,25,52,53,55],built:[52,53],bull:55,burden:[6,9,14,25],burdensom:[9,14,25],burn:[47,49,53],burnham:55,butler:55,c:[4,9,10,14,15,22,25,52,53,55],cabrero:55,cageo:55,calcul:[14,25,55],calibr:[2,3,5,6,25,55,58],call:[23,25,47,49,53],calvin:55,cambridg:55,camco:55,cameron:55,camp:[2,3],campaign:[6,25],campolongo:55,can:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,15,22,23,25,35,47,49,53,54,55],cancel:[2,3],candid:[9,14,25],cannot:[0,2,3,4,9,10,14,23,25],cao:55,capabl:[0,54],captur:[0,13,14,25,54,58],carbon:55,care:[0,55],carefulli:0,cariboni:55,carlier:55,carlin:55,carlo:[1,3,6,11,14,25,55,58],carpent:[2,3,55],cash:[2,3,55],categori:[4,5,25,47,49,53],caus:[2,3,5,6,14,25],caveat:[23,25],centenni:[2,3],central:0,centuri:[13,14,25,55],chain:[55,58],challeng:[0,2,3,8,14,25,49,53,54,55],cham:55,chan:55,chanc:55,chang:[0,2,3,5,6,7,8,13,14,25,35,54,55],chao:55,chaotic:55,chapman:55,chapter:0,charackli:55,character:[0,1,3,11,13,14,25,35,54,55],characterist:[2,3,14,25],chart:[1,3],chatterje:55,che:55,cheap:[23,25],check:58,chem:55,chemic:55,chemistri:55,chen:55,chiaromont:55,chichest:55,choic:[0,1,2,3,14,25,35,47,49,53,54,55],choos:58,chosen:[4,14,25],ciccazzo:55,circl:0,circuit:55,circul:55,citat:54,cite:[13,14,25],citi:[14,25],civ:55,claim:[0,2,3],clarifi:[14,25,55],clark:55,classifi:[1,2,3,49,53],clear:[2,3,23,25],clim:55,climat:[2,3,13,14,25,55],close:[2,3,14,25],cluster:[11,14,25],co:[0,2,3,35,54,55],coast:55,coastal:55,code:[52,53,55],coeffici:55,coerc:55,coher:55,cole:55,colei:55,collabor:55,collect:[0,55],collin:55,color:0,combin:[0,3,9,14,25,55],come:[1,3,13,14,25],commod:[13,14,25],common:[0,2,3,6,10,14,25,52,58],commonli:[2,3,5,9,14,25],commun:[0,2,3,55],compar:[2,3,9,10,14,25,47,49,53,55],comparison:[13,14,22,25,35,55,58],compartment:55,compchemeng:55,compensatori:[2,3],complet:0,complex:[0,2,3,4,5,6,14,23,25,54,55],complic:[2,3],compon:[0,5,6,25],compos:[12,14,25],compound:[0,13,14,25,55],comprehens:[9,14,25,54,55],comput:[0,2,3,5,6,8,9,14,23,25,47,49,52,53,54,55],computation:[9,14,23,25],concentr:55,concept:[0,2,3,54],conceptu:[0,1,3,49,53,54,55],conclud:[0,47,49,53],conclus:[6,23,25,58],condit:[0,2,3,14,25,47,49,53,54],conduct:[2,3,4,14,25,49,53],confer:55,confid:[2,3],confirm:55,confound:[9,14,25],confront:0,connor:55,conov:55,consequ:[2,3,54,55],consequenti:[1,2,3,5,7,13,14,25,35,54,58],consid:[0,2,3,4,5,6,7,9,13,14,23,25,35,54],consider:[4,25,52,53],consist:[0,1,3,14,25,35,54],constrain:[10,14,23,25],constraint:[14,25,55],construct:[47,49,53,55],consult:0,consum:[5,25],contain:[2,3,54],content:59,context:[2,3,5,6,13,14,25,55],contigu:0,continent:0,continu:[2,3,9,14,25],continuum:0,contour:[4,25],contrast:[2,3,4,25],contribut:[0,6,25,55,57],control:[1,3,5,10,14,25,55,58],conveni:[4,25],converg:[23,25,47,49,53,55],convers:[6,25],cook:[0,55],cope:55,copula:55,core:[2,3,11,14,25],cornford:55,corones:55,correctli:[2,3],correl:[10,13,14,25],correspond:[9,14,25],cost:[5,8,9,14,25],costli:[4,25],could:[0,6,14,25,54],council:[0,55],coupl:[0,1,2,3,14,25,54,55],cover:[0,55],coverag:[10,14,25],cozannet:55,crc:55,creation:55,credenc:[5,25],credibl:[2,3,55],cremaschi:55,criteria:[11,14,25,55],criterion:[2,3,55],critic:[2,3,8,14,25,54,55],crombecq:55,cross:55,crucial:[6,25],crutzen:55,csrg:55,cube:55,cukier:[12,14,25,55],current:55,currin:55,cut:[6,25],cutoff:[6,25],cycl:[54,55],d:[9,10,14,23,25,55],da:55,dac:55,daedalu:55,dai:[2,3],dalal:[11,14,25,55],damag:55,data:[0,1,2,3,5,14,25,35,54,55],dauaek:55,de:55,dead:55,deal:[9,14,25],decad:[2,3],decarbon:55,deci:55,decid:[9,14,25],decis:[0,1,2,3,5,7,25,55],decompos:[1,3],deconto:55,deem:[2,3,47,49,53],deep:[0,13,14,25,49,53,54,55,58],deepli:[35,55],defens:[6,25],defici:[2,3],defin:[0,1,2,3,14,25,35,55],definit:[4,22,25,54],degre:[6,23,25],del:55,delib:[3,54],deliv:[23,25],delsa:55,delv:0,demand:[6,7,13,14,25,55],demarta:55,den:55,denot:[6,12,14,25,57],densiti:58,departur:[2,3],depend:[0,2,3,4,6,10,23,35,47,49,53,54],depth:54,der:55,deriv:[10,14,58],describ:[0,22,25,35,49,53],descript:55,design:[0,6,9,10,11,54,55,58],desir:[2,3],despit:[14,25],dessai:55,detail:[0,5,6,12,14,22,25,54],detc2003:55,detect:55,determin:[0,47,49,53],determinist:[0,2,3,55],dev:55,develop:[0,2,3,5,12,13,14,25,54,55],deviat:[11,14,25],dhaen:55,di:55,diagnos:[2,3,47,49,53],diagnost:[0,6,25,47,49,53,54,55,58],dickson:55,dietari:[7,25],differ:[0,2,3,4,5,8,12,14,25,35,47,49,53],differenti:55,difficult:0,digit:55,dimens:[2,3,11,14,25,55],dimension:[2,3,4,10,14,49,53,55,58],dioxid:55,direct:[1,3,5,12,14,25],directli:[47,49,53],disadvantag:[14,25,49,53],disast:55,discard:[47,49,53],disciplin:[0,54],disciplinari:[0,54],discontinu:[23,25],discov:[0,2,3,5,7,14,25],discoveri:[0,5,55,58],discredit:[5,25],discrep:[55,58],discret:[9,10,14,25],discuss:[0,4,13,14,22,25,47,49,52,53,54],dismiss:[2,3],dispers:[47,49,53],dispos:55,disrupt:55,distinguish:[0,2,3,10,14,25,54],distribut:[0,1,3,9,11,12,14,25,35,47,49,55,58],diverg:[2,3],divers:[0,10,14,25,54],divid:[10,14,25],dnf:55,doc:57,doctrin:55,doe:[2,3,4,25,55],doi:55,dokken:55,domain:[0,14,25,55],dominantli:[1,3],dorazio:55,doucet:55,draw:[47,49,53],drawn:[5,14,25],dream:55,drive:55,driven:[0,55],drought:[0,13,14,25,55],dubiou:[5,25],due:[4,25,47,49,53],dunson:55,duong:55,dure:[14,25],dyn:55,dynam:[0,2,3,47,49,53,54,55],e0156574:55,e0170052:55,e0190115:55,e0190641:55,e1278:55,e19100555:55,e2019wr024918:55,e2020wr028079:55,e2508:55,e:[0,1,2,3,4,5,6,7,8,9,11,13,14,25,52,53,54,55,57],e_:57,eaba2423:55,each:[0,2,3,4,6,8,9,10,12,14,22,23,25,49,52,53],eam:55,earli:[2,3],earth:55,eas:[8,14,25],eason:55,ebi:55,ebook:57,ecklei:55,ecolog:55,econ:55,econom:[0,55],ed:55,edinburgh:55,edit:55,edmond:55,edward:[6,25,55],effect:[0,2,3,4,5,6,8,9,10,14,47,49,53,55,57,58],effici:[14,25,55],effort:[5,25,52,53],eight:55,either:[1,3,5,14,25],ejor:55,eker:[2,3,55],el:55,elabor:[2,3,6,7,14,25],electr:55,electron:55,element:[0,14,25,55],elementari:58,elev:55,elicit:[14,25],elimin:[6,11,14,25],ellsberg:55,elsawah:[0,55],elsewher:[1,3],elucid:[6,25],embark:[14,25],embrac:[2,3,55],emerg:0,emiss:55,emphas:[0,2,3,54],empir:[1,2,3,4,25],empiric:[2,3],emploi:[0,2,3,11,14,25],emul:[49,52,53,55],enabl:[0,52,53],encompass:0,encourag:[13,14,25],encyclopedia:55,end:[14,25],energi:[0,2,3,7,13,14,25,55],eng:55,engag:0,engin:[0,55],england:55,english:55,enhanc:[1,2,3,55],enough:[7,25],ensembl:[0,1,2,3,55],ensur:[11,14,25,47,49,53],entir:[0,4,14,25],entropi:55,env:55,environ:55,environment:55,environmetr:55,envsoft:55,eo:55,eoq:55,epistem:[9,14,25,55],equal:[2,3,10,14,25],equat:55,equifin:[2,3,55],equival:[7,25,55],error:[0,2,3,7,11,14,25,47,49,53,55,58],escap:[47,49,53],esm:55,especi:[0,2,3,13,14,25,35],ess:[47,49,53],essai:55,establish:[0,2,3,5,7,25],estim:[9,11,13,14,23,25,47,49,53,55],estrada:55,et:[0,1,2,3,4,6,9,10,11,12,13,14,15,22,23,25,35,47,49,53,54,55],etc:[13,14,25],eur:55,euro:55,european:55,eutroph:55,evalu:[0,5,7,14,15,22,23,47,49,53,54,55,58],evan:55,even:[2,3,11,14,25],evenli:[11,14,25],evensen:55,event:[13,14,25,55,58],eventu:[47,49,53],ever:[14,25],everi:[10,14,25],evid:[6,25,55],evolut:[0,2,3],evolutionari:[0,2,3,35,54],evolv:[0,2,3],exacerb:55,exact:0,exactli:35,examin:[10,13,14,25],exampl:[0,2,3,6,7,9,10,12,13,14,23,25,47,49,54,58],execut:[6,25],exert:[14,25],exhibit:[0,2,3],exist:[0,12,13,14,25,47,49,52,53],expans:[2,3],expect:[57,61],expens:[23,25],experi:[1,2,3,4,6,9,54,55,58],experiment:[3,10,14,25,54],expert:[0,14,25,55],explain:[1,3,6,25],explanatori:[2,3],explicitli:[2,3],exploit:0,explor:[0,2,3,4,8,13,14,15,22,25,35,47,49,53,54,55],exploratori:[0,1,2,3,5,13,14,49,53,54,55,58],exponenti:[9,14,25],exposur:55,extend:[2,3,12,14,25,55],extens:[5,25],extent:[47,49,53],extract:[14,25],extrapol:[5,25],extrem:[0,9,13,14,25,55,58],f:55,face:[0,2,3,6,25,55],facilit:[47,49,53,55],fact:[2,3],factor:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,25,47,49,53,55,57],factori:[55,58],faith:[7,25],falkenmark:55,fals:55,falsif:55,famili:[2,3,54],far:[11,12,14,25,47,49,53],farmer:55,fashion:[2,3,54],fast:[12,14,25,55],favor:[2,3],feasibl:[0,4,14,25,54,55],featur:[22,25,54],fed:[14,25],feedback:0,fennel:55,ferdou:55,ferretti:55,feuillard:55,few:[0,14,25],fewer:[6,11,14,25],feyen:55,fidel:[0,2,3,5,7,58],field:[14,25,55],fifth:55,fig:[4,6,7,9,10,14,23,25],figur:[0,1,3,23,25,54,55],filatova:55,fill:[10,11,14,25],filter:[6,25,55],find:[2,3],finer:[2,3],finger:[2,3,55],finit:55,first:[1,2,3,12,14,23,25,57],fischbach:55,fisher:[14,25,55],fit:[2,3,55],fix:[6,7,8,12,14,25,49,53],fl:55,flegal:[47,49,53,55],flexibl:0,flood:55,flow:55,focu:[0,1,2,3,6,7,8,14,25,35,49,52,53],focus:[0,13,14,25],follow:[5,13,14,15,22,25,57],forecast:55,forest:55,form:[0,1,2,3,5,14,25,35,54,55],formal:[0,1,2,3,14,25,47,49,53,54,55],formul:[2,3,55],fortran:[52,53],fortuin:55,forward:3,fosa:55,fossil:55,found:[2,3,5,7,14,25],foundat:[1,3,5,25,55],four:[9,14,25],fourier:[12,14,25],fourth:55,frac:[15,22,25,57],fraction:[55,58],frame:[0,2,3,5,25,54],framework:[1,3,14,25,52,53,55],francisco:55,frankignoul:[35,55],frechett:55,free:[52,53],freedman:55,freeli:[6,25],freer:55,freita:55,frequenc:55,from:[0,1,2,3,6,10,11,14,25,47,49,53,54,55],frontlin:55,fu:55,fuel:55,full:[0,47,49,53,58],fuller:55,fulli:[0,2,3],fundament:[14,25],funtowicz:[2,3,55],further:[0,2,3,6,13,14,25],furthermor:[2,3,23,25],fusion:55,futur:[1,2,3,6,25,55],g:[0,1,2,3,4,6,7,9,11,13,14,15,22,25,52,53,54,55],ga:55,gabri:55,galeotti:55,game:55,ganeshan:55,gao:55,gap:[0,11,14,25,55],garner:55,gass:55,gatelli:55,gaug:[5,6,25],gaussian:[52,53,55],gave:[2,3],gelfand:55,gelman:[47,49,53,55],geman:55,gener:[0,1,3,4,9,10,11,12,47,49,53,55,58],genova:55,gent:55,geo:55,geogr:55,geophi:55,geosci:55,geostroph:55,geri:55,get:[1,3],geyer:[47,49,53,55],ghan:55,ghanem:55,giardino:55,gibb:[55,58],gidanski:55,gilligan:55,girma:55,given:[0,6,8,14,23,25],glob:55,global:[0,1,3,6,55,58],gloenvcha:55,glossari:[0,54,58],glue:[55,58],gmd:55,goal:[0,2,3,4,25],gober:55,godsil:55,gold:[47,49,53],gomez:55,good:[47,49,53,55],goodman:55,gordon:55,gorissen:55,grand:55,grang:55,graphic:[23,25],greatest:[6,25],grei:0,grid:55,group:[5,25,55],grow:[0,7,9,14,25,49,53],growth:[0,7,9,14,25,55],gsa:[1,3],guan:55,guarante:[2,3,10,11,14,25,47,49,53],guid:[0,2,3,54,55],guidanc:[0,54],guilla:55,gupta:[2,3,55],guston:55,h:55,ha:[0,1,3,4,5,6,7,9,14,25,35,47,49,53],haan:55,haasnoot:55,hack:55,hackbarth:55,hadjimichael:[3,55],haer:55,hagan:55,haim:[0,55],hall:55,hamilton:55,hamiltonian:[55,58],hamm:[4,25,55],han:55,hand:[5,14,23,25],handbook:55,haran:55,hardli:[14,25],hargreav:55,hartin:55,harwayn:55,hasselmann:[35,55],hast:[55,58],have:[0,2,3,4,5,6,8,9,12,13,14,23,25,47,49,53,54],hazard:55,health:55,heat:0,heather:55,heavili:[0,4,25],hector:55,heidelberg:55,height:55,helb:[0,55],held:[2,3],help:[2,3,5,6,25,54,55],helton:[14,25,55],hendri:55,heng:55,here:[0,1,2,3,23,25,49,52,53],herman:[13,14,25,55],herrera:[13,14,25,55],hess:55,hester:55,heterogen:[2,3],heurist:[2,3,5,25,47,49,53],hibbard:55,high:[2,3,7,13,14,25],higher:[0,2,3,9,14,25],highli:[0,1,2,3,5,8,10,14,25,55],highlight:0,hill:55,hindcast:55,hing:[0,7,25],hinkel:55,hinton:55,hirsch:55,histor:[0,2,3,4,25],histori:[2,3,13,14,25,54,55],hit:55,hoboken:55,hoc:[23,25],hoffman:55,hole:[14,25],holland:55,holst:55,holzmann:55,hop:55,horizont:0,hornberg:[6,25,55],horton:55,host:[5,25],hot:[13,14,25],hous:55,houtekam:55,how:[0,1,2,3,4,8,14,15,22,54,55,58],howev:[2,3,4,7,12,14,25,47,49,52,53],http:55,httpcran:55,hu:55,hub:55,human:[0,1,2,3,14,25,54,55],hunter:[9,14,25,55],hurk:55,hurrel:55,hurtt:55,hurvich:55,husbi:55,hwang:55,hybrid:[49,53,55],hydro:55,hydrol:55,hydrolog:[2,3,55],hyp:55,hypercub:[9,11,55,58],hypothes:[0,2,3,35,54],hypothesi:[0,1,3,55],i:[2,3,5,6,8,11,14,15,22,25,55,57],iaij:55,ic:55,idea:[47,49,53,55],ideal:[0,1,3,14,25,47,49,53],identif:55,identifi:[1,2,3,5,6,25],iee:55,ieee:55,ignor:[2,3],ii:55,illumin:[6,25],illustr:[0,1,3,4,25,55],im3:[0,2,3,54],imag:55,immedi:[13,14,25],immers:55,impact:[0,13,14,25,55],implement:[2,3,8,14,25,55,58],impli:0,implic:[0,55],imposs:[4,6,14,25],imprecis:55,improv:[0,1,3,6,7,11,14,25,47,49,53,55],includ:[0,2,3,4,12,14,25,52,53,54],inconsequenti:[7,25],inconsist:[52,53],incorpor:[13,14,25,52,53],increas:[0,6,7,8,9,14,23,25,55],increment:[2,3],inde:[9,14,25],independ:[1,3,8,14,23,47,49,53,55,58],index:[15,22,23,25,57,58],indic:[4,6,14,23,25,55,57],individu:[0,1,3,6,8,13,14,25,57],induc:55,infer:[0,1,2,3,5,7,14,25,35,47,49,52,53,54,55],influenc:[0,1,2,3,4,8,14,25,35,54,55],influenti:[5,6,25],info:55,inform:[0,1,2,3,4,5,6,8,9,10,13,14,23,25,35,55,62],infrastruct:55,infrastructur:55,inher:[2,3,52,53],initi:[1,3,47,49,53],inlet:55,innov:[0,1,3,54],input:[0,1,2,3,4,6,12,15,22,35,55,58],inquiri:[2,3,5,25],insid:55,insight:[0,2,3,54],inspir:[2,3],inst:55,instabl:55,instal:[],instanc:[5,6,7,25,52,53],instead:[2,3,4,5,9,14,25,35],institution:[2,3,55],insuffici:[2,3,13,14,25],integ:60,integr:[0,1,3,55,58],intel:55,intens:[23,25],interact:[0,1,2,3,4,6,8,9,14,23,25],interconnect:55,interdepend:[0,55],interdiscip:55,interest:[0,1,3,5,6,8,13,14,25,58],intergovernment:55,intermedi:55,intern:55,interpret:[2,3,47,49,53],intersci:55,intersect:[2,3],interv:[9,14,25],introduc:[14,25],introduct:[54,55,58],invari:[23,25,55],invers:55,invest:[5,25,55],investig:[5,25,55],involv:[47,49,53,54],iooss:[11,14,25,55],ip:55,issu:[2,3,55],iter:[3,8,14,25,47,49,53,54,55],its:[0,1,2,3,4,5,6,8,9,14,25,47,49,53,55],itself:[6,25],j:55,jacob:55,jakeman:55,jamuna:55,janssen:55,jasra:55,jayn:55,jhydrol:55,jin:[11,14,25,55],johansson:55,john:55,johnson:55,joint:0,jone:55,journal:55,judg:[2,3],judgement:[47,49,53],julia:[52,53],justif:[2,3,35],justifi:35,k:[9,14,25,55],kai:55,kainuma:55,kalman:55,kandlikar:55,karniadaki:55,kasmalkar:55,kass:55,keep:57,kei:[0,14,25,47,49,52,53,54],keith:55,keller:55,kennedi:55,kept:[8,14,25],kershaw:55,kettner:55,kiehl:55,kind:[14,25],kingdom:55,kirsch:[13,14,25,55],kissel:55,kleindorf:[2,3,55],klufa:55,km:55,know:[0,2,3,5,25],knowledg:[2,3,55],known:[0,2,3,6,25],kolstad:55,koomei:55,kopp:55,koutsoyianni:55,kovat:55,kraan:55,kram:55,krauss:55,krei:55,kucherenko:[11,14,25,55],kullback:55,kumar:[2,3,55],kundzewicz:55,kunreuth:55,kushner:55,kwakkel:[0,55],l:55,laboratori:[14,25],lack:[47,49,53],lade:55,lamarqu:55,lamontagn:[13,14,25,55],lamperti:55,land:0,landeg:55,landscap:0,lang:55,langan:55,languag:[7,25,52,53],lant:55,larg:[0,2,3,4,13,14,25,54,55],larger:[2,3,10,14,25,47,49,53],largest:[6,25],larson:55,later:[0,5,14,25],latest:57,latin:[9,55,58],latorr:55,lawrenc:55,le:55,lead:[2,3,6,7,25],learn:[2,3,54,55],least:[5,23,25],leav:[14,25,55],lee:55,left:[6,25],legitimaci:[2,3,55],leibler:55,lempert:[5,25,55],len:54,lend:[5,25],length:[14,25,55],less:[10,14,25,35],let:[15,22,25],lett:55,lettenmai:55,leve:55,level:[0,2,3,9,10,11,14,23,25,55],levi:55,lh:[11,58],li:55,life:55,like:[5,6,23,25],likelihood:[0,35,47,49,53,55],limit:[0,2,3,4,5,8,9,11,13,14,22,23,25,49,52,53,54,55],lin:55,lind:55,lindsai:55,linear:[0,4,14,23,25,55],linearli:[8,14,25],link:55,lipscomb:55,literatur:[0,2,3,4,6,9,12,13,14,25],liu:55,loa:55,local:[0,6,8,14,15,22,55,58],locat:[8,14,25],log:55,logic:55,lond:55,longer:[5,25],look:[3,55],lorenz:55,loss:[2,3,55],lost:[2,3],lot:0,louck:[13,14,25,55],louisiana:55,low:[2,3,55,58],lower:[9,11,14,25],lukac:55,lump:55,lumpi:[11,14,25],m:[4,25,55],maccracken:55,mach:55,machin:55,made:[2,3,6,14,25],magnitud:[23,25],mahecha:55,mahowald:55,mai:[0,2,3,35,47,49,52,53],main:[8,9,14,25,54],major:[5,25],make:[0,2,3,6,7,9,14,23,25,54,55,61],makowski:55,manag:55,manceau:55,mandat:[23,25],mani:[0,2,3,5,14,25,47,49,52,53,55],maniyar:55,manner:[5,6,8,10,14,25],mao:55,map:[0,4,6,7,14,25,54],maraun:55,marchau:[35,55],margin:[47,49,53],markov:[55,58],markovian:[47,49,53],marrel:55,marsh:55,marshal:55,martin:55,martiu:55,maruri:55,massmann:[14,25,55],mastrandrea:55,masui:55,mater:55,materi:55,math:55,mathemat:[11,14,15,22,25,55],matlab:[52,53],matott:55,matthew:55,max:55,maxim:[10,14,25],maximum:[47,49,53],mcinernei:55,mckai:[10,14,25,55],mcmc:[47,49,52,53,55],mcmcpack:55,mcneil:55,mea:55,mean:[1,3,11,14,25],meant:0,measur:[2,3,5,6,9,11,14,15,22,23,25,55],mech:55,mechan:[2,3,52,53,55],mechanist:55,mechanobiolog:55,medda:[13,14,25,55],mediterranean:55,meinshausen:55,membership:0,meng:55,mentaschi:55,mention:[7,12,14,25],meridion:55,met:[2,3],metamodel:55,meteorol:55,method:[0,2,3,4,6,8,10,11,12,13,14,52,54,55,58],methodnam:61,methodol:55,methodolog:[0,2,3,14,25,55],metric:[1,2,3,5,6,7,8,14,25],metropoli:[14,25,55,58],miao:55,mice:55,microsimul:55,might:[5,13,14,25],mijer:55,milli:[13,14,25,55],mimick:[2,3],mind:[7,13,14,25],mine:[14,25],minim:[11,14,25],minimum:[0,6,14,23,25],mirror:[2,3,54],miscalcul:[4,25],misconverg:[47,49,53],miss:[8,14,23,25],mitchel:[11,14,25,55],mitig:[14,25],mix:[47,49,53,55],moallemi:[0,35,55],modal:55,mode:[0,2,3,6,25,55],model:[0,4,5,8,9,11,13,14,15,22,47,49,54,55,57,58,59],modern:[54,55],modif:[10,14,25],modifi:[10,14,25,55],modul:[55,58,59],moment:58,mon:55,monetari:[5,25],mont:[1,3,6,11,14,25,55,58],montgomeri:[9,14,25,55],moral:55,more:[0,1,2,3,4,6,7,8,9,12,14,23,25,35,47,49,52,53],moreov:[49,53,54],morgan:55,morri:[11,12,14,25,55],most:[1,2,3,4,5,6,7,10,11,14,22,23,25,35,49,53,54],mostafavi:55,motiv:[5,14,23,25],move:[12,14,25],msd:0,msd_uncertainty_ebook:57,much:[10,11,14,25],multi:[0,13,14,25,54,55],multiag:55,multidecad:55,multidimension:[11,14,25],multimodel:55,multipl:[2,3,8,14,23,25,47,49,53,55],multiscal:[0,2,3],multisector:[0,2,3],multitud:[0,2,3],must:[2,3,47,49,53],my:57,n:[4,9,10,12,14,23,25,55],nakicenov:55,narr:0,nasa:55,nat:55,natarajan:55,nation:[0,55],nativ:[52,53],natl:55,natur:[0,1,2,3,14,25,54,55],nature12047:55,nature17145:55,naylor:[2,3,55],nb:[6,25],ncrg:55,neal:55,necessari:[0,1,2,3,6,9,14,25],necessarili:[2,3],need:[0,1,2,3,7,14,25],neg:[14,25,55],negat:[8,14,25],neglect:[2,3,55],neglig:[6,9,14,25],negoti:[2,3],neill:55,nelder:55,network:55,neural:55,never:[2,3,14,25],next:[4,25],nexu:0,nhess:55,nielsen:55,nikoletopoulo:55,nj:55,nk:[9,14,25],noah:55,nois:55,nolan:55,nomin:[5,6,14,15,22,25],non:[0,6,13,14,25,47,49,53,54,55],noncommensur:55,nonlinear:[4,8,14,23,25,55],nonperiod:55,nor:[0,14,25],norfolk:55,normal:55,norton:[14,25,55],nossent:[23,25],notat:[15,22,25],note:[0,49,53,54],novel:[0,55],now:[7,25,55],nowak:[0,55],nuclear:55,number:[0,2,3,6,9,10,11,12,14,23,25,47,49,53,54],numer:[0,2,3,4,5,6,14,25,55],ny:55,o:55,oat:[15,22,58],oberstein:55,object:[0,6,25,54,55],observ:[0,2,3,6,13,14,23,25,54,55],obtain:[9,14,25,47,49,53,57],occur:[2,3],ocean:55,oddo:55,often:[2,3,4,13,14,25,55],oliv:55,onc:[12,14,25,47,49,53],one:[1,2,3,5,6,8,9,10,11,12,14,15,22,25,35,47,49,53,55],onli:[2,3,4,8,9,14,15,22,23,25,47,49,53,55],oort:55,open:[0,54],oper:[52,53,55],opinion:[0,14,25,55],oppenheim:55,opportun:0,oppos:0,opr:55,optim:[11,14,25,55],order:[6,9,14,23,25,57],oresk:[2,3,55],org:55,organ:[5,25],orgpackag:55,orient:[2,3],origin:[1,3],orlean:55,orthogon:[10,14,25,55],ortiz:55,ortolano:55,other:[2,3,4,5,6,8,9,23,47,49,53,58],otherwis:[4,25],ouarda:55,our:[0,1,2,3,5,25,54],out:[2,3,4,25,55],outcom:[0,1,3,5,6,25,35,55,58],output:[0,1,2,3,4,5,6,7,8,11,14,15,22,25,55,57],ouyang:55,over:[14,25],overal:54,overcom:[8,14,25],overlap:[10,14,25],overturn:55,overview:[0,23,52,53,54,55,58],overyear:55,oxford:55,p:[9,14,25,55],packag:[52,53,55,59],page:58,pair:[5,25],panel:[4,7,25,55],paradigm:55,paradox:55,paramet:[0,1,2,3,4,5,8,9,10,14,25,47,49,53,55,60],parameter:[1,2,3,5,25,35],parametr:[1,3,4,25,49,53],pari:55,park:[11,14,25,55],part:55,partial:[4,15,22,25],participatori:55,particl:[55,58],particular:[1,3,14,15,22,25,49,53],particularli:0,past:[13,14,25,55],path:[0,2,3,35,54,55],pathwai:55,pattern:55,pawson:55,pearson:55,peel:55,peng:55,pepper:55,per:[2,3,9,14,25],percent:57,perfectli:[2,3],perform:[0,1,2,3,4,7,9,14,15,22,23,55,58],perfunctori:55,perish:55,persist:[13,14,25],perspect:[0,7,25,49,53,54,55,58],pertin:[5,25,52,53],perturb:[4,12,14,15,22,25],petschek:55,philo:55,philosoph:[54,55],philosophi:55,phim:55,phy:55,physic:[2,3,9,14,25,55],pianosi:[6,12,14,15,22,25,55],pie:[1,3],pillo:55,pinpoint:[6,25],pip:62,place:[0,2,3,7,25],placehold:60,plan:55,platform:[14,25,52,53,54,55],plausibl:[14,25],pleas:[52,53],plischk:[4,25,55],plo:55,plot:[0,4,25,47,49,53],plot_experimental_design:57,plural:[2,3],pna:55,point:[2,3,4,11,12,14,25,55],polici:55,policymak:55,pollard:55,polynomi:55,pone:55,pool:[47,49,53],poon:55,popper:55,popul:[0,7,14,25],popular:[8,14,25,52,53],pose:0,posit:55,positivist:[2,3],possibl:[2,3,6,9,14,25,35,55],posterior:[47,49,55,58],posteriori:[47,49,53],potenti:[0,2,3,9,11,13,14,23,25,47,49,53,55],power:[0,2,3],pp:55,pract:55,practic:[2,3,4,25,47,49,53,55],practition:[2,3],pre:[6,25,58],precalibr:55,precipit:55,precis:[2,3,55],predict:[2,3,35,55,58],prefac:[],prefer:[2,3,4,25,49,52,53],preliminari:55,premis:[2,3],prescrib:3,presenc:0,present:[0,1,2,3,4,5,6,9,14,23,25,52,53,54,55],press:55,preval:[2,3],previou:35,previous:[7,25],price:[13,14,25],prieto:55,primari:0,primarili:[2,3,4,8,14,25],primer:55,prior:[0,35,49,55,58],priori:[14,25],priorit:[6,7,25,49,53],probab:55,probabilist:[0,35,55],probabl:[0,35,55,58],problem:[6,9,14,25,55],proc:55,procedur:[2,3,14,25],process:[0,2,3,5,6,8,9,13,14,25,52,53,55],processor:55,produc:[2,3,4,5,7,9,14,23,25],prog:55,program:[52,53,57],progress:[2,3,54],proj:55,project:[0,2,3,35,54,55],promot:0,propag:[1,3],properli:[4,25],properti:[0,5,11,14,25,47,49,53,55],propheci:55,propos:[2,3,4,6,9,14,25],protect:55,proven:[4,23,25],provid:[0,1,2,3,4,13,14,22,25,54,55],proxi:[47,49,53],prudent:[23,25],pruett:55,pseudo:[11,14,25],psychol:55,publish:55,pure:[2,3],purpos:[2,3,5,13,14,23,25,54],put:0,python:[52,53],q:55,qualit:[2,3],qualiti:[2,3],quantif:[0,47,49,52,55,58],quantifi:[1,3,35,49,53],quantit:[0,2,3,11,14,25,55],quantiti:[35,47,49,53],quasi:[11,14,25,55],quest:55,question:[0,1,2,3,5,14,23,25,47,49,53,54],quinn:55,quixot:55,r:[12,14,25,52,53,55],racin:55,radar:55,rafteri:55,ragulina:55,rahman:55,rainfal:55,rais:[1,3],rakovec:[4,25,55],ramallo:55,ramo:55,random:[11,14,25,55],randomli:[10,12,14,25,47,49,53],rang:[0,2,3,4,6,9,14,25],rank:[6,25],rapid:0,rapidli:[0,49,53],rare:[13,14,25],rasch:55,rasmussen:55,rate:55,rather:[2,3,7,25,49,53],ratio:[47,49,53],raton:55,ratto:55,raymond:55,re:55,reaction:55,reader:[2,3,6,12,13,14,23,25,54],readili:[14,25],real:[0,5,7,14,25],realiti:[2,3,55],realiz:55,realli:55,reason:[2,3,5,8,9,13,14,25],recal:[23,25],recent:[9,14,25,55],recherch:55,recogn:[2,3],recommend:54,reconcil:55,reconfigur:55,record:[13,14,25],reduc:[1,2,3,5,6,9,14,25,55],reduct:[6,25,47,49,53,55],reed:55,refer:[0,4,6,9,12,14,25,35,47,49,53,58],refin:[5,25],reflect:[0,2,3,7,14,23,25],regard:[23,25],regardless:[5,25,47,49,53],region:[6,7,58],regress:[23,55,58],reitan:55,rel:[0,2,3,5,25,47,49,53,54,55],relat:[0,2,3,14,25,54,55],relationship:[0,1,3,4,14,23,25],relativist:[2,3,7,25],relax:55,relev:[0,2,3,5,7,25],reli:[47,49,53],reliab:55,reliabl:55,remain:[6,25],remark:0,rep:55,repeat:[8,14,25],repetit:[12,14,25],replac:[47,49,53],report:[0,14,25,55],repres:[0,2,3,5,6,13,14,25,47,49,53,55],represent:[0,1,2,3,5,7,25,35],requir:[0,1,2,3,23,25,35,47,49,52,53],research:[0,5,14,23,25,55],reservoir:55,resili:[0,1,2,3,54,55],resolut:[0,2,3,14,25,55],resour:55,resourc:[5,25,55],respect:57,respond:[0,55],respons:[0,1,3,4,7,14,25,55],ress:55,rest:[14,25],restor:55,result:[0,1,2,3,4,5,6,9,10,11,14,25,47,49,53,55],retreat:55,rev:55,reveal:[4,5,25],revel:55,review:[0,13,14,25,49,53,54,55],riahi:55,rich:[1,3,13,14,25],ridder:55,right:[2,3,7,25],rigor:[0,54],risa:55,risbei:55,rise:[2,3,55],risk:[0,6,25,55,58],river:55,riverin:55,road:55,robert:55,robinson:55,robust:55,rocki:55,rohmer:55,root:55,rose:55,rosenbluth:55,roson:55,rotman:55,rougier:55,rovenskaya:55,roventini:55,rozenberg:55,rspa:55,rst:[],rubin:[47,49,53,55],ruckert:55,ruddel:55,run:[1,3,6,7,9,14,25,47,49,52,53],runoff:55,runtest:61,ryan:55,s00382:55,s00477:55,s00521:55,s0304:55,s0378:55,s10463:55,s10584:55,s1064827501387826:55,s11222:55,s40070:55,s41467:55,s41598:55,s43017:55,s:[0,1,2,3,4,5,6,7,8,14,15,22,25,52,53,55],s_i:[15,22,25,57],sa:[5,25],sack:55,saf:55,safe:55,safeti:55,saisana:55,salienc:[2,3,55],sallaberri:55,salmond:55,saltelli:[0,1,2,3,4,6,8,9,10,12,14,23,25,55,57],same:[2,3,5,9,10,11,14,23,25,52,53],samo:55,sampl:[1,3,8,11,15,22,23,55,58],sampler:55,san:55,sanderson:55,sarewitz:55,savag:55,scale:[0,2,3,14,15,22,23,25,47,49,53,54,55],scenario:[0,2,3,5,13,14,35,55,58],schaibli:55,scheme:[8,9,10,14,25],schepdael:[14,25,55],schlather:55,schlesing:55,schwarz:55,sci:55,sciadv:55,scienc:[0,2,3,47,49,53,55],scientif:[0,2,3,14,25,54,55],scope:[0,2,3,13,14,25,49,53],scotland:55,scott:55,screen:[6,25],script:[0,54],sdr:55,se:[2,3],sea:55,search:58,second:[1,2,3,23,25],section:[0,2,3,6,7,13,22,23,35,49,53,54],sector:[54,55],secur:55,see:[52,53,54],seed:[14,25],seek:[0,2,3,54],seen:[2,3],select:[2,3,8,12,14,23,25,49,54,55,58],sennevil:55,sensit:[0,1,2,3,8,10,12,13,14,15,49,53,54,55,57,58],sensor:[9,14,25],seq:55,sequenc:[8,55,58],sequenti:[52,53,55],ser:55,serafin:55,seri:[55,58],serv:55,servic:55,sesmo:55,set:[0,1,2,3,4,5,6,14,23,25,35,47,49,53,55],sever:[2,3,4,12,13,14,23,25,47,49,53],shaffer:55,shafii:55,shahidehpour:55,shalizi:55,shape:[0,5,6,14,25,54,55],sharp:55,sheet:55,shelf:55,sherwen:55,shewbridg:55,shibata:55,shift:[2,3],shock:0,shortag:55,should:[0,2,3,4,6,14,23,25,47,49,53,54],show:[6,25],showcas:[0,54],shown:[0,1,3],shrader:55,shuler:55,si_67_8:55,siam:55,siamasa:55,siblei:55,side:[2,3],sign:[47,49,53],signal:55,signatur:[2,3],signific:[0,2,3,5,6,25,55],significantli:[5,9,14,25],silva:55,sim_i:57,similar:[2,3,8,14,25],simpl:55,simplic:[15,22,25],simplif:[1,3,5,25],simplifi:[2,3,5,23,25],simpson:55,simul:[1,2,3,9,11,14,25,55],sin:55,singl:[2,3,15,22,25,55],singular:[0,55],site:55,situat:[0,2,3,6,25],size:[11,23,47,49,53],skew:[47,49,53],skill:[47,49,53],slope:55,slowli:55,sluij:55,small:[4,5,11,14,25,47,49,53,55],smallest:[6,25],smart:55,sminchisescu:55,smith:55,so:[2,3,5,6,12,14,25,47,49,53,55],sobol:[11,12,14,23,25,55,57],soc:55,societi:55,socio:55,sociol:55,softw:55,softwar:58,solut:57,solv:55,some:[0,5,8,14,22,25,47,49,53],someth:[11,14,25],son:55,song:55,sophist:[6,8,14,25],sorooshian:55,sourc:[0,1,3,4,25,57],space:[0,1,2,3,4,6,8,9,10,11,12,14,25,35,47,49,53,55],span:[0,2,3],spano:55,spatial:[2,3,13,14,25,55],spear:[6,25,55],specif:[0,1,2,3,4,5,6,12,14,25,35,52,53,54,55],specifi:[0,55],spectral:55,spent:[47,49,53],spiegelhalt:55,springer:55,springett:55,srikrishnan:55,sriver:55,ss:55,stabl:[5,23,25,55],stai:[2,3],stakehold:[2,3,7,25],standard:[47,49,53],stano:55,stark:55,start:[12,14,25],stat:55,statap:55,state:[0,2,3,7,14,25,55],statement:[23,25],stationar:[13,14,25,55],stationari:[47,49,53,55],statist:[0,1,2,3,11,13,14,25,55,58],statistician:[47,49,53],statistiqu:55,statm:55,steding:[13,14,25,55],stefan:55,steinschneid:[13,14,25,55],stem:[2,3],step:[1,3],stephenson:55,stern:55,still:[2,3,55],stoch:55,stochast:[13,14,25,55],stone:55,storli:55,storm:55,stouffer:55,strategi:[9,12,14,25,55],stratifi:[14,25],streamflow:[13,14,25,55],strength:[47,49,53],stress:[47,49,53],stressor:[0,2,3,13,14,25,35,54],strict:[7,25],strong:[0,13,14,25,55],strongli:54,structur:[0,1,2,3,5,9,14,25,35,49,53,55],sts257:55,studi:[0,4,5,13,14,25,47,49,53,55],style:[11,14,25],styliz:[6,25],su5030955:55,subinterv:[10,14,25],subject:[2,3,55],submit:55,submodul:59,subpackag:59,subroutin:[52,53],subsequ:[6,7,10,12,14,23,25],subset:[52,53,55],subspac:[11,14,25],substanti:[14,25,35],suckal:55,sudjianto:55,suffici:[0,2,3,5,7,13,14,25,47,49,53,55],suggest:[6,25,54],suit:[13,14,25,52,53],suitabl:[47,49,53],sum:60,sum_int:[60,61],summar:[14,25],summari:[0,55],suppli:55,support:[0,2,3,54,55],sure:[57,61],surfac:55,surg:55,surowiec:[9,14,25,55],surrog:[55,58],surround:[5,25,55],survei:[0,55],sustain:55,svahn:55,syke:55,synthes:0,synthesi:[0,23,25],synthet:[55,58],syst:55,system:[0,1,2,3,5,7,8,14,23,25,54,55,58],systemat:[47,49,53,55],t:55,tabl:[14,25],tac:55,tail:[47,49,55,58],take:[0,11,14,25],tang:[4,10,14,25,55],tarantola:[6,25,55],target:[47,49,53,55,62],task:0,team:[0,9,14,25,54],tech:55,techfor:55,technic:[0,2,3,55],techniqu:[3,4,5,6,8,10,11,12,14,23,25,54,55],technol:55,technolog:0,technometr:55,tell:55,teller:55,tellu:55,tellusa:55,temperatur:[7,9,14,25,55],tempor:[2,3,13,14,25],tend:[7,25],tenuou:[2,3],term:[0,2,3,4,7,25],terminolog:54,test:[0,1,2,3,5,9,12,14,47,49,53,54,55,58,59,60],test_model:[59,60],test_sum_int:61,testcas:61,testmodel:61,teul:55,text:[0,2,3,4,5,13,14,25,54],th:[15,22,25],than:[0,2,3,7,9,10,14,25,47,49,53],theater:55,thei:[0,2,3,4,5,9,13,14,23,25,47,49,53],them:[5,6,7,14,25],themselv:[1,3],theoret:[0,55],theori:[0,2,3,55],therefor:[2,3,4,5,6,14,25,47,49,53],thermoclin:55,thi:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,22,23,35,47,49,53,54,57],thieri:55,think:[49,53,55],third:55,thompson:55,thomson:55,thorough:54,those:[0,1,2,3,4,25,54],though:[2,3,11,14,25,47,49,53],three:[6,9,14,25,55],threshold:[0,6,25,47,49,53,55],through:[1,2,3,10,11,14,25,54],throughout:0,thu:[47,49,53],tian:[35,55],time:[12,23,47,49,53,55,58],tissu:55,tit:55,togeth:[0,1,3],tol:55,toler:[6,25],tolson:55,tommasi:55,tool:[0,1,2,3,6,13,14,25,54,58],toolkit:58,top:[7,25],topic:[0,13,14,25],toronto:55,total:[6,9,14,23,25],toulmin:[2,3,55],toward:[2,3,55],tpami:55,trace:[6,25],track:57,tradit:[0,2,3],tradition:[9,14,25],traffic:55,trait:58,trajectori:[12,14,25],tran:55,transdisciplinari:[0,54,55],transfer:55,transit:[0,2,3],translat:0,transp:55,transport:55,travi:55,trayanov:55,treat:[9,13,14,25,49,53],treatment:[0,2,3,4,14,25],treich:55,tremblai:55,trenner:55,trivial:[0,54],tropospher:55,trust:55,truth:[2,3],trygg:55,tsai:55,tsunami:55,tune:[47,49,53],turn:[14,23,25,55],two:[1,2,3,4,5,9,14,25,47,49,52,53,55,60],type:[0,2,3,6,55,58],typic:[0,9,14,23,25,47,49,52,53],typolog:55,u:55,uc:[0,1,3,35,54],uijlenhoet:55,uk:55,ulam:[14,25,55],ullah:55,ulph:55,ultim:[8,14,25,54],unaccept:[7,25],unaid:55,uncertain:[0,1,2,3,4,5,6,8,9,10,12,14,23,25,35,49,53,55],uncertainti:[0,1,2,3,4,5,6,9,13,14,25,47,49,52,54,55],under:[0,2,3,7,14,25,54,55,58],underestim:[4,25,55],underli:[0,2,3],underlin:[15,22,25],undermin:[2,3],understand:[0,2,3,5,23,25,54,55,58],understood:[0,55],undesir:[2,3,5,6,7,25],uniform:[9,11,14,25,55],uniformli:[11,12,14,25,55],unimport:[5,25],union:55,uniqu:[2,3],unit:[0,55],unittest:61,univers:[47,49,53,55],unnecessari:[6,25],unpreced:[2,3],up:[2,3,14,25,55],upon:[52,53],upper:55,uq:[0,14,25,35,52,53,54],urban:[0,55],url:55,us:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,25,35,47,49,52,53,54,55],usa:55,user:[52,53],ussr:55,usual:[0,6,12,14,25],util:[0,2,3,11,14,25],v10:55,v29i4:55,v:[6,25,55,57],v_:57,va:55,vafeidi:55,valeur:55,valid:[2,3,4,25,55],valu:[0,2,3,4,5,6,7,9,14,15,22,23,25,47,49,53,55,57,61],valuabl:[13,14,25],van:[14,25,55],vari:[0,4,6,14,23,25,47,49,53,55,57],variabl:[2,3,4,6,9,10,11,14,25,55],varianc:[6,11,14,47,49,53,55,57,58],variat:[12,14,25],varieti:[2,3,6,12,13,14,25],variou:[0,1,2,3,11,14,25,47,49,53,54,55],vast:[2,3],vavru:55,vector:[15,22,25,55],vega:55,vehtari:55,veri:[2,3,10,13,14,25,54],verif:[2,3,55],version:55,versu:[6,13,14,58],vertenstein:55,vertic:0,via:55,view:[0,2,3,7,25,54],viewpoint:[2,3],vignotto:55,vihola:55,visual:55,vivek:55,vogel:[13,14,25,55],von:55,voukouvala:55,vousdouka:55,vroman:55,vrugt:55,vs:[2,3],vulner:[0,1,2,3,54,55],vuuren:55,w:[0,55],wa:[14,25],wagen:55,wai:[2,3,5,6,14,25,55],waic:55,walker:[0,35,55],wallach:55,warm:55,warrant:[5,25],wasec:55,washington:55,wast:55,wat2:55,watanab:55,water:[0,55,57],watersh:55,wave:0,wcc:55,we:[0,1,2,3,4,5,22,25,49,52,53,54,55,62],weak:[47,49,53],wear:55,weather:[13,14,25,55],weaver:[2,3,54,55],webster:55,wedderburn:55,weert:55,welch:55,well:[0,2,3,5,6,13,14,22,25,35,47,49,53,54,55],were:[2,3,14,25],werkhoven:55,wesselink:55,west:55,westhoff:55,westra:55,what:[1,3,23,25,58],when:[0,1,2,3,5,6,7,9,10,11,13,14,23,25,35,47,49,53,54,55],whenev:[4,25],where:[0,1,2,3,4,9,12,14,15,22,23,25,35,54],wherea:[2,3],whether:[2,3],which:[0,1,2,3,4,5,6,7,8,11,14,23,25,35,47,49,53],white:[2,3,55],whither:55,whitman:55,why:[1,3,55,58],wi:55,wicked:55,wide:[0,2,3,4,9,11,14,22,25,55],wiener:55,wikman:55,wilbi:[13,14,25,55],wildl:55,wilei:55,wilk:[13,14,25,55],willem:55,wilson:55,wire:55,wirtz:[0,55],within:[0,2,3,4,6,9,11,14,25,47,49,52,53,55],without:[4,10,14,25,35,52,53],wolff:55,wong:55,wood:55,word:[4,9,14,23,25],work:[54,55],workflow:[0,54,55],world:[0,5,14,25],would:[0,6,7,8,14,23,25],wr011i005p00681:55,wr024i009p01483:55,wr99107:55,wr:55,wrapper:[52,53],wrcr:55,wrong:[2,3,55],wu:55,wutich:55,wynn:55,x1:[4,6,9,14,25],x2:[4,6,9,14,25],x3:[6,9,14,25],x:[4,6,14,15,22,25,55,57],x_i:57,xi:[6,25],xian:55,xiu:55,xn:[4,25],xx:[47,49,53],y1:[4,25],y2:[4,25],y:[4,6,14,25,55,57],yapo:55,ye:[14,25,55],year:[13,14,25],yet:[47,49,53],yield:[1,3,5,25,35],ylvisak:55,ym:[4,25],york:55,you:[1,3],your:[5,25,57],z:55,zare:55,zarekarizi:55,zaremba:[11,14,25,55],zeff:55,zellner:[35,55],zhang:55,zhao:55,zscheischler:55},titles:["Introduction","Overview of model diagnostics","Perspectives on diagnostic model evaluation","Diagnostic Modeling Overview and Perspectives","Global Versus Local Sensitivity","Why Perform Sensitivity Analysis","Sensitivity Analysis Applications for Model Evaluation and Fidelity Testing","Sensitivity Analysis Applications for Exploratory Modeling and Scenario Discovery","One-At-a-Time (OAT)","Full and Fractional Factorial Sampling","Latin Hypercube (LH) Sampling","Low-Discrepancy Sequences","Other types of sampling","Synthetic generation of input time series","Design of Experiments","Derivative-based Methods","Elementary Effect Methods","Regression-based Methods","Regional Sensitivity Analysis","Variance-based Methods","Analysis of Variance (ANOVA)","Moment-Independent (Density-Based) Methods","Sensitivity Analysis Methods","How To Choose A Sensitivity Analysis Method: Model Traits And Dimensionality","Software Toolkits","Sensitivity Analysis: The Basics","Understanding Errors: What Is Controlling Model Performance?","Consequential Dynamics: What is Controlling Model Behaviors of Interest?","Consequential Scenarios: What is Controlling Consequential Outcomes?","Sensitivity Analysis: Diagnostic & Exploratory Modeling","Why is Uncertainty Quantification Important for Understanding MultiSector System Dynamics?","Uncertainty Quantification for Exploratory Modeling","Bayesian Uncertainty Quantification","Uncertainty Quantification Under (Deep) Uncertainty","Integrating Model Diagnostics and Uncertainty Quantification","Uncertainty Quantification: The Basics","Understanding Risk: How Probable Are Extreme Events?","Understanding Tails: Statistical Modeling of Extreme Events","How to Choose an Appropriate Method?","How to Select a Prior Distribution?","Posterior Predictive Checking","Model Selection and Comparison","Scenario Discovery","Pre-calibration/GLUE","Metropolis-Hastings","Gibbs Sampling","Hamiltonian Monte Carlo","Markov Chain Monte Carlo","Particle-based Methods","What are Common Methods?","Markov Chain Monte Carlo with the True Model","Markov Chain Monte Carlo with Surrogate Models","What are Example Software Implementations?","Uncertainty Quantification: A Tool For Capturing Risks & Extremes","Conclusion","References","Glossary","Examples","Addressing Uncertainty in MultiSector Dynamics Research","nanites","nanites package","nanites.tests package","Preface"],titleterms:{"import":[30,35],"true":[50,52,53],A:[23,25,53],And:[23,25],At:[8,14,25],For:53,Is:[26,29],One:[8,14,25],The:[25,35],To:[23,25],acknowledg:62,address:58,an:[38,53],analysi:[5,6,7,18,20,22,23,25,29],anova:[20,22,25],applic:[6,7,25],appropri:[38,53],ar:[36,49,52,53],audienc:62,base:[15,17,19,21,22,25,48,49,53],basic:[25,35],bayesian:[32,35],behavior:[27,29],book:62,build:57,calibr:[43,49,53],caption:57,captur:53,carlo:[46,47,49,50,51,52,53],chain:[47,49,50,51,52,53],check:[40,53],choos:[23,25,38,53],codeblock:57,common:[49,53],comparison:[41,53],conclus:54,consequenti:[27,28,29],content:[60,61],control:[26,27,28,29],deep:[33,35],densiti:[21,22,25],depend:[14,25],deriv:[15,22,25],design:[14,25],diagnost:[1,2,3,29,34,35],dimension:[23,25],discoveri:[7,25,42,49,53],discrep:[11,14,25],distribut:[39,53],dynam:[27,29,30,35,58],ebook:[],effect:[16,22,25],elementari:[16,22,25],error:[26,29],evalu:[2,3,6,25],event:[36,37,53],exampl:[52,53,57],experi:[14,25],exploratori:[7,25,29,31,35],extrem:[36,37,53],factori:[9,14,25],fidel:[6,25],figur:57,fraction:[9,14,25],full:[9,14,25],gener:[13,14,25],gibb:[45,47,49,53],global:[4,25],glossari:56,glue:[43,49,53],hamiltonian:[46,47,49,53],hast:[44,47,49,53],how:[23,25,36,38,39,53,62],hypercub:[10,14,25],implement:[52,53],independ:[21,22,25],indic:58,input:[13,14,25],insert:57,instal:62,integr:[34,35],interest:[27,29],introduct:0,latin:[10,14,25],lh:[10,14,25],local:[4,25],low:[11,14,25],markov:[47,49,50,51,52,53],mathjax:57,method:[15,16,17,19,21,22,23,25,38,48,49,53],metropoli:[44,47,49,53],model:[1,2,3,6,7,23,25,26,27,29,31,34,35,37,41,50,51,52,53,60],modul:[60,61],moment:[21,22,25],mont:[46,47,49,50,51,52,53],msd_uncertainty_ebook:62,multisector:[30,35,58],nanit:[59,60,61],note:57,oat:[8,14,25],other:[12,14,25],out:57,outcom:[28,29],overview:[1,3,14,25],packag:[60,61,62],particl:[48,49,53],perform:[5,25,26,29],perspect:[2,3],posterior:[40,53],pre:[43,49,53],predict:[40,53],prefac:62,prior:[39,53],probabl:[36,53],python:62,quantif:[30,31,32,33,34,35,53],refer:55,region:[18,22,25],regress:[17,22,25],research:58,risk:[36,53],sampl:[9,10,12,14,25,45,47,49,53],scenario:[7,25,28,29,42,49,53],section:[14,25],select:[39,41,53],sensit:[4,5,6,7,18,22,23,25,29],sequenc:[11,14,25],seri:[13,14,25],size:[14,25],softwar:[24,25,52,53],statist:[37,53],submodul:[60,61],subpackag:60,summari:[14,25],surrog:[51,52,53],synthet:[13,14,25],system:[30,35],tabl:58,tail:[37,53],test:[6,25,57,61],test_model:61,thi:[14,25,62],time:[8,13,14,25],tool:53,toolkit:[24,25],trait:[23,25],type:[12,14,25],uncertainti:[30,31,32,33,34,35,53,58],under:[33,35],understand:[26,29,30,35,36,37,53],us:[57,62],varianc:[19,20,22,25],versu:[4,25],what:[26,27,28,29,49,52,53],why:[5,25,30,35]}}) \ No newline at end of file diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.aux b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.aux index 994315cd..fdd5df75 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.aux +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.aux @@ -93,8 +93,88 @@ \newlabel{3_sensitivity_analysis_the_basics:how-to-choose-a-sensitivity-analysis-method-model-traits-and-dimensionality}{{3.7}{17}{How To Choose A Sensitivity Analysis Method: Model Traits And Dimensionality}{section.3.7}{}} \@writefile{toc}{\contentsline {section}{\numberline {3.8}Software Toolkits}{18}{section.3.8}\protected@file@percent } \newlabel{3_sensitivity_analysis_the_basics:software-toolkits}{{3.8}{18}{Software Toolkits}{section.3.8}{}} -\@writefile{toc}{\contentsline {chapter}{\numberline {4}Indices and tables}{19}{chapter.4}\protected@file@percent } +\@writefile{toc}{\contentsline {chapter}{\numberline {4}Sensitivity Analysis: Diagnostic \& Exploratory Modeling}{19}{chapter.4}\protected@file@percent } \@writefile{lof}{\addvspace {10\p@ }} \@writefile{lot}{\addvspace {10\p@ }} -\newlabel{index:indices-and-tables}{{4}{19}{Indices and tables}{chapter.4}{}} -\gdef \@abspage@last{23} +\newlabel{4_sensitivity_analysis_diagnostic_and_exploratory_modeling:sensitivity-analysis-diagnostic-exploratory-modeling}{{4}{19}{Sensitivity Analysis: Diagnostic \& Exploratory Modeling}{chapter.4}{}} +\newlabel{4_sensitivity_analysis_diagnostic_and_exploratory_modeling::doc}{{4}{19}{Sensitivity Analysis: Diagnostic \& Exploratory Modeling}{chapter.4}{}} +\@writefile{toc}{\contentsline {section}{\numberline {4.1}Understanding Errors: What Is Controlling Model Performance?}{19}{section.4.1}\protected@file@percent } +\newlabel{4_sensitivity_analysis_diagnostic_and_exploratory_modeling:understanding-errors-what-is-controlling-model-performance}{{4.1}{19}{Understanding Errors: What Is Controlling Model Performance?}{section.4.1}{}} +\@writefile{toc}{\contentsline {section}{\numberline {4.2}Consequential Dynamics: What is Controlling Model Behaviors of Interest?}{19}{section.4.2}\protected@file@percent } +\newlabel{4_sensitivity_analysis_diagnostic_and_exploratory_modeling:consequential-dynamics-what-is-controlling-model-behaviors-of-interest}{{4.2}{19}{Consequential Dynamics: What is Controlling Model Behaviors of Interest?}{section.4.2}{}} +\@writefile{toc}{\contentsline {section}{\numberline {4.3}Consequential Scenarios: What is Controlling Consequential Outcomes?}{19}{section.4.3}\protected@file@percent } +\newlabel{4_sensitivity_analysis_diagnostic_and_exploratory_modeling:consequential-scenarios-what-is-controlling-consequential-outcomes}{{4.3}{19}{Consequential Scenarios: What is Controlling Consequential Outcomes?}{section.4.3}{}} +\@writefile{toc}{\contentsline {chapter}{\numberline {5}Uncertainty Quantification: The Basics}{21}{chapter.5}\protected@file@percent } +\@writefile{lof}{\addvspace {10\p@ }} +\@writefile{lot}{\addvspace {10\p@ }} +\newlabel{5_uncertainty_quantification_the_basics:uncertainty-quantification-the-basics}{{5}{21}{Uncertainty Quantification: The Basics}{chapter.5}{}} +\newlabel{5_uncertainty_quantification_the_basics::doc}{{5}{21}{Uncertainty Quantification: The Basics}{chapter.5}{}} +\@writefile{toc}{\contentsline {section}{\numberline {5.1}Why is Uncertainty Quantification Important for Understanding MultiSector System Dynamics?}{21}{section.5.1}\protected@file@percent } +\newlabel{5_uncertainty_quantification_the_basics:why-is-uncertainty-quantification-important-for-understanding-multisector-system-dynamics}{{5.1}{21}{Why is Uncertainty Quantification Important for Understanding MultiSector System Dynamics?}{section.5.1}{}} +\@writefile{toc}{\contentsline {section}{\numberline {5.2}Uncertainty Quantification for Exploratory Modeling}{21}{section.5.2}\protected@file@percent } +\newlabel{5_uncertainty_quantification_the_basics:uncertainty-quantification-for-exploratory-modeling}{{5.2}{21}{Uncertainty Quantification for Exploratory Modeling}{section.5.2}{}} +\@writefile{toc}{\contentsline {section}{\numberline {5.3}Bayesian Uncertainty Quantification}{21}{section.5.3}\protected@file@percent } +\newlabel{5_uncertainty_quantification_the_basics:bayesian-uncertainty-quantification}{{5.3}{21}{Bayesian Uncertainty Quantification}{section.5.3}{}} +\@writefile{toc}{\contentsline {section}{\numberline {5.4}Uncertainty Quantification Under (Deep) Uncertainty}{21}{section.5.4}\protected@file@percent } +\newlabel{5_uncertainty_quantification_the_basics:uncertainty-quantification-under-deep-uncertainty}{{5.4}{21}{Uncertainty Quantification Under (Deep) Uncertainty}{section.5.4}{}} +\@writefile{toc}{\contentsline {section}{\numberline {5.5}Integrating Model Diagnostics and Uncertainty Quantification}{21}{section.5.5}\protected@file@percent } +\newlabel{5_uncertainty_quantification_the_basics:integrating-model-diagnostics-and-uncertainty-quantification}{{5.5}{21}{Integrating Model Diagnostics and Uncertainty Quantification}{section.5.5}{}} +\@writefile{toc}{\contentsline {chapter}{\numberline {6}Uncertainty Quantification: A Tool For Capturing Risks \& Extremes}{23}{chapter.6}\protected@file@percent } +\@writefile{lof}{\addvspace {10\p@ }} +\@writefile{lot}{\addvspace {10\p@ }} +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:uncertainty-quantification-a-tool-for-capturing-risks-extremes}{{6}{23}{Uncertainty Quantification: A Tool For Capturing Risks \& Extremes}{chapter.6}{}} +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes::doc}{{6}{23}{Uncertainty Quantification: A Tool For Capturing Risks \& Extremes}{chapter.6}{}} +\@writefile{toc}{\contentsline {section}{\numberline {6.1}Understanding Risk: How Probable Are Extreme Events?}{23}{section.6.1}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:understanding-risk-how-probable-are-extreme-events}{{6.1}{23}{Understanding Risk: How Probable Are Extreme Events?}{section.6.1}{}} +\@writefile{toc}{\contentsline {section}{\numberline {6.2}Understanding Tails: Statistical Modeling of Extreme Events}{23}{section.6.2}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:understanding-tails-statistical-modeling-of-extreme-events}{{6.2}{23}{Understanding Tails: Statistical Modeling of Extreme Events}{section.6.2}{}} +\@writefile{toc}{\contentsline {section}{\numberline {6.3}How to Choose an Appropriate Method?}{23}{section.6.3}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:how-to-choose-an-appropriate-method}{{6.3}{23}{How to Choose an Appropriate Method?}{section.6.3}{}} +\@writefile{toc}{\contentsline {section}{\numberline {6.4}How to Select a Prior Distribution?}{23}{section.6.4}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:how-to-select-a-prior-distribution}{{6.4}{23}{How to Select a Prior Distribution?}{section.6.4}{}} +\@writefile{toc}{\contentsline {section}{\numberline {6.5}Posterior Predictive Checking}{23}{section.6.5}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:posterior-predictive-checking}{{6.5}{23}{Posterior Predictive Checking}{section.6.5}{}} +\@writefile{toc}{\contentsline {section}{\numberline {6.6}Model Selection and Comparison}{23}{section.6.6}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:model-selection-and-comparison}{{6.6}{23}{Model Selection and Comparison}{section.6.6}{}} +\@writefile{toc}{\contentsline {section}{\numberline {6.7}What are Common Methods?}{23}{section.6.7}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:what-are-common-methods}{{6.7}{23}{What are Common Methods?}{section.6.7}{}} +\@writefile{toc}{\contentsline {subsection}{\numberline {6.7.1}Scenario Discovery}{24}{subsection.6.7.1}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:scenario-discovery}{{6.7.1}{24}{Scenario Discovery}{subsection.6.7.1}{}} +\@writefile{toc}{\contentsline {subsection}{\numberline {6.7.2}Pre\sphinxhyphen {}calibration/GLUE}{24}{subsection.6.7.2}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:pre-calibration-glue}{{6.7.2}{24}{Pre\sphinxhyphen {}calibration/GLUE}{subsection.6.7.2}{}} +\@writefile{toc}{\contentsline {subsection}{\numberline {6.7.3}Markov Chain Monte Carlo}{24}{subsection.6.7.3}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:markov-chain-monte-carlo}{{6.7.3}{24}{Markov Chain Monte Carlo}{subsection.6.7.3}{}} +\@writefile{toc}{\contentsline {subsubsection}{\numberline {6.7.3.1}Metropolis\sphinxhyphen {}Hastings}{24}{subsubsection.6.7.3.1}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:metropolis-hastings}{{6.7.3.1}{24}{Metropolis\sphinxhyphen {}Hastings}{subsubsection.6.7.3.1}{}} +\@writefile{toc}{\contentsline {subsubsection}{\numberline {6.7.3.2}Gibbs Sampling}{24}{subsubsection.6.7.3.2}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:gibbs-sampling}{{6.7.3.2}{24}{Gibbs Sampling}{subsubsection.6.7.3.2}{}} +\@writefile{toc}{\contentsline {subsubsection}{\numberline {6.7.3.3}Hamiltonian Monte Carlo}{24}{subsubsection.6.7.3.3}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:hamiltonian-monte-carlo}{{6.7.3.3}{24}{Hamiltonian Monte Carlo}{subsubsection.6.7.3.3}{}} +\@writefile{toc}{\contentsline {subsection}{\numberline {6.7.4}Particle\sphinxhyphen {}based Methods}{24}{subsection.6.7.4}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:particle-based-methods}{{6.7.4}{24}{Particle\sphinxhyphen {}based Methods}{subsection.6.7.4}{}} +\@writefile{toc}{\contentsline {section}{\numberline {6.8}What are Example Software Implementations?}{24}{section.6.8}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:what-are-example-software-implementations}{{6.8}{24}{What are Example Software Implementations?}{section.6.8}{}} +\@writefile{toc}{\contentsline {subsection}{\numberline {6.8.1}Markov Chain Monte Carlo with the True Model}{25}{subsection.6.8.1}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:markov-chain-monte-carlo-with-the-true-model}{{6.8.1}{25}{Markov Chain Monte Carlo with the True Model}{subsection.6.8.1}{}} +\@writefile{toc}{\contentsline {subsection}{\numberline {6.8.2}Markov Chain Monte Carlo with Surrogate Models}{25}{subsection.6.8.2}\protected@file@percent } +\newlabel{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:markov-chain-monte-carlo-with-surrogate-models}{{6.8.2}{25}{Markov Chain Monte Carlo with Surrogate Models}{subsection.6.8.2}{}} +\@writefile{toc}{\contentsline {chapter}{\numberline {7}Conclusion}{27}{chapter.7}\protected@file@percent } +\@writefile{lof}{\addvspace {10\p@ }} +\@writefile{lot}{\addvspace {10\p@ }} +\newlabel{7_conclusion:conclusion}{{7}{27}{Conclusion}{chapter.7}{}} +\newlabel{7_conclusion::doc}{{7}{27}{Conclusion}{chapter.7}{}} +\@writefile{toc}{\contentsline {chapter}{\numberline {8}References}{29}{chapter.8}\protected@file@percent } +\@writefile{lof}{\addvspace {10\p@ }} +\@writefile{lot}{\addvspace {10\p@ }} +\newlabel{8_references:references}{{8}{29}{References}{chapter.8}{}} +\newlabel{8_references::doc}{{8}{29}{References}{chapter.8}{}} +\@writefile{toc}{\contentsline {chapter}{\numberline {9}Glossary}{37}{chapter.9}\protected@file@percent } +\@writefile{lof}{\addvspace {10\p@ }} +\@writefile{lot}{\addvspace {10\p@ }} +\newlabel{9_glossary:glossary}{{9}{37}{Glossary}{chapter.9}{}} +\newlabel{9_glossary::doc}{{9}{37}{Glossary}{chapter.9}{}} +\@writefile{toc}{\contentsline {chapter}{\numberline {10}Indices and tables}{39}{chapter.10}\protected@file@percent } +\@writefile{lof}{\addvspace {10\p@ }} +\@writefile{lot}{\addvspace {10\p@ }} +\newlabel{index:indices-and-tables}{{10}{39}{Indices and tables}{chapter.10}{}} +\gdef \@abspage@last{43} diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.fdb_latexmk b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.fdb_latexmk index 76d92993..036b9b0c 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.fdb_latexmk +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.fdb_latexmk @@ -1,10 +1,10 @@ # Fdb version 3 -["makeindex addressinguncertaintyinmultisectordynamicsresearch.idx"] 1622173776 "addressinguncertaintyinmultisectordynamicsresearch.idx" "addressinguncertaintyinmultisectordynamicsresearch.ind" "addressinguncertaintyinmultisectordynamicsresearch" 1622212414 - "addressinguncertaintyinmultisectordynamicsresearch.idx" 1622212413 0 d41d8cd98f00b204e9800998ecf8427e "pdflatex" +["makeindex addressinguncertaintyinmultisectordynamicsresearch.idx"] 1622173776 "addressinguncertaintyinmultisectordynamicsresearch.idx" "addressinguncertaintyinmultisectordynamicsresearch.ind" "addressinguncertaintyinmultisectordynamicsresearch" 1622216225 + "addressinguncertaintyinmultisectordynamicsresearch.idx" 1622216224 0 d41d8cd98f00b204e9800998ecf8427e "pdflatex" (generated) "addressinguncertaintyinmultisectordynamicsresearch.ind" "addressinguncertaintyinmultisectordynamicsresearch.ilg" -["pdflatex"] 1622212412 "addressinguncertaintyinmultisectordynamicsresearch.tex" "addressinguncertaintyinmultisectordynamicsresearch.pdf" "addressinguncertaintyinmultisectordynamicsresearch" 1622212414 +["pdflatex"] 1622216223 "addressinguncertaintyinmultisectordynamicsresearch.tex" "addressinguncertaintyinmultisectordynamicsresearch.pdf" "addressinguncertaintyinmultisectordynamicsresearch" 1622216225 "/usr/local/texlive/2021/texmf-dist/fonts/enc/dvips/tex-gyre/q-ec.enc" 1529098226 2457 aaabbccba4df2a7f7371410ee4e075a3 "" "/usr/local/texlive/2021/texmf-dist/fonts/enc/dvips/tex-gyre/q-ts1.enc" 1529098226 3124 3813fd4c981d99822890a2861b0d274c "" "/usr/local/texlive/2021/texmf-dist/fonts/map/fontname/texfonts.map" 1577235249 3524 cb3e574dea2d1052e39280babc910dc8 "" @@ -140,18 +140,18 @@ "/usr/local/texlive/2021/texmf-var/fonts/map/pdftex/updmap/pdftex.map" 1616695603 5169178 a2ce6e2d73f603b690db6777d06dbccb "" "/usr/local/texlive/2021/texmf-var/web2c/pdftex/pdflatex.fmt" 1622119425 2724741 fab0e2aa17d02fc4386894d351bd4006 "" "/usr/local/texlive/2021/texmf.cnf" 1616695596 577 4c1911a7228e554c6032256d3cf5567d "" - "addressinguncertaintyinmultisectordynamicsresearch.aux" 1622212414 16100 802cb559117f0432a39227364be59df3 "pdflatex" + "addressinguncertaintyinmultisectordynamicsresearch.aux" 1622216225 27207 b8b5eebe280bee8a00ef77228d788342 "pdflatex" "addressinguncertaintyinmultisectordynamicsresearch.ind" 1622173776 0 d41d8cd98f00b204e9800998ecf8427e "makeindex addressinguncertaintyinmultisectordynamicsresearch.idx" - "addressinguncertaintyinmultisectordynamicsresearch.out" 1622212414 6338 6cc1716505ee448ab7290a0293e847d0 "pdflatex" - "addressinguncertaintyinmultisectordynamicsresearch.tex" 1622212409 65814 ebcdd776ab37c12479033cca74785e67 "" - "addressinguncertaintyinmultisectordynamicsresearch.toc" 1622212414 2821 af6ca7c7630c79394a47ff99fcc50279 "pdflatex" + "addressinguncertaintyinmultisectordynamicsresearch.out" 1622216225 14478 cfdfe8e76528191e7cef8ab4eb0c584a "pdflatex" + "addressinguncertaintyinmultisectordynamicsresearch.tex" 1622216221 130282 1a5994c2f4eb6dbc9318b1e8ddf2f82b "" + "addressinguncertaintyinmultisectordynamicsresearch.toc" 1622216225 6067 eade09e14809b622e60ff2103e0150b9 "pdflatex" "figure1_state_of_the_science.png" 1622172293 57841 c8aee8d3cf5a71c8359d95609e510cf0 "" "figure2_idealized_uc.png" 1622174982 166028 44bbaea22f905869661692784fc39e0d "" "figure3_global_versus_local.png" 1622205961 395615 0d1b39495f78854678e10e9b32f50173 "" "figure4_factor_mapping.png" 1622207313 239434 1d143bcd8974f9ad4d41e540602d6318 "" "figure5_alternative_designs.png" 1622208986 66067 6abaa08d906ea0a6d872dc189f8655c5 "" "sphinx.sty" 1622058511 13210 1fc7fddaf4a9dba457504448864afe1f "" - "sphinxhighlight.sty" 1622212409 6679 76d10c62e0f0661410b46f5db6118e26 "" + "sphinxhighlight.sty" 1622216221 6679 76d10c62e0f0661410b46f5db6118e26 "" "sphinxlatexadmonitions.sty" 1622058511 6238 2d867d769abf3f72abc17ef52adff78b "" "sphinxlatexgraphics.sty" 1622058511 4840 a9578332b6f3b35e198751fb632c9b79 "" "sphinxlatexindbibtoc.sty" 1622058511 2066 b93f8504d02f6337fde3074b179de55e "" @@ -165,14 +165,14 @@ "sphinxlatexstyletext.sty" 1622058511 5771 a578900e23c0331bf3e01507feb96320 "" "sphinxlatextables.sty" 1622058511 21848 2827eb0b11b99b185a8b77317d3e131c "" "sphinxmanual.cls" 1622058511 4236 124cd90deb92742b5d3922bfc2cd70c0 "" - "sphinxmessages.sty" 1622212409 745 3f5fcd6cdd7964ed608767954a8ced6f "" + "sphinxmessages.sty" 1622216221 745 3f5fcd6cdd7964ed608767954a8ced6f "" "sphinxoptionsgeometry.sty" 1622058511 2061 47bb34b8ed8a78823eb0c886abfb9f4d "" "sphinxoptionshyperref.sty" 1622058511 1093 22dd608777751296c61a93174b0f4bb8 "" "sphinxpackagefootnote.sty" 1622058511 13248 e2c77871396275ccf592403f8130f6f2 "" (generated) "addressinguncertaintyinmultisectordynamicsresearch.pdf" + "addressinguncertaintyinmultisectordynamicsresearch.idx" "addressinguncertaintyinmultisectordynamicsresearch.toc" - "addressinguncertaintyinmultisectordynamicsresearch.out" "addressinguncertaintyinmultisectordynamicsresearch.aux" "addressinguncertaintyinmultisectordynamicsresearch.log" - "addressinguncertaintyinmultisectordynamicsresearch.idx" + "addressinguncertaintyinmultisectordynamicsresearch.out" diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.log b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.log index 80ca73ed..41355a1a 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.log +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.log @@ -1,4 +1,4 @@ -This is pdfTeX, Version 3.141592653-2.6-1.40.22 (TeX Live 2021) (preloaded format=pdflatex 2021.5.27) 28 MAY 2021 10:33 +This is pdfTeX, Version 3.141592653-2.6-1.40.22 (TeX Live 2021) (preloaded format=pdflatex 2021.5.27) 28 MAY 2021 11:37 entering extended mode restricted \write18 enabled. %&-line parsing enabled. @@ -782,20 +782,19 @@ File: umsb.fd 2013/01/14 v3.01 AMS symbols B ] -(./addressinguncertaintyinmultisectordynamicsresearch.toc) +(./addressinguncertaintyinmultisectordynamicsresearch.toc [1 + +]) \tf@toc=\write6 \openout6 = `addressinguncertaintyinmultisectordynamicsresearch.toc'. - [1 - -] [2 - -] + [2] Chapter 1. [1 + ] - + LaTeX Font Info: Trying to load font information for T1+txtt on input line 1 07. (/usr/local/texlive/2021/texmf-dist/tex/latex/txfonts/t1txtt.fd @@ -811,14 +810,14 @@ Package pdftex.def Info: figure1_state_of_the_science.png used on input line 1 ] Chapter 2. - + File: figure2_idealized_uc.png Graphic file (type png) Package pdftex.def Info: figure2_idealized_uc.png used on input line 129. (pdftex.def) Requested size: 469.75311pt x 287.44142pt. [5] [6 <./figure2_idealized_uc.png>] [7] [8] Chapter 3. - + File: figure3_global_versus_local.png Graphic file (type png) Package pdftex.def Info: figure3_global_versus_local.png used on input line 17 @@ -827,13 +826,13 @@ Package pdftex.def Info: figure3_global_versus_local.png used on input line 17 [9 ] [10 <./figure3_global_versus_local.png>] - + File: figure4_factor_mapping.png Graphic file (type png) Package pdftex.def Info: figure4_factor_mapping.png used on input line 213. (pdftex.def) Requested size: 376.40381pt x 314.06319pt. [11] [12 <./figure4_factor_mapping.png>] [13] - + File: figure5_alternative_designs.png Graphic file (type png) Package pdftex.def Info: figure5_alternative_designs.png used on input line 32 @@ -847,22 +846,46 @@ LaTeX Font Info: Trying to load font information for TS1+qtm on input line 3 File: ts1qtm.fd 2009/09/25 v1.2 font definition file for TS1/qtm ) [17] [18] Chapter 4. -(./addressinguncertaintyinmultisectordynamicsresearch.ind) [19 +[19 + +] [20 + +] +Chapter 5. +[21] [22 + +] +Chapter 6. +[23] [24] [25] [26 + +] +Chapter 7. +[27] [28 + +] +Chapter 8. +[29] [30] [31] [32] [33] [34] [35] [36] +Chapter 9. +[37 + +] [38 ] +Chapter 10. +(./addressinguncertaintyinmultisectordynamicsresearch.ind) [39] (./addressinguncertaintyinmultisectordynamicsresearch.aux) Package rerunfilecheck Info: File `addressinguncertaintyinmultisectordynamicsre search.out' has not changed. -(rerunfilecheck) Checksum: 6CC1716505EE448AB7290A0293E847D0;6338. +(rerunfilecheck) Checksum: CFDFE8E76528191E7CEF8AB4EB0C584A;14478. ) Here is how much of TeX's memory you used: - 13594 strings out of 478994 - 200629 string characters out of 5858184 - 549533 words of memory out of 5000000 - 30656 multiletter control sequences out of 15000+600000 + 13716 strings out of 478994 + 204918 string characters out of 5858184 + 711347 words of memory out of 5000000 + 30727 multiletter control sequences out of 15000+600000 462595 words of font info for 68 fonts, out of 8000000 for 9000 1142 hyphenation exceptions out of 8191 - 83i,17n,88p,3812b,393s stack positions out of 5000i,500n,10000p,200000b,80000s + 83i,17n,88p,3812b,569s stack positions out of 5000i,500n,10000p,200000b,80000s {/usr/local/texlive/2021/texmf-dist/fonts/enc/dvips/tex-gyre/q-ts1.enc}{/usr/ local/texlive/2021/texmf-dist/fonts/enc/dvips/tex-gyre/q-ec.enc} -Output written on addressinguncertaintyinmultisectordynamicsresearch.pdf (23 pa -ges, 1014769 bytes). +Output written on addressinguncertaintyinmultisectordynamicsresearch.pdf (43 pa +ges, 1085625 bytes). PDF statistics: - 325 PDF objects out of 1000 (max. 8388607) - 271 compressed objects within 3 object streams - 57 named destinations out of 1000 (max. 500000) - 262 words of extra memory for PDF output out of 10000 (max. 10000000) + 741 PDF objects out of 1000 (max. 8388607) + 663 compressed objects within 7 object streams + 108 named destinations out of 1000 (max. 500000) + 510 words of extra memory for PDF output out of 10000 (max. 10000000) diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.out b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.out index eb6c5f6f..b40f912d 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.out +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.out @@ -24,4 +24,35 @@ \BOOKMARK [2][-]{subsection.3.6.7}{\376\377\000M\000o\000m\000e\000n\000t\000-\000I\000n\000d\000e\000p\000e\000n\000d\000e\000n\000t\000\040\000\050\000D\000e\000n\000s\000i\000t\000y\000-\000B\000a\000s\000e\000d\000\051\000\040\000M\000e\000t\000h\000o\000d\000s}{section.3.6}% 24 \BOOKMARK [1][-]{section.3.7}{\376\377\000H\000o\000w\000\040\000T\000o\000\040\000C\000h\000o\000o\000s\000e\000\040\000A\000\040\000S\000e\000n\000s\000i\000t\000i\000v\000i\000t\000y\000\040\000A\000n\000a\000l\000y\000s\000i\000s\000\040\000M\000e\000t\000h\000o\000d\000:\000\040\000M\000o\000d\000e\000l\000\040\000T\000r\000a\000i\000t\000s\000\040\000A\000n\000d\000\040\000D\000i\000m\000e\000n\000s\000i\000o\000n\000a\000l\000i\000t\000y}{chapter.3}% 25 \BOOKMARK [1][-]{section.3.8}{\376\377\000S\000o\000f\000t\000w\000a\000r\000e\000\040\000T\000o\000o\000l\000k\000i\000t\000s}{chapter.3}% 26 -\BOOKMARK [0][-]{chapter.4}{\376\377\000I\000n\000d\000i\000c\000e\000s\000\040\000a\000n\000d\000\040\000t\000a\000b\000l\000e\000s}{}% 27 +\BOOKMARK [0][-]{chapter.4}{\376\377\000S\000e\000n\000s\000i\000t\000i\000v\000i\000t\000y\000\040\000A\000n\000a\000l\000y\000s\000i\000s\000:\000\040\000D\000i\000a\000g\000n\000o\000s\000t\000i\000c\000\040\000\046\000\040\000E\000x\000p\000l\000o\000r\000a\000t\000o\000r\000y\000\040\000M\000o\000d\000e\000l\000i\000n\000g}{}% 27 +\BOOKMARK [1][-]{section.4.1}{\376\377\000U\000n\000d\000e\000r\000s\000t\000a\000n\000d\000i\000n\000g\000\040\000E\000r\000r\000o\000r\000s\000:\000\040\000W\000h\000a\000t\000\040\000I\000s\000\040\000C\000o\000n\000t\000r\000o\000l\000l\000i\000n\000g\000\040\000M\000o\000d\000e\000l\000\040\000P\000e\000r\000f\000o\000r\000m\000a\000n\000c\000e\000?}{chapter.4}% 28 +\BOOKMARK [1][-]{section.4.2}{\376\377\000C\000o\000n\000s\000e\000q\000u\000e\000n\000t\000i\000a\000l\000\040\000D\000y\000n\000a\000m\000i\000c\000s\000:\000\040\000W\000h\000a\000t\000\040\000i\000s\000\040\000C\000o\000n\000t\000r\000o\000l\000l\000i\000n\000g\000\040\000M\000o\000d\000e\000l\000\040\000B\000e\000h\000a\000v\000i\000o\000r\000s\000\040\000o\000f\000\040\000I\000n\000t\000e\000r\000e\000s\000t\000?}{chapter.4}% 29 +\BOOKMARK [1][-]{section.4.3}{\376\377\000C\000o\000n\000s\000e\000q\000u\000e\000n\000t\000i\000a\000l\000\040\000S\000c\000e\000n\000a\000r\000i\000o\000s\000:\000\040\000W\000h\000a\000t\000\040\000i\000s\000\040\000C\000o\000n\000t\000r\000o\000l\000l\000i\000n\000g\000\040\000C\000o\000n\000s\000e\000q\000u\000e\000n\000t\000i\000a\000l\000\040\000O\000u\000t\000c\000o\000m\000e\000s\000?}{chapter.4}% 30 +\BOOKMARK [0][-]{chapter.5}{\376\377\000U\000n\000c\000e\000r\000t\000a\000i\000n\000t\000y\000\040\000Q\000u\000a\000n\000t\000i\000f\000i\000c\000a\000t\000i\000o\000n\000:\000\040\000T\000h\000e\000\040\000B\000a\000s\000i\000c\000s}{}% 31 +\BOOKMARK [1][-]{section.5.1}{\376\377\000W\000h\000y\000\040\000i\000s\000\040\000U\000n\000c\000e\000r\000t\000a\000i\000n\000t\000y\000\040\000Q\000u\000a\000n\000t\000i\000f\000i\000c\000a\000t\000i\000o\000n\000\040\000I\000m\000p\000o\000r\000t\000a\000n\000t\000\040\000f\000o\000r\000\040\000U\000n\000d\000e\000r\000s\000t\000a\000n\000d\000i\000n\000g\000\040\000M\000u\000l\000t\000i\000S\000e\000c\000t\000o\000r\000\040\000S\000y\000s\000t\000e\000m\000\040\000D\000y\000n\000a\000m\000i\000c\000s\000?}{chapter.5}% 32 +\BOOKMARK [1][-]{section.5.2}{\376\377\000U\000n\000c\000e\000r\000t\000a\000i\000n\000t\000y\000\040\000Q\000u\000a\000n\000t\000i\000f\000i\000c\000a\000t\000i\000o\000n\000\040\000f\000o\000r\000\040\000E\000x\000p\000l\000o\000r\000a\000t\000o\000r\000y\000\040\000M\000o\000d\000e\000l\000i\000n\000g}{chapter.5}% 33 +\BOOKMARK [1][-]{section.5.3}{\376\377\000B\000a\000y\000e\000s\000i\000a\000n\000\040\000U\000n\000c\000e\000r\000t\000a\000i\000n\000t\000y\000\040\000Q\000u\000a\000n\000t\000i\000f\000i\000c\000a\000t\000i\000o\000n}{chapter.5}% 34 +\BOOKMARK [1][-]{section.5.4}{\376\377\000U\000n\000c\000e\000r\000t\000a\000i\000n\000t\000y\000\040\000Q\000u\000a\000n\000t\000i\000f\000i\000c\000a\000t\000i\000o\000n\000\040\000U\000n\000d\000e\000r\000\040\000\050\000D\000e\000e\000p\000\051\000\040\000U\000n\000c\000e\000r\000t\000a\000i\000n\000t\000y}{chapter.5}% 35 +\BOOKMARK [1][-]{section.5.5}{\376\377\000I\000n\000t\000e\000g\000r\000a\000t\000i\000n\000g\000\040\000M\000o\000d\000e\000l\000\040\000D\000i\000a\000g\000n\000o\000s\000t\000i\000c\000s\000\040\000a\000n\000d\000\040\000U\000n\000c\000e\000r\000t\000a\000i\000n\000t\000y\000\040\000Q\000u\000a\000n\000t\000i\000f\000i\000c\000a\000t\000i\000o\000n}{chapter.5}% 36 +\BOOKMARK [0][-]{chapter.6}{\376\377\000U\000n\000c\000e\000r\000t\000a\000i\000n\000t\000y\000\040\000Q\000u\000a\000n\000t\000i\000f\000i\000c\000a\000t\000i\000o\000n\000:\000\040\000A\000\040\000T\000o\000o\000l\000\040\000F\000o\000r\000\040\000C\000a\000p\000t\000u\000r\000i\000n\000g\000\040\000R\000i\000s\000k\000s\000\040\000\046\000\040\000E\000x\000t\000r\000e\000m\000e\000s}{}% 37 +\BOOKMARK [1][-]{section.6.1}{\376\377\000U\000n\000d\000e\000r\000s\000t\000a\000n\000d\000i\000n\000g\000\040\000R\000i\000s\000k\000:\000\040\000H\000o\000w\000\040\000P\000r\000o\000b\000a\000b\000l\000e\000\040\000A\000r\000e\000\040\000E\000x\000t\000r\000e\000m\000e\000\040\000E\000v\000e\000n\000t\000s\000?}{chapter.6}% 38 +\BOOKMARK [1][-]{section.6.2}{\376\377\000U\000n\000d\000e\000r\000s\000t\000a\000n\000d\000i\000n\000g\000\040\000T\000a\000i\000l\000s\000:\000\040\000S\000t\000a\000t\000i\000s\000t\000i\000c\000a\000l\000\040\000M\000o\000d\000e\000l\000i\000n\000g\000\040\000o\000f\000\040\000E\000x\000t\000r\000e\000m\000e\000\040\000E\000v\000e\000n\000t\000s}{chapter.6}% 39 +\BOOKMARK [1][-]{section.6.3}{\376\377\000H\000o\000w\000\040\000t\000o\000\040\000C\000h\000o\000o\000s\000e\000\040\000a\000n\000\040\000A\000p\000p\000r\000o\000p\000r\000i\000a\000t\000e\000\040\000M\000e\000t\000h\000o\000d\000?}{chapter.6}% 40 +\BOOKMARK [1][-]{section.6.4}{\376\377\000H\000o\000w\000\040\000t\000o\000\040\000S\000e\000l\000e\000c\000t\000\040\000a\000\040\000P\000r\000i\000o\000r\000\040\000D\000i\000s\000t\000r\000i\000b\000u\000t\000i\000o\000n\000?}{chapter.6}% 41 +\BOOKMARK [1][-]{section.6.5}{\376\377\000P\000o\000s\000t\000e\000r\000i\000o\000r\000\040\000P\000r\000e\000d\000i\000c\000t\000i\000v\000e\000\040\000C\000h\000e\000c\000k\000i\000n\000g}{chapter.6}% 42 +\BOOKMARK [1][-]{section.6.6}{\376\377\000M\000o\000d\000e\000l\000\040\000S\000e\000l\000e\000c\000t\000i\000o\000n\000\040\000a\000n\000d\000\040\000C\000o\000m\000p\000a\000r\000i\000s\000o\000n}{chapter.6}% 43 +\BOOKMARK [1][-]{section.6.7}{\376\377\000W\000h\000a\000t\000\040\000a\000r\000e\000\040\000C\000o\000m\000m\000o\000n\000\040\000M\000e\000t\000h\000o\000d\000s\000?}{chapter.6}% 44 +\BOOKMARK [2][-]{subsection.6.7.1}{\376\377\000S\000c\000e\000n\000a\000r\000i\000o\000\040\000D\000i\000s\000c\000o\000v\000e\000r\000y}{section.6.7}% 45 +\BOOKMARK [2][-]{subsection.6.7.2}{\376\377\000P\000r\000e\000-\000c\000a\000l\000i\000b\000r\000a\000t\000i\000o\000n\000/\000G\000L\000U\000E}{section.6.7}% 46 +\BOOKMARK [2][-]{subsection.6.7.3}{\376\377\000M\000a\000r\000k\000o\000v\000\040\000C\000h\000a\000i\000n\000\040\000M\000o\000n\000t\000e\000\040\000C\000a\000r\000l\000o}{section.6.7}% 47 +\BOOKMARK [3][-]{subsubsection.6.7.3.1}{\376\377\000M\000e\000t\000r\000o\000p\000o\000l\000i\000s\000-\000H\000a\000s\000t\000i\000n\000g\000s}{subsection.6.7.3}% 48 +\BOOKMARK [3][-]{subsubsection.6.7.3.2}{\376\377\000G\000i\000b\000b\000s\000\040\000S\000a\000m\000p\000l\000i\000n\000g}{subsection.6.7.3}% 49 +\BOOKMARK [3][-]{subsubsection.6.7.3.3}{\376\377\000H\000a\000m\000i\000l\000t\000o\000n\000i\000a\000n\000\040\000M\000o\000n\000t\000e\000\040\000C\000a\000r\000l\000o}{subsection.6.7.3}% 50 +\BOOKMARK [2][-]{subsection.6.7.4}{\376\377\000P\000a\000r\000t\000i\000c\000l\000e\000-\000b\000a\000s\000e\000d\000\040\000M\000e\000t\000h\000o\000d\000s}{section.6.7}% 51 +\BOOKMARK [1][-]{section.6.8}{\376\377\000W\000h\000a\000t\000\040\000a\000r\000e\000\040\000E\000x\000a\000m\000p\000l\000e\000\040\000S\000o\000f\000t\000w\000a\000r\000e\000\040\000I\000m\000p\000l\000e\000m\000e\000n\000t\000a\000t\000i\000o\000n\000s\000?}{chapter.6}% 52 +\BOOKMARK [2][-]{subsection.6.8.1}{\376\377\000M\000a\000r\000k\000o\000v\000\040\000C\000h\000a\000i\000n\000\040\000M\000o\000n\000t\000e\000\040\000C\000a\000r\000l\000o\000\040\000w\000i\000t\000h\000\040\000t\000h\000e\000\040\000T\000r\000u\000e\000\040\000M\000o\000d\000e\000l}{section.6.8}% 53 +\BOOKMARK [2][-]{subsection.6.8.2}{\376\377\000M\000a\000r\000k\000o\000v\000\040\000C\000h\000a\000i\000n\000\040\000M\000o\000n\000t\000e\000\040\000C\000a\000r\000l\000o\000\040\000w\000i\000t\000h\000\040\000S\000u\000r\000r\000o\000g\000a\000t\000e\000\040\000M\000o\000d\000e\000l\000s}{section.6.8}% 54 +\BOOKMARK [0][-]{chapter.7}{\376\377\000C\000o\000n\000c\000l\000u\000s\000i\000o\000n}{}% 55 +\BOOKMARK [0][-]{chapter.8}{\376\377\000R\000e\000f\000e\000r\000e\000n\000c\000e\000s}{}% 56 +\BOOKMARK [0][-]{chapter.9}{\376\377\000G\000l\000o\000s\000s\000a\000r\000y}{}% 57 +\BOOKMARK [0][-]{chapter.10}{\376\377\000I\000n\000d\000i\000c\000e\000s\000\040\000a\000n\000d\000\040\000t\000a\000b\000l\000e\000s}{}% 58 diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.pdf b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.pdf index c5311647..24b7f321 100644 Binary files a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.pdf and b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.pdf differ diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.tex b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.tex index 807a9059..32d2f262 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.tex +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.tex @@ -397,6 +397,359 @@ \section{How To Choose A Sensitivity Analysis Method: Model Traits And Dimension \section{Software Toolkits} \label{\detokenize{3_sensitivity_analysis_the_basics:software-toolkits}} +\chapter{Sensitivity Analysis: Diagnostic \& Exploratory Modeling} +\label{\detokenize{4_sensitivity_analysis_diagnostic_and_exploratory_modeling:sensitivity-analysis-diagnostic-exploratory-modeling}}\label{\detokenize{4_sensitivity_analysis_diagnostic_and_exploratory_modeling::doc}} + +\section{Understanding Errors: What Is Controlling Model Performance?} +\label{\detokenize{4_sensitivity_analysis_diagnostic_and_exploratory_modeling:understanding-errors-what-is-controlling-model-performance}} + +\section{Consequential Dynamics: What is Controlling Model Behaviors of Interest?} +\label{\detokenize{4_sensitivity_analysis_diagnostic_and_exploratory_modeling:consequential-dynamics-what-is-controlling-model-behaviors-of-interest}} + +\section{Consequential Scenarios: What is Controlling Consequential Outcomes?} +\label{\detokenize{4_sensitivity_analysis_diagnostic_and_exploratory_modeling:consequential-scenarios-what-is-controlling-consequential-outcomes}} + +\chapter{Uncertainty Quantification: The Basics} +\label{\detokenize{5_uncertainty_quantification_the_basics:uncertainty-quantification-the-basics}}\label{\detokenize{5_uncertainty_quantification_the_basics::doc}} +\sphinxAtStartPar +As described in the previous sections, uncertainty characterization (UC) can be defined as exploratory modeling where alternative hypotheses for the co\sphinxhyphen{}evolutionary dynamics of influences, stressors, as well as path\sphinxhyphen{}dependent changes in the form and function of systems are explored (Marchau et al., 2019). UC exploratory modeling has a consistent focus on the assumptions, structural model forms, alternative parameterizations, and input data sets that are used to characterize the behavioral space of one or more models. The focus of UC is not to exactly quantify and predict probabilistic likelihoods for all possible quantities, but instead to inform which modeling choices yield the most consequential behavioral changes or outcomes, especially when considering deeply uncertain, scenario\sphinxhyphen{}informed projections (Moallemi et al., 2020b; Walker et al., 2013). + +\sphinxAtStartPar +In comparison, uncertainty quantification (UQ) refers to the representation of uncertainties using probability distributions. The act of quantification requires specific assumptions about distributional forms and likelihoods, which may be more or less justified depending on prior information about the system or model behavior (Frankignoul and Hasselmann, 1977; Zellner and Tian, 1964). Without this justification, alternative specifications may yield substantially different inferences. + + +\section{Why is Uncertainty Quantification Important for Understanding MultiSector System Dynamics?} +\label{\detokenize{5_uncertainty_quantification_the_basics:why-is-uncertainty-quantification-important-for-understanding-multisector-system-dynamics}} + +\section{Uncertainty Quantification for Exploratory Modeling} +\label{\detokenize{5_uncertainty_quantification_the_basics:uncertainty-quantification-for-exploratory-modeling}} + +\section{Bayesian Uncertainty Quantification} +\label{\detokenize{5_uncertainty_quantification_the_basics:bayesian-uncertainty-quantification}} + +\section{Uncertainty Quantification Under (Deep) Uncertainty} +\label{\detokenize{5_uncertainty_quantification_the_basics:uncertainty-quantification-under-deep-uncertainty}} + +\section{Integrating Model Diagnostics and Uncertainty Quantification} +\label{\detokenize{5_uncertainty_quantification_the_basics:integrating-model-diagnostics-and-uncertainty-quantification}} + +\chapter{Uncertainty Quantification: A Tool For Capturing Risks \& Extremes} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:uncertainty-quantification-a-tool-for-capturing-risks-extremes}}\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes::doc}} + +\section{Understanding Risk: How Probable Are Extreme Events?} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:understanding-risk-how-probable-are-extreme-events}} + +\section{Understanding Tails: Statistical Modeling of Extreme Events} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:understanding-tails-statistical-modeling-of-extreme-events}} + +\section{How to Choose an Appropriate Method?} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:how-to-choose-an-appropriate-method}} + +\section{How to Select a Prior Distribution?} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:how-to-select-a-prior-distribution}} + +\section{Posterior Predictive Checking} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:posterior-predictive-checking}} + +\section{Model Selection and Comparison} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:model-selection-and-comparison}} + +\section{What are Common Methods?} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:what-are-common-methods}} +\sphinxAtStartPar +There are many methods to quantify uncertainty. Each method has advantages and disadvantages for a particular analysis. Here we focus on parametric uncertainty quantification, as a discussion of structural uncertainty quantification is beyond the scope of this review. Moreover, we prefer to think about structural uncertainty from the perspective of exploratory modeling and deep uncertainty, rather than from the perspective of quantification and selection or averaging. + +\sphinxAtStartPar +Uncertainty quantification methods can be broadly classified as Markov Chain Monte Carlo (MCMC) approaches, particle\sphinxhyphen{}based approaches, and emulation\sphinxhyphen{}based approaches, though there are some hybrid methods. Several of the most common approaches for uncertainty quantification are described below. In all cases, the computational and conceptual challenges associated with parametric uncertainty quantification grow rapidly with the number of model parameters. As noted in the prior sections, sensitivity analyses are useful for dimensionality reduction prior to conducting parametric uncertainty quantification. Both factor fixing and factor prioritization can be used to limit the number of parameters which are treated as uncertain. + + +\subsection{Scenario Discovery} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:scenario-discovery}} + +\subsection{Pre\sphinxhyphen{}calibration/GLUE} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:pre-calibration-glue}} + +\subsection{Markov Chain Monte Carlo} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:markov-chain-monte-carlo}} +\sphinxAtStartPar +Markov chain Monte Carlo (MCMC) is a “gold standard” approach to full uncertainty quantification. MCMC refers to a category of algorithms which systematically sample from a target distribution (in this case, the posterior distribution) by constructing a Markov chain. MCMC algorithms rely on the mixing properties of the resulting Markov chain to guarantee asymptotic convergence to the posterior distribution, as the chain is constructed so that the posterior is its stationary distribution. It should be stressed that this guarantee exists only asymptotically. Studies use heuristics to test for signs of misconvergence and to assess the skill of the approximation (xx) + +\sphinxAtStartPar +MCMC algorithms begin with the choice of some initial value for the Markov chain. This value can be randomly determined, or can be some other quantity such as a maximum likelihood or maximum a posteriori estimates. While the Markov chain will eventually converge to the posterior regardless of the choice of initial value, the amount of time required to escape the transient dynamics of the Markov chain is dependent on this value. Typically, transient samples are discarded as burn\sphinxhyphen{}in, as they may skew the sample distribution if the burn\sphinxhyphen{}in is relatively long compared to the number of iterations spent exploring the posterior, though this practice is not universal and has been questioned by some statisticians (Geyer, 2011). However, when not discarding the transient area, the chain must be run for a larger number of iterations to ensure that these samples do not bias the sample distribution. + +\sphinxAtStartPar +Diagnosing the convergence of the Markov chain to the posterior is more art than science, relying on heuristics and judgement. One example heuristic is to run many Markov chains from different initial conditions, ideally well\sphinxhyphen{}dispersed across the parameter space; one may be able to conclude that the chains have not yet converged if the resulting marginal parameter distributions are sufficiently different when plotted. The Gelman\sphinxhyphen{}Rubin diagnostic formalizes this idea by comparing the within\sphinxhyphen{}chain and pooled variances of multiple chains (Gelman and Rubin, 1992). The ratio of these two quantities, called the potential scale reduction factor, can diagnose a lack of convergence if it is sufficiently far from 1 (typically using a threshold such as 1.1 or 1.05). Thus, it is generally good practice to use several MCMC runs to facilitate the diagnoses of non\sphinxhyphen{}convergence. + +\sphinxAtStartPar +Another key value is the effective sample size (ESS). Due to the Markovian property, the samples obtained using MCMC are autocorrelated, and therefore not independent. As a result, the number of samples obtained using MCMC are not directly useful when interpreting the extent of exploration (or computing quantities such as the Monte Carlo standard error (Flegal et al., 2008)). For example, it may not be appropriate to draw inferences about tail properties for a small ESS. + +\sphinxAtStartPar +Many MCMC algorithms exist, with varying strengths and weaknesses, discussed below. For example, some require more tuning to improve the ESS than others. All of these algorithms involve the evaluation of the model at various parameter settings. Once a Markov chain is constructed and deemed to suitably represent the posterior distribution, parameter values can be sampled from it with replacement as a proxy for directly sampling from the posterior. + + +\subsubsection{Metropolis\sphinxhyphen{}Hastings} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:metropolis-hastings}} + +\subsubsection{Gibbs Sampling} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:gibbs-sampling}} + +\subsubsection{Hamiltonian Monte Carlo} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:hamiltonian-monte-carlo}} + +\subsection{Particle\sphinxhyphen{}based Methods} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:particle-based-methods}} + +\section{What are Example Software Implementations?} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:what-are-example-software-implementations}} +\sphinxAtStartPar +There exist many software platforms to implement uncertainty assessment. Each implementation is built upon a specific programming language including, but not limited to R, Python, C++, Fortran, MATLAB, and Julia. Two key considerations are the user’s preferred programming language and the computer model’s native code. For instance, a computer model running in C++ may be better suited for a software implementation based on the same language. For inconsistencies, please see the discussion on wrappers below. + +\sphinxAtStartPar +Here, we present an overview of popular packages inherent to R, Python, and Julia. The user is free to code the UQ implementation without incorporating these existing packages; however, it may require more effort to code the pertinent subroutines (e.g., MCMC and building surrogate models). Uncertainty quantification for computer models typically operates within the Bayesian framework (see What are Common Methods?). Each implementation includes a mechanism that enables Bayesian inference using MCMC, Gaussian process emulation, or Sequential Monte Carlo. We focus on a subset of the common approaches. + + +\subsection{Markov Chain Monte Carlo with the True Model} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:markov-chain-monte-carlo-with-the-true-model}} + +\subsection{Markov Chain Monte Carlo with Surrogate Models} +\label{\detokenize{6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes:markov-chain-monte-carlo-with-surrogate-models}} + +\chapter{Conclusion} +\label{\detokenize{7_conclusion:conclusion}}\label{\detokenize{7_conclusion::doc}} +\sphinxAtStartPar +As noted in the Introduction (Section 1.0), the computational and conceptual challenges of the multi\sphinxhyphen{}model, transdisciplinary workflows that characterize ambitious projects such as IM3 have limited UC and UQ analyses. Moreover, the very nature and purpose of modeling and diagnostic model evaluation can have very diverse philosophical framings depending on the disciplines involved (see Figure 1 and Section 2.0). The guidance provided in this text can be used to frame consistent and rigorous experimental designs for better understanding the consequences and insights from our modeling choices when seeking to capture complex human\sphinxhyphen{}natural systems. The progression of sections of this text provide a thorough introduction of the concepts and definitions of diagnostic model evaluation, sensitivity analysis, UC, and UQ. In addition, we comprehensively discuss how specific modeling objectives and applications should guide the selection of appropriate techniques; broadly, these can include model diagnostics, in\sphinxhyphen{}depth analysis of the behavior of the abstracted system, and projections under conditions of deep uncertainty. This text also contains a detailed presentation of the main sensitivity analysis, UC, and UQ analysis methods and a discussion of their features and main limitations. Readers are also provided with an overview of computer tools and platforms that have been developed and could be considered in addressing IM3 scientific questions. The appendices of this text include a terminology glossary of the key concepts as well as example test cases and scripts to showcase various UC related capabilities. + +\sphinxAtStartPar +Although we distinguish the UC and UQ model diagnostics, the reader should note that we suggest an overall consistent approach to both in this text by emphasizing “exploratory modeling” (see review add citation). Although data support, model complexity, and computational limits strongly distinguish the feasibility and appropriateness of the UC and UQ diagnostic tools (e.g., see Figure 18), we overall recommend that modelers view their work through the lens of cycles of learning. Iterative and deliberative exploration of model\sphinxhyphen{}based hypotheses and inferences for transdisciplinary teams is non\sphinxhyphen{}trivial and ultimately critical for mapping where innovations or insights are most consequential. Overall, we recommend approaching modeling with an openness to the diverse disciplinary perspectives such as those mirrored by the IM3 family of models in a progression from evaluating models relative to observed history to advanced formalized analyses to make inferences on multi\sphinxhyphen{}sector, multi\sphinxhyphen{}scale vulnerabilities and resilience. Exploratory modeling approaches can help fashion experiments with large numbers of alternative hypotheses on the co\sphinxhyphen{}evolutionary dynamics of influences, stressors, as well as path\sphinxhyphen{}dependent changes in the form and function of coupled human\sphinxhyphen{}natural systems (Weaver et al., 2013). This text guides the reader through the use of sensitivity analysis and uncertainty methods across the diverse perspectives that have shaped modern diagnostic and exploratory modeling. + + +\chapter{References} +\label{\detokenize{8_references:references}}\label{\detokenize{8_references::doc}} +\sphinxAtStartPar +Akaike, H., 1978. On the Likelihood of a Time Series Model. J. R. Stat. Soc. Ser. Stat. 27, 217\textendash{}235. \sphinxurl{https://doi.org/10.2307/2988185} +Akaike, H., 1974. A new look at the statistical model identification. IEEE Trans. Autom. Control 19, 716\textendash{}723. \sphinxurl{https://doi.org/10.1109/TAC.1974.1100705} +Anderson, B., Borgonovo, E., Galeotti, M., Roson, R., 2014. Uncertainty in Climate Change Modeling: Can Global Sensitivity Analysis Be of Help? Risk Anal. 34, 271\textendash{}293. \sphinxurl{https://doi.org/10.1111/risa.12117} +Annan, J.D., Hargreaves, J.C., 2004. Efficient parameter estimation for a highly chaotic system. Tellus A 56, 520\textendash{}526. \sphinxurl{https://doi.org/10.1111/j.1600-0870.2004.00073.x} +Bakker, A.M.R., Wong, T.E., Ruckert, K.L., Keller, K., 2017. Sea\sphinxhyphen{}level projections representing the deeply uncertain contribution of the West Antarctic ice sheet. Sci Rep 7, 3880. \sphinxurl{https://doi.org/10.1038/s41598-017-04134-5} +Bankes, S., 1993. Exploratory Modeling for Policy Analysis. Oper. Res. 41, 435\textendash{}449. \sphinxurl{https://doi.org/10.1287/opre.41.3.435} +Barlas, Y., 1996. Formal aspects of model validity and validation in system dynamics. Syst. Dyn. Rev. J. Syst. Dyn. Soc. 12, 183\textendash{}210. +Barlas, Y., Carpenter, S., 1990. Philosophical roots of model validation: Two paradigms. Syst. Dyn. Rev. 6, 148\textendash{}166. \sphinxurl{https://doi.org/10.1002/sdr.4260060203} +Bastos, L.S., O’Hagan, A., 2009. Diagnostics for Gaussian Process Emulators. Technometrics 51, 425\textendash{}438. \sphinxurl{https://doi.org/10.1198/TECH.2009.08019} +Batouli, M., Mostafavi, A., 2018. Multiagent Simulation for Complex Adaptive Modeling of Road Infrastructure Resilience to Sea\sphinxhyphen{}Level Rise. Comput.\sphinxhyphen{}Aided Civ. Infrastruct. Eng. 33, 393\textendash{}410. \sphinxurl{https://doi.org/10.1111/mice.12348} +Bayes, T., 1763. An Essay towards Solving a Problem in the Doctrine of Chance. Philos. Trans. R. Soc. Lond. 53, 370\textendash{}418. +Betancourt, M., 2017. A Conceptual Introduction to Hamiltonian Monte Carlo. ArXiv StatME. +Beven, K., 2002. Towards a coherent philosophy for modelling the environment. Proc. R. Soc. Lond. Ser. Math. Phys. Eng. Sci. 458, 2465\textendash{}2484. \sphinxurl{https://doi.org/10.1098/rspa.2002.0986} +Beven, K., 1993. Prophecy, reality and uncertainty in distributed hydrological modelling. Adv. Water Resour., Research Perspectives in Hydrology 16, 41\textendash{}51. \sphinxurl{https://doi.org/10.1016/0309-1708(93)90028-E} +Beven, K., Binley, A., 1992. The future of distributed models: Model calibration and uncertainty prediction. Hydrol. Process. 6, 279\textendash{}298. \sphinxurl{https://doi.org/10.1002/hyp.3360060305} +Beven, K., Freer, J., 2001. Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology. J Hydrol. +Beven, K.J., 2018. On hypothesis testing in hydrology: Why falsification of models is still a really good idea. WIREs Water 5, e1278. \sphinxurl{https://doi.org/10.1002/wat2.1278} +Bobée, B., 1975. The Log Pearson type 3 distribution and its application in hydrology. Water Resour Res 11, 681\textendash{}689. \sphinxurl{https://doi.org/10.1029/WR011i005p00681} +Bobée, B., Rasmussen, P.F., 1995. Recent advances in flood frequency analysis. Rev Geophys 33, 1111\textendash{}1116. \sphinxurl{https://doi.org/10.1029/95RG00287} +Borgomeo, E., Farmer, C.L., Hall, J.W., 2015. Numerical rivers: A synthetic streamflow generator for water resources vulnerability assessments. Water Resour. Res. 51, 5382\textendash{}5405. \sphinxurl{https://doi.org/10.1002/2014WR016827} +Borgonovo, E., 2010. Sensitivity analysis with finite changes: An application to modified EOQ models. Eur. J. Oper. Res. 200, 127\textendash{}138. +Borgonovo, E., Plischke, E., 2016. Sensitivity analysis: A review of recent advances. Eur. J. Oper. Res. 248, 869\textendash{}887. \sphinxurl{https://doi.org/10.1016/j.ejor.2015.06.032} +Boukouvalas, A., Sykes, P., Cornford, D., Maruri\sphinxhyphen{}Aguilar, H., 2014. Bayesian Precalibration of a Large Stochastic Microsimulation Model. IEEE Trans Intell Transp Syst 15, 1337\textendash{}1347. \sphinxurl{https://doi.org/10.1109/tits.2014.2304394} +Box, G.E., Hunter, J.S., 1961. The 2 k—p fractional factorial designs. Technometrics 3, 311\textendash{}351. +Bryant, B.P., Lempert, R.J., 2010. Thinking inside the box: A participatory, computer\sphinxhyphen{}assisted approach to scenario discovery. Technol. Forecast. Soc. Change 77, 34\textendash{}49. \sphinxurl{https://doi.org/10.1016/j.techfore.2009.08.002} +Budescu, D.V., Broomell, S.B., Lempert, R.J., Keller, K., 2014. Aided and unaided decisions with imprecise probabilities in the domain of losses. EURO J. Decis. Process. 2, 31\textendash{}62. \sphinxurl{https://doi.org/10.1007/s40070-013-0023-4} +Bürkner, P.\sphinxhyphen{}C., Gabry, J., Vehtari, A., 2019. Approximate leave\sphinxhyphen{}future\sphinxhyphen{}out cross\sphinxhyphen{}validation for Bayesian time series models. ArXiv StatME. +Burnham, K.P., Anderson, D.R., 2004. Multimodel Inference: Understanding AIC and BIC in Model Selection. Sociol Methods Res 33, 261\textendash{}304. \sphinxurl{https://doi.org/10.1177/0049124104268644} +Burnham, K.P., Anderson, D.R., 2001. Kullback\sphinxhyphen{}Leibler information as a basis for strong inference in ecological studies. Wildl Res 28, 111\textendash{}119. \sphinxurl{https://doi.org/10.1071/wr99107} +Cao, Y., Li, B., 2019. Assessing models for estimation and methods for uncertainty quantification for spatial return levels. Environmetrics 30, e2508. +Cash, D.W., Clark, W.C., Alcock, F., Dickson, N.M., Eckley, N., Guston, D.H., Jäger, J., Mitchell, R.B., 2003. Knowledge systems for sustainable development. Proc. Natl. Acad. Sci. 100, 8086\textendash{}8091. +Ciccazzo, A., Pillo, G.D., Latorre, V., 2014. Support vector machines for surrogate modeling of electronic circuits. Neural Comput. Appl. 24, 69\textendash{}76. \sphinxurl{https://doi.org/10.1007/s00521-013-1509-5} +Coles, S., Bawa, J., Trenner, L., Dorazio, P., 2001. An introduction to statistical modeling of extreme values. Springer. +Cooke, R.M., 1991. Experts in uncertainty: Opinion and subjective probability in science, Experts in uncertainty: Opinion and subjective probability in science. Oxford University Press, New York, NY, US. +Coronese, M., Lamperti, F., Keller, K., Chiaromonte, F., Roventini, A., 2019. Evidence for sharp increase in the economic damages of extreme natural disasters. Proc Natl Acad Sci U A 201907826. \sphinxurl{https://doi.org/10.1073/pnas.1907826116} +Cukier, R.I., Fortuin, C.M., Shuler, K.E., Petschek, A.G., Schaibly, J.H., 1973. Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory. J. Chem. Phys. 59, 3873\textendash{}3878. \sphinxurl{https://doi.org/10.1063/1.1680571} +Currin, C., Mitchell, T., Morris, M., Ylvisaker, D., 1991. Bayesian Prediction of Deterministic Functions, with Applications to the Design and Analysis of Computer Experiments. J Am Stat Assoc 86, 953\textendash{}963. \sphinxurl{https://doi.org/10.1080/01621459.1991.10475138} +Dalal, I.L., Stefan, D., Harwayne\sphinxhyphen{}Gidansky, J., 2008. Low discrepancy sequences for Monte Carlo simulations on reconfigurable platforms, in: 2008 International Conference on Application\sphinxhyphen{}Specific Systems, Architectures and Processors. Presented at the 2008 International Conference on Application\sphinxhyphen{}Specific Systems, Architectures and Processors, pp. 108\textendash{}113. \sphinxurl{https://doi.org/10.1109/ASAP.2008.4580163} +DeConto, R.M., Pollard, D., 2016. Contribution of Antarctica to past and future sea\sphinxhyphen{}level rise. Nature 531, 591\textendash{}597. \sphinxurl{https://doi.org/10.1038/nature17145} +Del Moral, P., Doucet, A., Jasra, A., 2006. Sequential Monte Carlo samplers. J R Stat Soc Ser. B Stat Methodol 68, 411\textendash{}436. \sphinxurl{https://doi.org/10.1111/j.1467-9868.2006.00553.x} +Demarta, S., McNeil, A.J., 2005. The t copula and related copulas. Int. Stat. Rev. 73, 111\textendash{}129. +Doucet, A., Freitas, N., Gordon, N., 2001. An Introduction to Sequential Monte Carlo Methods. Seq. Monte Carlo Methods Pract. 3\textendash{}14. \sphinxurl{https://doi.org/10.1007/978-1-4757-3437-9\_1} +Doucet, A., Godsill, S., Andrieu, C., 2000. On sequential Monte Carlo sampling methods for Bayesian filtering. Stat Comput 10, 197\textendash{}208. \sphinxurl{https://doi.org/10.1023/A:1008935410038} +Eason, J., Cremaschi, S., 2014. Adaptive sequential sampling for surrogate model generation with artificial neural networks. Comput. Chem. Eng. 68, 220\textendash{}232. \sphinxurl{https://doi.org/10.1016/j.compchemeng.2014.05.021} +Edwards, N.R., Cameron, D., Rougier, J., 2011. Precalibrating an intermediate complexity climate model. Clim Dyn 37, 1469\textendash{}1482. \sphinxurl{https://doi.org/10.1007/s00382-010-0921-0} +Eker, S., Rovenskaya, E., Obersteiner, M., Langan, S., 2018. Practice and perspectives in the validation of resource management models. Nat. Commun. 9, 1\textendash{}10. \sphinxurl{https://doi.org/10.1038/s41467-018-07811-9} +El Adlouni, S., Bobée, B., Ouarda, T.B.M.J., 2008. On the tails of extreme event distributions in hydrology. J Hydrol 355, 16\textendash{}33. \sphinxurl{https://doi.org/10.1016/j.jhydrol.2008.02.011} +Ellsberg, D., 1961. Risk, Ambiguity, and the Savage Axioms. Q J Econ 75, 643\textendash{}669. \sphinxurl{https://doi.org/10.2307/1884324} +Elsawah, S., Filatova, T., Jakeman, A.J., Kettner, A.J., Zellner, M.L., Athanasiadis, I.N., Hamilton, S.H., Axtell, R.L., Brown, D.G., Gilligan, J.M., Janssen, M.A., Robinson, D.T., Rozenberg, J., Ullah, I.I.T., Lade, S.J., 2020. Eight grand challenges in socio\sphinxhyphen{}environmental systems modeling. Socio\sphinxhyphen{}Environ. Syst. Model. 2, 16226\textendash{}16226. \sphinxurl{https://doi.org/10.18174/sesmo.2020a16226} +Evensen, G., 1994. Sequential data assimilation with a nonlinear quasi\sphinxhyphen{}geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res. Oceans 99, 10143\textendash{}10162. \sphinxurl{https://doi.org/10.1029/94JC00572} +Ferdous, M.R., Wesselink, A., Brandimarte, L., Di Baldassarre, G., Rahman, M.M., 2019. The levee effect along the Jamuna River in Bangladesh. Water Int 44, 496\textendash{}519. \sphinxurl{https://doi.org/10.1080/02508060.2019.1619048} +Field, C.B., Barros, V.R., Dokken, D.J., Mach, K.J., Mastrandrea, M.D., Bilir, T.E., Chatterjee, M., Ebi, K.L., Estrada, Y.O., Genova, R.C., Girma, B., Kissel, E.S., Levy, A.N., MacCracken, S., Mastrandrea, P.R., White, L.L., 2014. Summary for Policymakers, in: Climate Change 2014: Impacts, Adaptation, and Vulnerability. Part A: Global and Sectoral Aspects. Contribution of Working Group II to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge, United Kingdom, and New York, NY, USA, pp. 1\textendash{}32. +Fisher, R.A., 1960. The design of experiments. Oliver and Boyd, Edinburgh, Scotland, UK. +Flegal, J.M., Haran, M., Jones, G.L., 2008. Markov Chain Monte Carlo: Can We Trust the Third Significant Figure? Stat Sci 23, 250\textendash{}260. \sphinxurl{https://doi.org/10.1214/08-STS257} +Frankignoul, C., Hasselmann, K., 1977. Stochastic climate models, Part II Application to sea\sphinxhyphen{}surface temperature anomalies and thermocline variability. Tell’Us 29, 289\textendash{}305. \sphinxurl{https://doi.org/10.3402/tellusa.v29i4.11362} +Fuller, R.W., Wong, T.E., Keller, K., 2017. Probabilistic inversion of expert assessments to inform projections about Antarctic ice sheet responses. PLoS One 12, e0190115. \sphinxurl{https://doi.org/10.1371/journal.pone.0190115} +Gao, L., Bryan, B.A., Nolan, M., Connor, J.D., Song, X., Zhao, G., 2016. Robust global sensitivity analysis under deep uncertainty via scenario analysis. Environ. Model. Softw. 76, 154\textendash{}166. \sphinxurl{https://doi.org/10.1016/j.envsoft.2015.11.001} +Gelfand, A.E., Hills, S.E., Racine\sphinxhyphen{}Poon, A., Adrian F. M. Smith, 1990. Illustration of Bayesian Inference in Normal Data Models Using Gibbs Sampling. J Am Stat Assoc 85, 972\textendash{}985. \sphinxurl{https://doi.org/10.2307/2289594} +Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B., 2013. Bayesian Data Analysis, Third. ed. CRC Press. +Gelman, A., Hwang, J., Vehtari, A., 2014. Understanding predictive information criteria for Bayesian models. Stat Comput 24, 997\textendash{}1016. \sphinxurl{https://doi.org/10.1007/s11222-013-9416-2} +Gelman, A., Meng, X.\sphinxhyphen{}L., Stern, H., 1996. Posterior predictive assessment of model fitness via realized discrepancies. Stat Sin 6, 733\textendash{}760. +Gelman, A., Rubin, D.B., 1992. Inference from Iterative Simulation Using Multiple Simulations. Stat Sci 7, 457\textendash{}511. \sphinxurl{https://doi.org/10.1214/ss/1177011136} +Gelman, A., Shalizi, C.R., 2013. Philosophy and the practice of Bayesian statistics. Br J Math Stat Psychol 66, 8\textendash{}38. \sphinxurl{https://doi.org/10.1111/j.2044-8317.2011.02037.x} +Gelman, A., Simpson, D., Betancourt, M., 2017. The Prior Can Often Only Be Understood in the Context of the Likelihood. Entropy 19, 555. \sphinxurl{https://doi.org/10.3390/e19100555} +Geman, S., Geman, D., 1984. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans Pattern Anal Mach Intell 6, 721\textendash{}741. \sphinxurl{https://doi.org/10.1109/tpami.1984.4767596} +Geyer, C.J., 2011. Introduction to Markov Chain Monte Carlo, in: Brooks, S., Gelman, A., Jones, G.L., Meng, X.\sphinxhyphen{}L. (Eds.), Handbook of Markov Chain Monte Carlo. Chapman \& Hall/CRC, Boca Raton, FL, pp. 3\textendash{}48. +Geyer, C.J., Johnson, L.T., 2013. mcmc: Markov Chain Monte Carlo. R Package Version 0 9\sphinxhyphen{}2 URL HttpCRAN R\sphinxhyphen{}Proj. Orgpackage Mcmc. +Ghanem, R.G., Spanos, P.D., 1991. Spectral stochastic finite\sphinxhyphen{}element formulation for reliability analysis. J Eng Mech 117, 2351\textendash{}2372. +Goodman, J., Weare, J., 2010. Ensemble samplers with affine invariance. Commun. Appl. Math. Comput. Sci. 5, 65\textendash{}80. \sphinxurl{https://doi.org/10.2140/camcos.2010.5.65} +Gordon, N.J., Salmond, D.J., Smith, A.F.M., 1993. Novel approach to nonlinear/non\sphinxhyphen{}Gaussian Bayesian state estimation. IEE Proc. F Radar Signal Process. 140, 107\textendash{}113. \sphinxurl{https://doi.org/10.1049/ip-f-2.1993.0015} +Gorissen, D., De Tommasi, L., Crombecq, K., Dhaene, T., 2009. Sequential modeling of a low noise amplifier with neural networks and active learning. Neural Comput. Appl. 18, 485\textendash{}494. \sphinxurl{https://doi.org/10.1007/s00521-008-0223-1} +Gupta, H.V., Clark, M.P., Vrugt, J.A., Abramowitz, G., Ye, M., 2012. Towards a comprehensive assessment of model structural adequacy. Water Resour. Res. 48. \sphinxurl{https://doi.org/10.1029/2011WR011044} +Gupta, H.V., Sorooshian, S., Yapo, P.O., 1998. Toward improved calibration of hydrologic models: Multiple and noncommensurable measures of information. Water Resour. Res. 34, 751\textendash{}763. \sphinxurl{https://doi.org/10.1029/97WR03495} +Gupta, H.V., Wagener, T., Liu, Y., 2008. Reconciling theory with observations: elements of a diagnostic approach to model evaluation. Hydrol. Process. Int. J. 22, 3802\textendash{}3813. +Hadjimichael, A., Quinn, J., Reed, P., 2020. Advancing Diagnostic Model Evaluation to Better Understand Water Shortage Mechanisms in Institutionally Complex River Basins. Water Resour. Res. 56, e2020WR028079. \sphinxurl{https://doi.org/10.1029/2020WR028079} +Haer, T., Husby, T.G., Botzen, W.J.W., Aerts, J.C.J.H., 2020. The safe development paradox: An agent\sphinxhyphen{}based model for flood risk under climate change in the European Union. Glob Env. Change 60, 102009. \sphinxurl{https://doi.org/10.1016/j.gloenvcha.2019.102009} +Haimes, Y.Y., 2018. Risk Modeling of Interdependent Complex Systems of Systems: Theory and Practice. Risk Anal. 38, 84\textendash{}98. \sphinxurl{https://doi.org/10.1111/risa.12804} +Hall, J.W., Lempert, R.J., Keller, K., Hackbarth, A., Mijere, C., McInerney, D.J., 2012. Robust climate policies under uncertainty: A comparison of robust decision making and info\sphinxhyphen{}gap methods. Risk Anal. 32, 1657\textendash{}1672. \sphinxurl{https://doi.org/10.1111/j.1539-6924.2012.01802.x} +Hamm, N.A.S., Hall, J.W., Anderson, M.G., 2006. Variance\sphinxhyphen{}based sensitivity analysis of the probability of hydrologically induced slope instability. Comput. Geosci. 32, 803\textendash{}817. \sphinxurl{https://doi.org/10.1016/j.cageo.2005.10.007} +Han, Y., Ash, K., Mao, L., Peng, Z.\sphinxhyphen{}R., 2020. An agent\sphinxhyphen{}based model for community flood adaptation under uncertain sea\sphinxhyphen{}level rise. Clim Change. \sphinxurl{https://doi.org/10.1007/s10584-020-02802-6} +Hastings, W.K., 1970. Monte Carlo Sampling Methods Using Markov Chains and Their Applications. Biometrika 57, 97\textendash{}109. \sphinxurl{https://doi.org/10.2307/2334940} +Helbing, D., 2013. Globally networked risks and how to respond. Nature 497, 51\textendash{}59. \sphinxurl{https://doi.org/10.1038/nature12047} +Helton, J.C., Johnson, J.D., Sallaberry, C.J., Storlie, C.B., 2006. Survey of sampling\sphinxhyphen{}based methods for uncertainty and sensitivity analysis. Reliab. Eng. Syst. Saf., The Fourth International Conference on Sensitivity Analysis of Model Output (SAMO 2004) 91, 1175\textendash{}1209. \sphinxurl{https://doi.org/10.1016/j.ress.2005.11.017} +Herman, J.D., Reed, P.M., Wagener, T., 2013. Time\sphinxhyphen{}varying sensitivity analysis clarifies the effects of watershed model formulation on model behavior. Water Resour. Res. 49, 1400\textendash{}1414. \sphinxurl{https://doi.org/10.1002/wrcr.20124} +Herman, J.D., Zeff, H.B., Lamontagne, J.R., Reed, P.M., Characklis, G.W., 2016. Synthetic Drought Scenario Generation to Support Bottom\sphinxhyphen{}Up Water Supply Vulnerability Assessments. J. Water Resour. Plan. Manag. 142, 04016050. \sphinxurl{https://doi.org/10.1061/(ASCE)WR.1943-5452.0000701} +Herrera, M., Natarajan, S., Coley, D.A., Kershaw, T., Ramallo\sphinxhyphen{}González, A.P., Eames, M., Fosas, D., Wood, M., 2017. A review of current and future weather data for building simulation. Build. Serv. Eng. Res. Technol. 38, 602\textendash{}627. \sphinxurl{https://doi.org/10.1177/0143624417705937} +Hoffman, M.D., Gelman, A., 2014. The No\sphinxhyphen{}U\sphinxhyphen{}Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo. J Mach Learn Res 15, 1593\textendash{}1623. +Houtekamer, P.L., Mitchell, H.L., 1998. Data Assimilation Using an Ensemble Kalman Filter Technique. Mon. Weather Rev. 126, 796\textendash{}811. \sphinxurl{https://doi.org/10.1175/1520-0493(1998})126\textless{}0796:DAUAEK\textgreater{}2.0.CO;2 +Hu, L., Keller, C.A., Long, M.S., Sherwen, T., Auer, B., Da Silva, A., Nielsen, J.E., Pawson, S., Thompson, M.A., Trayanov, A.L., Travis, K.R., Grange, S.K., Evans, M.J., Jacob, D.J., 2018. Global simulation of tropospheric chemistry at 12.5 km resolution: performance and evaluation of the GEOS\sphinxhyphen{}Chem chemical module (v10\sphinxhyphen{}1) within the NASA GEOS Earth system model (GEOS\sphinxhyphen{}5 ESM). Geosci Model Dev 11, 4603\textendash{}4620. \sphinxurl{https://doi.org/10.5194/gmd-11-4603-2018} +Hu, Y.\sphinxhyphen{}M., Hendry, M., Heng, I.S., 2014. Efficient Exploration of Multi\sphinxhyphen{}Modal Posterior Distributions. ArXiv Astro\sphinxhyphen{}PhIM. +Hurrell, J.W., Holland, M.M., Gent, P.R., Ghan, S., Kay, J.E., Kushner, P.J., Lamarque, J.\sphinxhyphen{}F., Large, W.G., Lawrence, D., Lindsay, K., Lipscomb, W.H., Long, M.C., Mahowald, N., Marsh, D.R., Neale, R.B., Rasch, P., Vavrus, S., Vertenstein, M., Bader, D., Collins, W.D., Hack, J.J., Kiehl, J., Marshall, S., 2013. The Community Earth System Model: A Framework for Collaborative Research. Bull Am Meteorol Soc 94, 1339\textendash{}1360. \sphinxurl{https://doi.org/10.1175/BAMS-D-12-00121.1} +Hurvich, C.M., Tsai, C.\sphinxhyphen{}L., 1989. Regression and time series model selection in small samples. Biometrika 76, 297\textendash{}307. +Iooss, B., Boussouf, L., Feuillard, V., Marrel, A., 2010. Numerical studies of the metamodel fitting and validation processes. Int. J. Adv. Syst. Meas. 3, 11\textendash{}21. +Jaynes, E.T., 2003. Probability Theory: The Logic of Science. Cambridge University Press. +Jin, R., Chen, W., Sudjianto, A., 2008. An Efficient Algorithm for Constructing Optimal Design of Computer Experiments. Presented at the ASME 2003 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers Digital Collection, pp. 545\textendash{}554. \sphinxurl{https://doi.org/10.1115/DETC2003/DAC-48760} +Johnson, D.R., Fischbach, J.R., Ortiz, D.S., 2013. Estimating Surge\sphinxhyphen{}Based Flood Risk with the Coastal Louisiana Risk Assessment Model. J Coast Res 109\textendash{}126. \sphinxurl{https://doi.org/10.2112/si\_67\_8} +Kandlikar, M., Risbey, J., Dessai, S., 2005. Representing and communicating deep uncertainty in climate\sphinxhyphen{}change assessments. C R Geosci 337, 443\textendash{}455. +Kasmalkar, I.G., Serafin, K.A., Miao, Y., Bick, I.A., Ortolano, L., Ouyang, D., Suckale, J., 2020. When floods hit the road: Resilience to flood\sphinxhyphen{}related traffic disruption in the San Francisco Bay Area and beyond. Sci Adv 6, eaba2423. \sphinxurl{https://doi.org/10.1126/sciadv.aba2423} +Kass, R.E., Raftery, A.E., 1995. Bayes Factors. J Am Stat Assoc 90, 773\textendash{}795. \sphinxurl{https://doi.org/10.1080/01621459.1995.10476572} +Keller, K., McInerney, D., 2008. The dynamics of learning about a climate threshold. Clim Dyn 30, 321\textendash{}332. \sphinxurl{https://doi.org/10.1007/s00382-007-0290-5} +Kennedy, M.C., O’Hagan, A., 2001. Bayesian calibration of computer models. J R Stat Soc Ser. B Stat Methodol 63, 425\textendash{}464. \sphinxurl{https://doi.org/10.1111/1467-9868.00294} +Kirsch, B.R., Characklis, G.W., Zeff, H.B., 2013. Evaluating the Impact of Alternative Hydro\sphinxhyphen{}Climate Scenarios on Transfer Agreements: Practical Improvement for Generating Synthetic Streamflows. J. Water Resour. Plan. Manag. 139, 396\textendash{}406. \sphinxurl{https://doi.org/10.1061/(ASCE)WR.1943-5452.0000287} +Kleindorfer, G.B., O’Neill, L., Ganeshan, R., 1998. Validation in simulation: Various positions in the philosophy of science. Manag. Sci. 44, 1087\textendash{}1099. +Koutsoyiannis, D., 2004. Statistics of extremes and estimation of extreme rainfall: I. Theoretical investigation/Statistiques de valeurs extrêmes et estimation de précipitations extrêmes: I. Recherche théorique. Hydrol. Sci. J. 49. +Kraan, B.C., Cooke, R.M., 2000. Uncertainty in compartmental models for hazardous materials \sphinxhyphen{} a case study. J Hazard Mater 71, 253\textendash{}268. \sphinxurl{https://doi.org/10.1016/S0304-3894(99)00082-5} +Kucherenko, S., Albrecht, D., Saltelli, A., 2015. Exploring multi\sphinxhyphen{}dimensional spaces: a Comparison of Latin Hypercube and Quasi Monte Carlo Sampling Techniques. ArXiv150502350 Stat. +Kullback, S., Leibler, R.A., 1951. On Information and Sufficiency. Ann Math Stat 22, 79\textendash{}86. \sphinxurl{https://doi.org/10.1214/aoms/1177729694} +Kumar, P., 2011. Typology of hydrologic predictability. Water Resour. Res. 47. \sphinxurl{https://doi.org/10.1029/2010WR009769} +Kwakkel, J.H., Walker, W.E., Haasnoot, M., 2016. Coping with the Wickedness of Public Policy Problems: Approaches for Decision Making under Deep Uncertainty. J. Water Resour. Plan. Manag. 142, 01816001. \sphinxurl{https://doi.org/10.1061/(ASCE)WR.1943-5452.0000626} +Lamontagne, J.R., Reed, P.M., Link, R., Calvin, K.V., Clarke, L.E., Edmonds, J.A., 2018. Large Ensemble Analytic Framework for Consequence\sphinxhyphen{}Driven Discovery of Climate Change Scenarios. Earths Future 6, 488\textendash{}504. \sphinxurl{https://doi.org/10.1002/2017EF000701} +Lamontagne, J.R., Stedinger, J.R., 2018. Generating Synthetic Streamflow Forecasts with Specified Precision. J. Water Resour. Plan. Manag. 144, 04018007. \sphinxurl{https://doi.org/10.1061/(ASCE)WR.1943-5452.0000915} +Landeg, O., Whitman, G., Walker\sphinxhyphen{}Springett, K., Butler, C., Bone, A., Kovats, S., 2019. Coastal flooding and frontline health care services: challenges for flood risk resilience in the English health care system. J Health Serv Res Policy 24, 219\textendash{}228. \sphinxurl{https://doi.org/10.1177/1355819619840672} +Lee, B.S., Haran, M., Fuller, R., Pollard, D., Keller, K., 2020. A Fast Particle\sphinxhyphen{}Based Approach for Calibrating a 3\sphinxhyphen{}D Model of the Antarctic Ice Sheet. Ann. Appl. Stat. in the press. +Lee, B.S., Haran, M., Keller, K., 2017. Multidecadal Scale Detection Time for Potentially Increasing Atlantic Storm Surges in a Warming Climate. Geophys Res Lett 44, 10,617\sphinxhyphen{}10,623. \sphinxurl{https://doi.org/10.1002/2017GL074606} +Lempert, R.J., 2002. A new decision sciences for complex systems. Proc. Natl. Acad. Sci. 99, 7309\textendash{}7313. \sphinxurl{https://doi.org/10.1073/pnas.082081699} +Liu, J., West, M., 2001. Combined Parameter and State Estimation in Simulation\sphinxhyphen{}Based Filtering, in: Doucet, A., de Freitas, N., Gordon, N. (Eds.), Sequential Monte Carlo Methods in Practice. Springer New York, New York, NY, pp. 197\textendash{}223. \sphinxurl{https://doi.org/10.1007/978-1-4757-3437-9\_10} +Liu, X., Guillas, S., 2017. Dimension Reduction for Gaussian Process Emulation: An Application to the Influence of Bathymetry on Tsunami Heights. SIAMASA J Uncertain. Quantif. 5, 787\textendash{}812. \sphinxurl{https://doi.org/10.1137/16M1090648} +Lorenz, E.N., 1963. Deterministic Nonperiodic Flow. J Atmos Sci 20, 130\textendash{}141. \sphinxurl{https://doi.org/10.1175/1520-0469(1963})020\textless{}0130:DNF\textgreater{}2.0.CO;2 +Loucks, D.P., Beek, E. van, 2017. Water Resource Systems Planning and Management: An Introduction to Methods, Models, and Applications. Springer International Publishing. +Lukacs, P.M., Burnham, K.P., Anderson, D.R., 2010. Model selection bias and Freedman’s paradox. Ann Inst Stat Math 62, 117\textendash{}125. \sphinxurl{https://doi.org/10.1007/s10463-009-0234-4} +Makowski, D., Wallach, D., Tremblay, M., 2002. Using a Bayesian approach to parameter estimation; comparison of the GLUE and MCMC methods. Agronomie 22, 191\textendash{}203. +Maniyar, D.M., Cornford, D., Boukouvalas, A., 2007. Dimensionality Reduction in the Emulator Setting (No. NCRG/2007/005). Neural Computing Research Group. +Marchau, V.A.W.J., Walker, W.E., Bloemen, P.J.T.M., Popper, S.W. (Eds.), 2019. Decision Making under Deep Uncertainty: From Theory to Practice. Springer International Publishing. \sphinxurl{https://doi.org/10.1007/978-3-030-05252-2} +Martin, A.D., 2010. MCMCpack: Markov chain Monte Carlo (MCMC) Package. R package version 1.0\sphinxhyphen{}7. HttpCRAN R\sphinxhyphen{}Proj. Orgpackage MCMCpack. +Massmann, C., Wagener, T., Holzmann, H., 2014. A new approach to visualizing time\sphinxhyphen{}varying sensitivity indices for environmental model diagnostics across evaluation time\sphinxhyphen{}scales. Environ. Model. Softw. 51, 190\textendash{}194. \sphinxurl{https://doi.org/10.1016/j.envsoft.2013.09.033} +Matthews, P., 1993. A slowly mixing Markov chain with implications for Gibbs sampling. Stat Probab Lett 17, 231\textendash{}236. \sphinxurl{https://doi.org/10.1016/0167-7152(93)90172-F} +McInerney, D., Keller, K., 2008. Economically optimal risk reduction strategies in the face of uncertain climate thresholds. Clim Change 91, 29\textendash{}41. \sphinxurl{https://doi.org/10.1007/s10584-006-9137-z} +McKay, M.D., Beckman, R.J., Conover, W.J., 1979. A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code. Technometrics 21, 239\textendash{}245. \sphinxurl{https://doi.org/10.2307/1268522} +Medda, S., Bhar, K.K., 2019. Comparison of single\sphinxhyphen{}site and multi\sphinxhyphen{}site stochastic models for streamflow generation. Appl. Water Sci. 9, 67. +Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E., 1953. Equation of State Calculations by Fast Computing Machines. J Chem Phys 21, 1087\textendash{}1092. \sphinxurl{https://doi.org/10.1063/1.1699114} +Metropolis, N., Ulam, S., 1949. The Monte Carlo Method. J. Am. Stat. Assoc. 44, 335\textendash{}341. \sphinxurl{https://doi.org/10.1080/01621459.1949.10483310} +Milly, P.C.D., Betancourt, J., Falkenmark, M., Hirsch, R.M., Kundzewicz, Z.W., Lettenmaier, D.P., Stouffer, R.J., 2008. Stationarity is dead: Whither water management? Science 319, 573\textendash{}574. \sphinxurl{https://doi.org/10.1126/science.1151915} +Moallemi, E.A., Kwakkel, J., de Haan, F.J., Bryan, B.A., 2020a. Exploratory modeling for analyzing coupled human\sphinxhyphen{}natural systems under uncertainty. Glob. Environ. Change 65, 102186. \sphinxurl{https://doi.org/10.1016/j.gloenvcha.2020.102186} +Moallemi, E.A., Zare, F., Reed, P.M., Elsawah, S., Ryan, M.J., Bryan, B.A., 2020b. Structuring and evaluating decision support processes to enhance the robustness of complex human\textendash{}natural systems. Environ. Model. Softw. 123, 104551. \sphinxurl{https://doi.org/10.1016/j.envsoft.2019.104551} +Montgomery, D.C., 2017. Design and analysis of experiments. John Wiley \& Sons. +Morgan, M.G., Keith, D.W., 2008. Improving the way we think about projecting future energy use and emissions of carbon dioxide. Clim Change 90, 189\textendash{}215. \sphinxurl{https://doi.org/10.1007/s10584-008-9458-1} +Morris, M.D., 1991. Factorial Sampling Plans for Preliminary Computational Experiments. Technometrics 33, 161\textendash{}174. \sphinxurl{https://doi.org/10.1080/00401706.1991.10484804} +Morris, M.D., Mitchell, T.J., 1995. Exploratory designs for computational experiments. J. Stat. Plan. Inference 43, 381\textendash{}402. \sphinxurl{https://doi.org/10.1016/0378-3758(94)00035-T} +National Research Council, 2014. Convergence: facilitating transdisciplinary integration of life sciences, physical sciences, engineering, and beyond. National Academies Press, Washington, D.C., USA. +Naylor, T.H., Finger, J.M., 1967. Verification of computer simulation models. Manag. Sci. 14, B\sphinxhyphen{}92. +Neal, R.M., 2011. MCMC Using Hamiltonian Dynamics, in: Brooks, S., Gelman, A., Jones, G.L., Meng, X.\sphinxhyphen{}L. (Eds.), Handbook of Markov Chain Monte Carlo, Handbooks of Modern Statistical Methods. CRC Press, Boca Raton, FL, pp. 113\textendash{}162. +Nearing, G.S., Ruddell, B.L., Bennett, A.R., Prieto, C., Gupta, H.V., 2020. Does Information Theory Provide a New Paradigm for Earth Science? Hypothesis Testing. Water Resour. Res. 56, e2019WR024918. \sphinxurl{https://doi.org/10.1029/2019WR024918} +Nelder, J.A., Wedderburn, R.W.M., 1972. Generalized Linear Models. J R Stat Soc Ser A 135, 370\textendash{}384. \sphinxurl{https://doi.org/10.2307/2344614} +Norton, J., 2015. An introduction to sensitivity assessment of simulation models. Environ. Model. Softw. 69, 166\textendash{}174. \sphinxurl{https://doi.org/10.1016/j.envsoft.2015.03.020} +Oddo, P.C., Lee, B.S., Garner, G.G., Srikrishnan, V., Reed, P.M., Forest, C.E., Keller, K., 2017. Deep Uncertainties in Sea\sphinxhyphen{}Level Rise and Storm Surge Projections: Implications for Coastal Flood Risk Management. Risk Anal. \sphinxurl{https://doi.org/10.1111/risa.12888} +O’Neill, B.C., Crutzen, P., Grübler, A., Duong, M.H., Keller, K., Kolstad, C., Koomey, J., Lange, A., Obersteiner, M., Oppenheimer, M., Pepper, W., Sanderson, W., Schlesinger, M., Treich, N., Ulph, A., Webster, M., Wilson, C., 2006. Learning and climate change. Clim Policy 6, 585\textendash{}589. \sphinxurl{https://doi.org/10.1080/14693062.2006.9685623} +Oppenheimer, M., O’Neill, B.C., Webster, M., 2008. Negative learning. Clim Change 89, 155\textendash{}172. \sphinxurl{https://doi.org/10.1007/s10584-008-9405-1} +Oreskes, N., Shrader\sphinxhyphen{}Frechette, K., Belitz, K., 1994. Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences. Science 263, 641\textendash{}646. \sphinxurl{https://doi.org/10.1126/science.263.5147.641} +Park, J.\sphinxhyphen{}S., 1994. Optimal Latin\sphinxhyphen{}hypercube designs for computer experiments. J. Stat. Plan. Inference 39, 95\textendash{}111. \sphinxurl{https://doi.org/10.1016/0378-3758(94)90115-5} +Pianosi, F., Beven, K., Freer, J., Hall, J.W., Rougier, J., Stephenson, D.B., Wagener, T., 2016. Sensitivity analysis of environmental models: A systematic review with practical workflow. Environ. Model. Softw. 79, 214\textendash{}232. \sphinxurl{https://doi.org/10.1016/j.envsoft.2016.02.008} +Pollard, D., DeConto, R.M., 2012. Description of a hybrid ice sheet\sphinxhyphen{}shelf model, and application to Antarctica. Geosci. Model Dev. 5, 1273\textendash{}1295. \sphinxurl{https://doi.org/10.5194/gmd-5-1273-2012} +Pruett, W.A., Hester, R.L., 2016. The Creation of Surrogate Models for Fast Estimation of Complex Model Outcomes. PLOS ONE 11, e0156574. \sphinxurl{https://doi.org/10.1371/journal.pone.0156574} +Ragulina, G., Reitan, T., 2017. Generalized extreme value shape parameter and its nature for extreme precipitation using long time series and the Bayesian approach. Hydrol. Sci. J. 62, 863\textendash{}879. +Rakovec, O., Hill, M.C., Clark, M.P., Weerts, A.H., Teuling, A.J., Uijlenhoet, R., 2014. Distributed Evaluation of Local Sensitivity Analysis (DELSA), with application to hydrologic models. Water Resour. Res. 50, 409\textendash{}426. \sphinxurl{https://doi.org/10.1002/2013WR014063} +Rasmussen, D.J., Buchanan, M.K., Kopp, R.E., Oppenheimer, M., 2020. A flood damage allowance framework for coastal protection with deep uncertainty in sea‐level rise. Earths Future. \sphinxurl{https://doi.org/10.1029/2019EF001340} +Robert, C.P., 2007. The Bayesian choice: from decision\sphinxhyphen{}theoretic foundations to computational implementation, 2nd ed. Springer, New York. +Rohmer, J., Le Cozannet, G., Manceau, J.\sphinxhyphen{}C., 2019. Addressing ambiguity in probabilistic assessments of future coastal flooding using possibility distributions. Clim Change. \sphinxurl{https://doi.org/10.1007/s10584-019-02443-4} +Ruckert, K.L., Shaffer, G., Pollard, D., Guan, Y., Wong, T.E., Forest, C.E., Keller, K., 2017. Assessing the Impact of Retreat Mechanisms in a Simple Antarctic Ice Sheet Model Using Bayesian Calibration. PLoS One 12, e0170052. \sphinxurl{https://doi.org/10.1371/journal.pone.0170052} +Ruckert, K.L., Srikrishnan, V., Keller, K., 2019. Characterizing the deep uncertainties surrounding coastal flood hazard projections: A case study for Norfolk, VA. Sci Rep 9, 11373. \sphinxurl{https://doi.org/10.1038/s41598-019-47587-6} +Sacks, J., Welch, W.J., Mitchell, T.J., Wynn, H.P., 1989. Design and Analysis of Computer Experiments. Stat Sci 4, 409\textendash{}423. +Saltelli, A., 2002. Making best use of model evaluations to compute sensitivity indices. Comput. Phys. Commun. 145, 280\textendash{}297. +Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., Wu, Q., 2019. Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices. Environ. Model. Softw. 114, 29\textendash{}39. \sphinxurl{https://doi.org/10.1016/j.envsoft.2019.01.012} +Saltelli, A., Annoni, P., 2010. How to avoid a perfunctory sensitivity analysis. Environ. Model. Softw. 25, 1508\textendash{}1517. \sphinxurl{https://doi.org/10.1016/j.envsoft.2010.04.012} +Saltelli, A., Funtowicz, S., 2014. When all models are wrong. Issues Sci. Technol. 30, 79\textendash{}85. +Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S., 2008. Global Sensitivity Analysis: The Primer, 1 edition. ed. Wiley\sphinxhyphen{}Interscience, Chichester, England; Hoboken, NJ. +Saltelli, A., Stark, P.B., Becker, W., Stano, P., 2015. Climate models as economic guides scientific challenge or quixotic quest? Issues Sci. Technol. 31, 79\textendash{}84. +Saltelli, A., Tarantola, S., 2002. On the relative importance of input factors in mathematical models: safety assessment for nuclear waste disposal. J. Am. Stat. Assoc. 97, 702\textendash{}709. +Saltelli, A., Tarantola, S., Campolongo, F., Ratto, M., 2004. Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. John Wiley \& Sons. +Saltelli, A., Tarantola, S., Chan, K.P.\sphinxhyphen{}S., 1999. A Quantitative Model\sphinxhyphen{}Independent Method for Global Sensitivity Analysis of Model Output. Technometrics 41, 39\textendash{}56. \sphinxurl{https://doi.org/10.1080/00401706.1999.10485594} +Schlather, M., 2002. Models for stationary max\sphinxhyphen{}stable random fields. Extremes 5, 33\textendash{}44. +Schwarz, G., 1978. Estimating the Dimension of a Model. Ann Stat 6, 461\textendash{}464. \sphinxurl{https://doi.org/10.1214/aos/1176344136} +Shafii, M., Tolson, B., Matott, L.S., 2014. Uncertainty\sphinxhyphen{}based multi\sphinxhyphen{}criteria calibration of rainfall\sphinxhyphen{}runoff models: a comparative study. Stoch Env. Res Risk Assess 28, 1493\textendash{}1510. \sphinxurl{https://doi.org/10.1007/s00477-014-0855-x} +Shibata, R., 1989. Statistical Aspects of Model Selection, in: Willems, J.C. (Ed.), From Data to Model. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 215\textendash{}240. \sphinxurl{https://doi.org/10.1007/978-3-642-75007-6\_5} +Sibley Heather M., Vroman Noah D., Shewbridge Scott E., 2017. Quantitative Risk\sphinxhyphen{}Informed Design of Levees. Geo\sphinxhyphen{}Risk 2017 76\textendash{}90. \sphinxurl{https://doi.org/10.1061/9780784480717.008} +Sminchisescu, C., Welling, M., Hinton, G., 2003. A mode\sphinxhyphen{}hopping MCMC sampler. Technical Report CSRG\sphinxhyphen{}478, University of Toronto, submitted to Machine …. +Sobol, I.M., 2001. Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math. Comput. Simul. 55, 271\textendash{}280. \sphinxurl{https://doi.org/10.1016/S0378-4754(00)00270-6} +Sobol, I.M., 1976. Uniformly distributed sequences with an additional uniform property. USSR Comput. Math. Math. Phys. 16, 236\textendash{}242. +Sobol’, I.M., 1967. On the distribution of points in a cube and the approximate evaluation of integrals. USSR Comput. Math. Math. Phys. 7, 86\textendash{}112. \sphinxurl{https://doi.org/10.1016/0041-5553(67)90144-9} +Spear, R.C., Hornberger, G.M., 1980. Eutrophication in peel inlet—II. Identification of critical uncertainties via generalized sensitivity analysis. Water Res. 14, 43\textendash{}49. \sphinxurl{https://doi.org/10.1016/0043-1354(80)90040-8} +Spiegelhalter, D.J., Best, N.G., Carlin, B.P., Van Der Linde, A., 2002. Bayesian measures of model complexity and fit. J R Stat Soc Ser. B Stat Methodol 64, 583\textendash{}639. \sphinxurl{https://doi.org/10.1111/1467-9868.00353} +Srikrishnan, V, Alley, R.B., Keller, K., 2019. Investing in Science and Using the Results to Improve Climate Risk Management. EOS. +Srikrishnan, Vivek, Guan, Y., Tol, R.S.J., Keller, K., 2019. Fossil fuel resources, decarbonization, and economic growth drive the feasibility of Paris climate targets. ArXiv StatAP. +Sriver, R.L., Lempert, R.J., Wikman\sphinxhyphen{}Svahn, P., Keller, K., 2018. Characterizing uncertain sea\sphinxhyphen{}level rise projections to support investment decisions. PLoS One 13, e0190641. \sphinxurl{https://doi.org/10.1371/journal.pone.0190641} +Stedinger, J.R., Vogel, R.M., Lee, S.U., Batchelder, R., 2008. Appraisal of the generalized likelihood uncertainty estimation (GLUE) method: APPRAISAL OF THE GLUE METHOD. Water Resour Res 44, 191. \sphinxurl{https://doi.org/10.1029/2008WR006822} +Steinschneider, S., Wi, S., Brown, C., 2015. The integrated effects of climate and hydrologic uncertainty on future flood risk assessments. Hydrol. Process. 29, 2823\textendash{}2839. \sphinxurl{https://doi.org/10.1002/hyp.10409} +Stone, M., 1977. An Asymptotic Equivalence of Choice of Model by Cross\sphinxhyphen{}Validation and Akaike’s Criterion. J R Stat Soc Ser. B Stat Methodol 39, 44\textendash{}47. +Surowiec, I., Vikström, L., Hector, G., Johansson, E., Vikström, C., Trygg, J., 2017. Generalized Subset Designs in Analytical Chemistry. Anal. Chem. 89, 6491\textendash{}6497. \sphinxurl{https://doi.org/10.1021/acs.analchem.7b00506} +Tang, B., 1993. Orthogonal Array\sphinxhyphen{}Based Latin Hypercubes. J. Am. Stat. Assoc. 88, 1392\textendash{}1397. \sphinxurl{https://doi.org/10.1080/01621459.1993.10476423} +Tang, Y., Reed, P., Wagener, T., Werkhoven, K. van, 2007. Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation. Hydrol. Earth Syst. Sci. 11, 793\textendash{}817. \sphinxurl{https://doi.org/10.5194/hess-11-793-2007} +Toulmin, S., 1977. From form to function: philosophy and history of science in the 1950s and now. Daedalus 143\textendash{}162. +Urban, N.M., Keller, K., 2010. Probabilistic hindcasts and projections of the coupled climate, carbon cycle and Atlantic meridional overturning circulation system: A Bayesian fusion of century\sphinxhyphen{}scale observations with a simple model. Tellus A 62, 737\textendash{}750. +Van Oort, N., 2011. Service reliability and urban public transport design. +Van Schepdael, A., Carlier, A., Geris, L., 2016. Sensitivity Analysis by Design of Experiments, in: Geris, L., Gomez\sphinxhyphen{}Cabrero, D. (Eds.), Uncertainty in Biology: A Computational Modeling Approach, Studies in Mechanobiology, Tissue Engineering and Biomaterials. Springer International Publishing, Cham, pp. 327\textendash{}366. \sphinxurl{https://doi.org/10.1007/978-3-319-21296-8\_13} +van Vuuren, D.P., Edmonds, J., Kainuma, M., Riahi, K., Thomson, A., Hibbard, K., Hurtt, G.C., Kram, T., Krey, V., Lamarque, J.\sphinxhyphen{}F., Masui, T., Meinshausen, M., Nakicenovic, N., Smith, S.J., Rose, S.K., 2011. The representative concentration pathways: an overview. Clim Change 109, 5. \sphinxurl{https://doi.org/10.1007/s10584-011-0148-z} +Vega‐Westhoff, B., Sriver, R.L., Hartin, C.A., Wong, T.E., Keller, K., 2019. Impacts of Observational Constraints Related to Sea Level on Estimates of Climate Sensitivity. Earths Future. \sphinxurl{https://doi.org/10.1029/2018EF001082} +Vehtari, A., Gelman, A., Gabry, J., 2017. Practical Bayesian model evaluation using leave\sphinxhyphen{}one\sphinxhyphen{}out cross\sphinxhyphen{}validation and WAIC. Stat Comput 27, 1413\textendash{}1432. +Vihola, M., 2012. Robust adaptive Metropolis algorithm with coerced acceptance rate. Stat Comput 22, 997\textendash{}1008. \sphinxurl{https://doi.org/10.1007/s11222-011-9269-5} +Vogel, R.M., 2017. Stochastic watershed models for hydrologic risk management. Water Secur. 1, 28\textendash{}35. \sphinxurl{https://doi.org/10.1016/j.wasec.2017.06.001} +Vogel, R.M., Stedinger, J.R., 1988. The value of stochastic streamflow models in overyear reservoir design applications. Water Resour. Res. 24, 1483\textendash{}1490. \sphinxurl{https://doi.org/10.1029/WR024i009p01483} +Vousdoukas, M.I., Bouziotas, D., Giardino, A., Bouwer, L.M., Mentaschi, L., Voukouvalas, E., Feyen, L., 2018. Understanding epistemic uncertainty in large\sphinxhyphen{}scale coastal flood risk assessment for present and future climates. Nat Hazards Earth Syst Sci 18, 2127\textendash{}2142. \sphinxurl{https://doi.org/10.5194/nhess-18-2127-2018} +Vrugt, J.A., Beven, K.J., 2018. Embracing equifinality with efficiency: Limits of Acceptability sampling using the DREAM(LOA) algorithm. J Hydrol 559, 954\textendash{}971. \sphinxurl{https://doi.org/10.1016/j.jhydrol.2018.02.026} +Walker, W., Haasnoot, M., Kwakkel, J., Walker, W.E., Haasnoot, M., Kwakkel, J.H., 2013. Adapt or Perish: A Review of Planning Approaches for Adaptation under Deep Uncertainty. Sustainability 5, 955\textendash{}979. \sphinxurl{https://doi.org/10.3390/su5030955} +Walker, W.E., Harremoës, P., Rotmans, J., Sluijs, J.P. van der, Asselt, M.B.A. van, Janssen, P., Krauss, M.P.K. von, 2003. Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model\sphinxhyphen{}Based Decision Support. Integr. Assess. 4, 5\textendash{}17. \sphinxurl{https://doi.org/10.1076/iaij.4.1.5.16466} +Walker, W.E., Lempert, R.J., Kwakkel, J.H., 2013. Deep Uncertainty, in: Gass, S.I., Fu, M.C. (Eds.), Encyclopedia of Operations Research and Management Science. Springer US, pp. 395\textendash{}402. \sphinxurl{https://doi.org/10.1007/978-1-4419-1153-7\_1140} +Watanabe, S., 2010. Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory. J Mach Learn Res 11, 3571\textendash{}3594. +Weaver, C.P., Lempert, R.J., Brown, C., Hall, J.A., Revell, D., Sarewitz, D., 2013. Improving the contribution of climate model information to decision making: the value and demands of robust decision frameworks. Wiley Interdiscip. Rev. Clim. Change 4, 39\textendash{}60. \sphinxurl{https://doi.org/10.1002/wcc.202} +White, D.D., Wutich, A., Larson, K.L., Gober, P., Lant, T., Senneville, C., 2010. Credibility, salience, and legitimacy of boundary objects: water managers’ assessment of a simulation model in an immersive decision theater. Sci. Public Policy 37, 219\textendash{}232. \sphinxurl{https://doi.org/10.3152/030234210X497726} +Wilks, D.S., Wilby, R.L., 1999. The weather generation game: a review of stochastic weather models. Prog. Phys. Geogr. 23, 329\textendash{}357. +Wirtz, D., Nowak, W., 2017. The rocky road to extended simulation frameworks covering uncertainty, inversion, optimization and control. Environ. Model. Softw. 93, 180\textendash{}192. \sphinxurl{https://doi.org/10.1016/j.envsoft.2016.10.003} +Wolff, C., Nikoletopoulos, T., Hinkel, J., Vafeidis, A.T., 2020. Future urban development exacerbates coastal exposure in the Mediterranean. Sci Rep 10, 14420. \sphinxurl{https://doi.org/10.1038/s41598-020-70928-9} +Wong, T.E., Keller, K., 2017. Deep Uncertainty Surrounding Coastal Flood Risk Projections: A Case Study for New Orleans. Earths Future 5, 1015\textendash{}1026. \sphinxurl{https://doi.org/10.1002/2017EF000607} +Wong, T.E., Klufas, A., Srikrishnan, V., Keller, K., 2018. Neglecting model structural uncertainty underestimates upper tails of flood hazard. Env. Res Lett 13, 074019. \sphinxurl{https://doi.org/10.1088/1748-9326/aacb3d} +Xian, S., Lin, N., Kunreuther, H., 2017. Optimal house elevation for reducing flood\sphinxhyphen{}related losses. J Hydrol 548, 63\textendash{}74. \sphinxurl{https://doi.org/10.1016/j.jhydrol.2017.02.057} +Xiu, D., Karniadakis, G.E., 2002. The Wiener\textendash{}Askey Polynomial Chaos for Stochastic Differential Equations. SIAM J. Sci. Comput. 24, 619\textendash{}644. \sphinxurl{https://doi.org/10.1137/S1064827501387826} +Zarekarizi, M., Srikrishnan, V., Keller, K., 2020. Neglecting Uncertainties Biases House\sphinxhyphen{}Elevation Decisions to Manage Riverine Flood Risks. Nat Commun. \sphinxurl{https://doi.org/10.1038/s41467-020-19188-9} +Zaremba, S.K., 1968. The mathematical basis of Monte Carlo and quasi\sphinxhyphen{}Monte Carlo methods. SIAM Rev. 10, 303\textendash{}314. +Zellner, A., Tian, G.C., 1964. Bayesian analysis of the regression model with autocorrelated errors. J Am Stat Assoc 763\textendash{}778. +Zhang, X., Che, L., Shahidehpour, M., Alabdulwahab, A.S., Abusorrah, A., 2015. Reliability\sphinxhyphen{}based optimal planning of electricity and natural gas interconnections for multiple energy hubs. IEEE Trans Smart Grid 8, 1658\textendash{}1667. +Zscheischler, J., Martius, O., Westra, S., Bevacqua, E., Raymond, C., Horton, R.M., van den Hurk, B., AghaKouchak, A., Jézéquel, A., Mahecha, M.D., Maraun, D., Ramos, A.M., Ridder, N.N., Thiery, W., Vignotto, E., 2020. A typology of compound weather and climate events. Nat. Rev. Earth Environ. 1\textendash{}15. \sphinxurl{https://doi.org/10.1038/s43017-020-0060-z} + + +\chapter{Glossary} +\label{\detokenize{9_glossary:glossary}}\label{\detokenize{9_glossary::doc}} + \chapter{Indices and tables} \label{\detokenize{index:indices-and-tables}}\begin{itemize} \item {} diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.toc b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.toc index a45d5065..217f8c17 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.toc +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.toc @@ -25,4 +25,35 @@ \contentsline {subsection}{\numberline {3.6.7}Moment\sphinxhyphen {}Independent (Density\sphinxhyphen {}Based) Methods}{17}{subsection.3.6.7}% \contentsline {section}{\numberline {3.7}How To Choose A Sensitivity Analysis Method: Model Traits And Dimensionality}{17}{section.3.7}% \contentsline {section}{\numberline {3.8}Software Toolkits}{18}{section.3.8}% -\contentsline {chapter}{\numberline {4}Indices and tables}{19}{chapter.4}% +\contentsline {chapter}{\numberline {4}Sensitivity Analysis: Diagnostic \& Exploratory Modeling}{19}{chapter.4}% +\contentsline {section}{\numberline {4.1}Understanding Errors: What Is Controlling Model Performance?}{19}{section.4.1}% +\contentsline {section}{\numberline {4.2}Consequential Dynamics: What is Controlling Model Behaviors of Interest?}{19}{section.4.2}% +\contentsline {section}{\numberline {4.3}Consequential Scenarios: What is Controlling Consequential Outcomes?}{19}{section.4.3}% +\contentsline {chapter}{\numberline {5}Uncertainty Quantification: The Basics}{21}{chapter.5}% +\contentsline {section}{\numberline {5.1}Why is Uncertainty Quantification Important for Understanding MultiSector System Dynamics?}{21}{section.5.1}% +\contentsline {section}{\numberline {5.2}Uncertainty Quantification for Exploratory Modeling}{21}{section.5.2}% +\contentsline {section}{\numberline {5.3}Bayesian Uncertainty Quantification}{21}{section.5.3}% +\contentsline {section}{\numberline {5.4}Uncertainty Quantification Under (Deep) Uncertainty}{21}{section.5.4}% +\contentsline {section}{\numberline {5.5}Integrating Model Diagnostics and Uncertainty Quantification}{21}{section.5.5}% +\contentsline {chapter}{\numberline {6}Uncertainty Quantification: A Tool For Capturing Risks \& Extremes}{23}{chapter.6}% +\contentsline {section}{\numberline {6.1}Understanding Risk: How Probable Are Extreme Events?}{23}{section.6.1}% +\contentsline {section}{\numberline {6.2}Understanding Tails: Statistical Modeling of Extreme Events}{23}{section.6.2}% +\contentsline {section}{\numberline {6.3}How to Choose an Appropriate Method?}{23}{section.6.3}% +\contentsline {section}{\numberline {6.4}How to Select a Prior Distribution?}{23}{section.6.4}% +\contentsline {section}{\numberline {6.5}Posterior Predictive Checking}{23}{section.6.5}% +\contentsline {section}{\numberline {6.6}Model Selection and Comparison}{23}{section.6.6}% +\contentsline {section}{\numberline {6.7}What are Common Methods?}{23}{section.6.7}% +\contentsline {subsection}{\numberline {6.7.1}Scenario Discovery}{24}{subsection.6.7.1}% +\contentsline {subsection}{\numberline {6.7.2}Pre\sphinxhyphen {}calibration/GLUE}{24}{subsection.6.7.2}% +\contentsline {subsection}{\numberline {6.7.3}Markov Chain Monte Carlo}{24}{subsection.6.7.3}% +\contentsline {subsubsection}{\numberline {6.7.3.1}Metropolis\sphinxhyphen {}Hastings}{24}{subsubsection.6.7.3.1}% +\contentsline {subsubsection}{\numberline {6.7.3.2}Gibbs Sampling}{24}{subsubsection.6.7.3.2}% +\contentsline {subsubsection}{\numberline {6.7.3.3}Hamiltonian Monte Carlo}{24}{subsubsection.6.7.3.3}% +\contentsline {subsection}{\numberline {6.7.4}Particle\sphinxhyphen {}based Methods}{24}{subsection.6.7.4}% +\contentsline {section}{\numberline {6.8}What are Example Software Implementations?}{24}{section.6.8}% +\contentsline {subsection}{\numberline {6.8.1}Markov Chain Monte Carlo with the True Model}{25}{subsection.6.8.1}% +\contentsline {subsection}{\numberline {6.8.2}Markov Chain Monte Carlo with Surrogate Models}{25}{subsection.6.8.2}% +\contentsline {chapter}{\numberline {7}Conclusion}{27}{chapter.7}% +\contentsline {chapter}{\numberline {8}References}{29}{chapter.8}% +\contentsline {chapter}{\numberline {9}Glossary}{37}{chapter.9}% +\contentsline {chapter}{\numberline {10}Indices and tables}{39}{chapter.10}% diff --git a/docs/source/6.7.3_markov_chain_monte_carlo.rst b/docs/source/6.7.3_markov_chain_monte_carlo.rst index 1fa45cb0..97349aab 100644 --- a/docs/source/6.7.3_markov_chain_monte_carlo.rst +++ b/docs/source/6.7.3_markov_chain_monte_carlo.rst @@ -11,8 +11,8 @@ Another key value is the effective sample size (ESS). Due to the Markovian prope Many MCMC algorithms exist, with varying strengths and weaknesses, discussed below. For example, some require more tuning to improve the ESS than others. All of these algorithms involve the evaluation of the model at various parameter settings. Once a Markov chain is constructed and deemed to suitably represent the posterior distribution, parameter values can be sampled from it with replacement as a proxy for directly sampling from the posterior. -..include:: 6.7.3.1_metropolis_hastings.rst +.. include:: 6.7.3.1_metropolis_hastings.rst -..include:: 6.7.3.2_gibbs_sampling.rst +.. include:: 6.7.3.2_gibbs_sampling.rst -..include:: 6.7.3.3_hamiltonian_monte_carlo.rst +.. include:: 6.7.3.3_hamiltonian_monte_carlo.rst diff --git a/docs/source/6.7_what_are_common_methods.rst b/docs/source/6.7_what_are_common_methods.rst index a730827d..38c9a466 100644 --- a/docs/source/6.7_what_are_common_methods.rst +++ b/docs/source/6.7_what_are_common_methods.rst @@ -1,2 +1,14 @@ What are Common Methods? ######################## + +There are many methods to quantify uncertainty. Each method has advantages and disadvantages for a particular analysis. Here we focus on parametric uncertainty quantification, as a discussion of structural uncertainty quantification is beyond the scope of this review. Moreover, we prefer to think about structural uncertainty from the perspective of exploratory modeling and deep uncertainty, rather than from the perspective of quantification and selection or averaging. + +Uncertainty quantification methods can be broadly classified as Markov Chain Monte Carlo (MCMC) approaches, particle-based approaches, and emulation-based approaches, though there are some hybrid methods. Several of the most common approaches for uncertainty quantification are described below. In all cases, the computational and conceptual challenges associated with parametric uncertainty quantification grow rapidly with the number of model parameters. As noted in the prior sections, sensitivity analyses are useful for dimensionality reduction prior to conducting parametric uncertainty quantification. Both factor fixing and factor prioritization can be used to limit the number of parameters which are treated as uncertain. + +.. include:: 6.7.1_scenario_discovery.rst + +.. include:: 6.7.2_pre_calibration_glue.rst + +.. include:: 6.7.3_markov_chain_monte_carlo.rst + +.. include:: 6.7.4_particle_based_methods.rst diff --git a/docs/source/6.8_what_are_example_software_implementations.rst b/docs/source/6.8_what_are_example_software_implementations.rst index daf591db..1af5c460 100644 --- a/docs/source/6.8_what_are_example_software_implementations.rst +++ b/docs/source/6.8_what_are_example_software_implementations.rst @@ -1,2 +1,10 @@ What are Example Software Implementations? ########################################## + +There exist many software platforms to implement uncertainty assessment. Each implementation is built upon a specific programming language including, but not limited to R, Python, C++, Fortran, MATLAB, and Julia. Two key considerations are the user’s preferred programming language and the computer model’s native code. For instance, a computer model running in C++ may be better suited for a software implementation based on the same language. For inconsistencies, please see the discussion on wrappers below. + +Here, we present an overview of popular packages inherent to R, Python, and Julia. The user is free to code the UQ implementation without incorporating these existing packages; however, it may require more effort to code the pertinent subroutines (e.g., MCMC and building surrogate models). Uncertainty quantification for computer models typically operates within the Bayesian framework (see What are Common Methods?). Each implementation includes a mechanism that enables Bayesian inference using MCMC, Gaussian process emulation, or Sequential Monte Carlo. We focus on a subset of the common approaches. + +.. include:: 6.8.1_markov_chain_monte_carlo_with_the_true_model.rst + +.. include:: 6.8.2_markov_chain_monte_carlo_with_surrogate_models.rst diff --git a/docs/source/8_references.rst b/docs/source/8_references.rst index 6dea0355..9a5f3f75 100644 --- a/docs/source/8_references.rst +++ b/docs/source/8_references.rst @@ -2,10 +2,7 @@ References ********** -.. target-notes:: - - .. _Akaike, H., 1978. On the Likelihood of a Time Series Model. J. R. Stat. Soc. Ser. Stat. 27, 217–235. https://doi.org/10.2307/2988185 - +Akaike, H., 1978. On the Likelihood of a Time Series Model. J. R. Stat. Soc. Ser. Stat. 27, 217–235. https://doi.org/10.2307/2988185 Akaike, H., 1974. A new look at the statistical model identification. IEEE Trans. Autom. Control 19, 716–723. https://doi.org/10.1109/TAC.1974.1100705 Anderson, B., Borgonovo, E., Galeotti, M., Roson, R., 2014. Uncertainty in Climate Change Modeling: Can Global Sensitivity Analysis Be of Help? Risk Anal. 34, 271–293. https://doi.org/10.1111/risa.12117 Annan, J.D., Hargreaves, J.C., 2004. Efficient parameter estimation for a highly chaotic system. Tellus A 56, 520–526. https://doi.org/10.1111/j.1600-0870.2004.00073.x @@ -169,7 +166,7 @@ Saltelli, A., 2002. Making best use of model evaluations to compute sensitivity Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., Wu, Q., 2019. Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices. Environ. Model. Softw. 114, 29–39. https://doi.org/10.1016/j.envsoft.2019.01.012 Saltelli, A., Annoni, P., 2010. How to avoid a perfunctory sensitivity analysis. Environ. Model. Softw. 25, 1508–1517. https://doi.org/10.1016/j.envsoft.2010.04.012 Saltelli, A., Funtowicz, S., 2014. When all models are wrong. Issues Sci. Technol. 30, 79–85. -Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S., 2008. Global Sensitivity Analysis: The Primer, 1 edition. ed. Wiley-Interscience, Chichester, England ; Hoboken, NJ. +Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S., 2008. Global Sensitivity Analysis: The Primer, 1 edition. ed. Wiley-Interscience, Chichester, England; Hoboken, NJ. Saltelli, A., Stark, P.B., Becker, W., Stano, P., 2015. Climate models as economic guides scientific challenge or quixotic quest? Issues Sci. Technol. 31, 79–84. Saltelli, A., Tarantola, S., 2002. On the relative importance of input factors in mathematical models: safety assessment for nuclear waste disposal. J. Am. Stat. Assoc. 97, 702–709. Saltelli, A., Tarantola, S., Campolongo, F., Ratto, M., 2004. Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. John Wiley & Sons. diff --git a/docs/source/index.rst b/docs/source/index.rst index 16b264b4..689e6249 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -14,6 +14,12 @@ Addressing Uncertainty in MultiSector Dynamics Research 1_introduction 2_diagnostic_modeling_overview_and_perspectives 3_sensitivity_analysis_the_basics + 4_sensitivity_analysis_diagnostic_and_exploratory_modeling + 5_uncertainty_quantification_the_basics + 6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes + 7_conclusion + 8_references + 9_glossary Indices and tables