diff --git a/docs/build/doctrees/1.0_Introduction.doctree b/docs/build/doctrees/1.0_Introduction.doctree deleted file mode 100644 index 962bed4..0000000 Binary files a/docs/build/doctrees/1.0_Introduction.doctree and /dev/null differ diff --git a/docs/build/doctrees/2.0_diagnostic_modeling_overview_and_perspectives.doctree b/docs/build/doctrees/2.0_diagnostic_modeling_overview_and_perspectives.doctree deleted file mode 100644 index 0317a0b..0000000 Binary files a/docs/build/doctrees/2.0_diagnostic_modeling_overview_and_perspectives.doctree and /dev/null differ diff --git a/docs/build/doctrees/ebook.doctree b/docs/build/doctrees/ebook.doctree deleted file mode 100644 index 68a254d..0000000 Binary files a/docs/build/doctrees/ebook.doctree and /dev/null differ diff --git a/docs/build/doctrees/environment.pickle b/docs/build/doctrees/environment.pickle index c78ed9a..fb6786c 100644 Binary files a/docs/build/doctrees/environment.pickle and b/docs/build/doctrees/environment.pickle differ diff --git a/docs/build/doctrees/index.doctree b/docs/build/doctrees/index.doctree index 7d232f7..aa26440 100644 Binary files a/docs/build/doctrees/index.doctree and b/docs/build/doctrees/index.doctree differ diff --git a/docs/build/html/.buildinfo b/docs/build/html/.buildinfo index 7fde3cb..559e144 100644 --- a/docs/build/html/.buildinfo +++ b/docs/build/html/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: 4cdafdf0dcb2a72c8544e6ee51579e52 +config: 6296320580c7687264ff89ca9fd7828a tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/docs/build/html/1.0_Introduction.html b/docs/build/html/1.0_Introduction.html deleted file mode 100644 index fa4dc71..0000000 --- a/docs/build/html/1.0_Introduction.html +++ /dev/null @@ -1,124 +0,0 @@ - - - - - - - - - Introduction — Addressing Uncertainty in MultiSector Dynamics Research v0.1.0 documentation - - - - - - - - - - - - - - - - - - - -
-
-
- - -
- -
-

Introduction

-

This guidance text has been developed in support of the Integrated Multisector Multiscale Modeling (IM3) Science Focus Area’s objective to formally integrate uncertainty into its research tasks. IM3 is focused on innovative modeling to explore how human and natural system landscapes in the United States co-evolve in response to short-term shocks and long-term influences. The project’s challenging scope is to advance our ability to study the interactions between energy, water, land, and urban systems, at scales ranging from local (~1km) to the contiguous United States, while consistently addressing influences such as population change, technology change, heat waves, and drought. Uncertainty and careful model-driven scientific insights are central to IM3’s key MultiSector Dynamics (MSD) science objectives shown below.

-

IM3 key MSD science objectives include:

-

Develop flexible, open-source, and integrated modeling capabilities that capture the structure, dynamic behavior, and emergent properties of the multiscale interactions within and between human and natural systems.

-

Use these capabilities to study the evolution, vulnerability, and resilience of interacting human and natural systems and landscapes from local to continental scales, including their responses to the compounding effects of long-term influences and short-term shocks.

-

Understand the implications of uncertainty in data, observations, models, and model coupling approaches for projections of human-natural system dynamics.

-

Addressing the objectives above poses a strong transdisciplinary challenge that heavily depends on a diversity of models and, more specifically, a consistent framing for making model-based science inferences. The term transdisciplinary science as used here formally implies a deep integration of disciplines to aid our hypothesis driven understanding of coupled human-natural systems–bridging differences in theory, hypothesis generation, modeling, and modes of inference (National Research Council, 2014). The IM3 MSD research focus and questions require a deep integration across disciplines, where new modes of analysis can emerge that rapidly synthesize and exploit advances for making decision relevant insights that at minimum acknowledge uncertainty and more ideally promote a rigorous quantitative mapping of its effects on the generality of claimed scientific insights. More broadly, diverse scientific disciplines engaged in the science of coupled human-natural systems, ranging from natural sciences to engineering and economics, employ a diversity of numerical computer models to study and understand their underlying systems of focus. The utility of these computer models hinges on their ability to represent the underlying real systems with sufficient fidelity and enable the inference of novel insights. This is particularly challenging in the case of coupled human-natural systems where there exists a multitude of interdependent human and natural processes taking place that could potentially be represented. These processes usually translate into modeled representations that are highly complex, non-linear, and exhibit strong interactions and threshold behaviors (Elsawah et al., 2020; Haimes, 2018; Helbing, 2013). Model complexity and detail have also been increasing as a result of our improving understanding of these processes, the availability of data, and the rapid growth in computing power (Saltelli et al., 2019). As model complexity grows, modelers need to specify a lot more information than before: additional model inputs and relationships as more processes are represented, higher resolution data as more observations are collected, new coupling relationships and interactions as models are put together to answer multisector questions (e.g., the land-water-energy nexus). Typically, not all of this information is well known, nor is the impact of these many uncertainties on model outputs well understood. It is further especially difficult to distinguish the effects of individual as well as interacting sources of uncertainty when modeling coupled systems with multisector and multiscale dynamics (Wirtz and Nowak, 2017).

-

Given the challenge and opportunity posed by the disciplinary diversity of IM3, we utilized a team-wide survey to allow the project’s membership to provide their views on how their areas typically address uncertainty, emphasizing key literature examples and domain-specific reviews. Our synthesis of this survey information in Figure 1 summaries the team’s perspectives, enabling a summary of the commonalities and differences for how different disciplinary areas are typically addressing uncertainty. Figure 1 highlights the non-trivial challenge posed by seeking to carefully consider uncertainty across an MSD focused transdisciplinary team. There are significant differences across the team’s contributing disciplines in terms of the methodological approaches and tools used in the treatment of uncertainty. The horizontal axis of the figure represents a conceptual continuum of methodological approaches, ranging from deterministic (no uncertainty) modeling to the theoretical case of fully engaging in modeling all sources of uncertainty. The vertical axis of plot maps the analysis tools that are used in the disciplines’ literature, spanning error-driven historical analyses to full uncertainty quantification. Given that Figure 1 is a conceptual illustration, the mapping of each discipline’s boundaries is not meant to imply exactness. They encompass the scope of feedback attained in the team-wide survey responses. The color circles designate specific sources of uncertainty that could be considered. Within the mapped disciplinary approaches, the color circles distinguish those sources of uncertainty that are addressed in the bodies of literature reported by respondents. Note the complete absence of grey circles designating that at present few if any studies report results for understanding how model coupling relationships shape uncertainty. We can briefly distinguish the key terms of uncertainty quantification (UQ) and uncertainty characterization (UC). UQ refers to the formal focus on the full specification of likelihoods as well as distributional forms necessary to infer the joint probabilistic response across all modeled factors of interest (Cooke, 1991). Alternatively, uncertainty characterization as defined here, refers to exploratory modeling of alternative hypotheses for the co-evolutionary dynamics of influences, stressors, as well as path dependent changes in the form and function of modelled systems (Moallemi et al., 2020a; Walker et al., 2003). Uncertain factors are any model component which is affected by uncertainty: inputs, resolution levels, coupling relationships, model relationships and parameters. When a model has been established as a sufficiently accurate representation of the system some of these factors may reflect elements of the real-world system that the model represents (for example, a population level parameter would reflect a sufficiently accurate representation of the population level in the system under study). As discussed in later sections, the choice of UQ or UC depends on the specific goals of studies, the availability of data, the types of uncertainties (e.g., well-characterized or deep), the complexity of underlying models as well as the computational limits. Deep uncertainty (as opposed to well-characterized) refers to situations where expert opinions consulted on a decision do not know or cannot agree on system boundaries, or the outcomes of interest and their relative importance, or the prior probability distribution for the various uncertain factors present (Kwakkel et al., 2016; W. E. Walker et al., 2013).

-
-Figure 1 -
-

State-of-the-art in different modeling communities, as reported in the survey distributed to IM3 teams. Deterministic Historical Evaluation: model evaluation under fully determined conditions defined using historical observations; Local Sensitivity Analysis: model evaluation performed by varying uncertain factors around specific reference values; Global Sensitivity Analysis: model evaluation performed by varying uncertain factors throughout their entire feasible value space; Uncertainty Characterization: model evaluation under alternative factor hypotheses to explore their implications for model output uncertainty; Uncertainty Quantification: representation of model output uncertainty using probability distributions; Traditional statistical inference: use of analysis results to describe deterministic or probabilistic outcomes resulting from the presence of uncertainty; Narrative scenarios: use of a limited decision-relevant number of scenarios to describe (sets of) changing system outcomes; Exploratory modeling for scenario discovery: use of large ensembles of uncertain conditions to discover decision-relevant combinations of uncertain factors

-
-
-

At present, there is no singular guide for confronting the computational and conceptual challenges of the multi-model, transdisciplinary workflows that characterize ambitious projects such as IM3 (Saltelli et al., 2015). The primary aim of this text is to begin to address this gap and provide guidance for facing these challenges. Chapter 2 provides an overview of diagnostic modeling and the different perspectives for how we should evaluate our models. Chapter 3 the basic methods and concepts for sensitivity analysis. Chapter 4 delves into more technical applications of sensitivity analysis to support diagnostic model evaluation and exploratory modeling. Chapter 5 transitions to an overview of the key concepts and tools for UQ. Chapter 6 transitions to the use of UQ to capture risks and extremes in MSD systems. Chapter 7 provides concluding remarks across the UC and UQ topics covered in this text. The appendices of this text include a glossary of the key concepts as well as example test cases and scripts to showcase various UC and UQ related tools.

-
- - -
- -
-
- -
-
-
- ©2021, Battelle Memorial Institute. - - | - Powered by Sphinx 4.0.2 - & Alabaster 0.7.12 - - | - Page source -
- - - - - - \ No newline at end of file diff --git a/docs/build/html/2.0_diagnostic_modeling_overview_and_perspectives.html b/docs/build/html/2.0_diagnostic_modeling_overview_and_perspectives.html deleted file mode 100644 index 2ab27d7..0000000 --- a/docs/build/html/2.0_diagnostic_modeling_overview_and_perspectives.html +++ /dev/null @@ -1,124 +0,0 @@ - - - - - - - - - Diagnostic Modeling Overview and Perspectives — Addressing Uncertainty in MultiSector Dynamics Research v0.1.0 documentation - - - - - - - - - - - - - - - - - - - -
-
-
- - -
- -
-

Diagnostic Modeling Overview and Perspectives

-

This text prescribes a formal model diagnostic approach to IM3 computational experimentation that is a deliberative and iterative combination of state-of-the-art UC and global sensitivity analysis techniques that progresses from observed history-based fidelity evaluations to forward looking resilience and vulnerability inferences (Gupta et al., 2008; Hadjimichael et al., 2020).

-
-

Perspectives on diagnostic model evaluation

-

When we judge or diagnose models the terms “verification and validation” are commonly used. However, their appropriateness in the context of numerical models representing complex coupled human-natural systems is questionable (Beven, 2002; Oreskes et al., 1994). The core issue relates to the fact that these systems are often not fully known or perfectly implemented when modeled. Rather, they are defined within specific system framings and boundary conditions in an evolving learning process with the goal of making continual progress towards attaining higher levels of fidelity. For example, observations used to evaluate the fidelity of parameterized processes are often measured at a finer resolution than is represented in the model and then must be scaled up for the evaluation. In other cases, numerical models may neglect or simplify system processes because the data is not available or the physical mechanisms are not fully known. If sufficient agreement between prediction and observation is not achieved, it is challenging to know whether these types of modeling choices are the cause, or if other issues, such as deficiencies in the input parameters and/or other modeling assumptions are the true cause of errors. Even if there is high agreement between prediction and observation, the model cannot necessarily be considered validated, as it is always possible that the right values were produced for the wrong reasons. For example, low error can stem from a situation where different errors in underlying assumptions or parameters cancel each other out (“compensatory errors”). Furthermore, coupled human-natural system models are often subject to “equifinality”, a situation where multiple parameterized formulations can produce similar outputs or equally acceptable representations of the observed data. There is therefore no uniquely “true” or validated model, and the common practice of selecting “the best” deterministic calibration set is more of an assumption than a finding (Beven, 1993; Beven and Binley, 1992). The situation becomes even more tenuous when observational data is limited in its scope and/or quality to be insufficient to distinguish model representations or their performance differences. -These limitations on model verification undermine any purely positivist treatment of model validity: that a model should correctly and precisely represent reality to be valid. Under this perspective, closely related to empiricism, statistical tests should be used to compare the model’s output with observations and only through empirical verification can a model or theory be deemed credible. A criticism to this viewpoint (besides the aforementioned challenges for model verification) is that it reduces the justification of a model to the single criterion of predictive ability and accuracy (Barlas and Carpenter, 1990). Authors have argued that this ignores the explanatory power held in models and other procedures, which can also advance scientific knowledge (Toulmin, 1977). These views gave rise to relativist perspectives of science, which instead place more value on model utility in terms of fitness for a specific purpose or inquiry, rather than representational accuracy and predictive ability (Kleindorfer et al., 1998). This viewpoint appears to be most prevalent among practitioners seeking decision relevant insights (i.e., inspire new views vs. predict future conditions). The relativist perspective argues for the use of models as heuristics that can enhance our understanding and conceptions of system behaviors or possibilities (Eker et al., 2018). In contrast, natural sciences favor a positivist perspective, emphasizing similarity between simulation and observation even in application contexts where it is clear that projections are being made for conditions that have never been observed and the system of focus will have evolved structurally beyond the model representation being employed (e.g., decadal to centennial evolution of human-natural systems).

-

These differences in prevalent perspectives are mirrored in how model validation is defined by the two camps: From the relativist perspective, validation is seen as a process of incremental “confidence building” in a model as a mechanism for insight (Barlas, 1996), whereas in natural sciences validation is framed as a way to classify a model as having an acceptable representation of physical reality (Oreskes et al., 1994). Even though the relativist viewpoint does not dismiss the importance of representational accuracy, it does place it within a larger process of establishing confidence through a variety of tools. These tools, not necessarily quantitative, include communicating information between practitioners and modelers, interpreting a multitude of model outputs, and contrasting preferences and viewpoints.

-

On the technical side of the argument, differing views on the methodology of model validation appear as early as in the 1960’s. (Naylor and Finger, 1967) argue that model validation should not be limited to a single metric or test of performance (e.g., a single error metric), but should rather be extended to multiple tests that reflect different aspects of a model’s structure and behavior. This and similar arguments are made in literature to this day (Beven, 2018; Gupta et al., 2012, 2008; Kumar, 2011; Nearing et al., 2020) and are primarily founded on two premises. First, that even though modelers widely recognize that their models are abstractions of the truth, they still make truth claims based on traditional performance metrics that measure the divergence of their model from observation (Nearing et al., 2020). Second, that the natural systems mimicked by the models contain many processes that exhibit significant heterogeneity at various temporal and spatial scales. This heterogeneity is lost when a single performance measure is used, as a result of the inherent loss of process information occurring when transitioning from a highly dimensional and interactive system to the dimension of a single metric (Beven, 2002). These arguments are further elaborated in section 4.1 Understanding Errors.

-

Multiple authors have proposed that, instead, the evaluation of several model performance signatures (characteristics) should be considered to identify model structural errors and achieve a sufficient assessment of model performance (Gupta et al., 1998). There is however a point of departure here, especially when models are used to produce inferences that can inform decisions. When agencies and practitioners use models of their systems for public decisions, those models have already met sufficient conditions for credibility (e.g., acceptable representational fidelity), but may face broader tests on their salience and legitimacy in informing negotiated decisions (Cash et al., 2003; Eker et al., 2018; White et al., 2010). This presents a new challenge to model validation, that of selecting decision-relevant performance metrics, reflective of the system’s stakeholders’ viewpoints, so that the most consequential uncertainties are identified and addressed (Saltelli and Funtowicz, 2014). For complex multisector models at the intersection of climatic, hydrologic, agricultural, energy, or other processes, the output space is made up of a multitude of states and variables, with very different levels of salience to the system’s stakeholders and to their goals being achieved. This is further complicated when such systems are also institutionally and dynamically complex. As a result, a broader set of qualitative and quantitative performance metrics is necessary to evaluate models of such complex systems, one that embraces the plurality of value systems, agencies and perspectives present. For IM3, even though the goal is to develop better projections of future vulnerability and resilience in co-evolving human-natural systems and not to provide decision support per se, it is critical for our multisector, multiscale model evaluation processes to represent stakeholders’ adaptive decision processes credibly.

-

As a final point, when a model is used in a projection mode, its results are also subject to additional uncertainty, as there is no guarantee that the model’s functionality and predictive ability will stay the same as the baseline, where the verification and validation tests were conducted. This challenge requires an additional expansion of the scope of model evaluation: a broader set of uncertain conditions needs to be explored, spanning beyond historical observation and exploring a wide range of unprecedented conditions. This perspective on modeling, termed exploratory (Bankes, 1993), views models as computational experiments that can be used to explore vast ensembles of potential scenarios so as to identify those with consequential effects. Exploratory modeling literature explicitly orients experiments toward stakeholder consequences and decision-relevant inferences and shifts the focus from predicting future conditions to discovering which conditions lead to undesirable or desirable consequences.

-

This evolution in modeling perspectives can be mirrored by the IM3 family of models in a progression from evaluating models relative to observed history to advanced formalized analyses to make inferences on multisector, multiscale vulnerabilities and resilience. Exploratory modeling approaches can help fashion experiments with large numbers of alternative hypotheses on the co-evolutionary dynamics of influences, stressors, as well as path-dependent changes in the form and function of human-natural systems (Weaver et al., 2013). The aim of this text is to therefore guide the reader through the use of sensitivity analysis and uncertainty methods across these perspectives on diagnostic and exploratory modeling.

-
-
- - -
- -
-
- -
-
-
- ©2021, Battelle Memorial Institute. - - | - Powered by Sphinx 4.0.2 - & Alabaster 0.7.12 - - | - Page source -
- - - - - - \ No newline at end of file diff --git a/docs/build/html/_sources/1.0_Introduction.rst.txt b/docs/build/html/_sources/1.0_Introduction.rst.txt deleted file mode 100644 index e319fa1..0000000 --- a/docs/build/html/_sources/1.0_Introduction.rst.txt +++ /dev/null @@ -1,25 +0,0 @@ -Introduction -============ - -This guidance text has been developed in support of the Integrated Multisector Multiscale Modeling (IM3) Science Focus Area’s objective to formally integrate uncertainty into its research tasks. IM3 is focused on innovative modeling to explore how human and natural system landscapes in the United States co-evolve in response to short-term shocks and long-term influences. The project’s challenging scope is to advance our ability to study the interactions between energy, water, land, and urban systems, at scales ranging from local (~1km) to the contiguous United States, while consistently addressing influences such as population change, technology change, heat waves, and drought. Uncertainty and careful model-driven scientific insights are central to IM3’s key MultiSector Dynamics (MSD) science objectives shown below. - -**IM3 key MSD science objectives include:** - -*Develop flexible, open-source, and integrated modeling capabilities that capture the structure, dynamic behavior, and emergent properties of the multiscale interactions within and between human and natural systems.* - -*Use these capabilities to study the evolution, vulnerability, and resilience of interacting human and natural systems and landscapes from local to continental scales, including their responses to the compounding effects of long-term influences and short-term shocks.* - -*Understand the implications of uncertainty in data, observations, models, and model coupling approaches for projections of human-natural system dynamics.* - -Addressing the objectives above poses a strong transdisciplinary challenge that heavily depends on a diversity of models and, more specifically, a consistent framing for making model-based science inferences. The term transdisciplinary science as used here formally implies a deep integration of disciplines to aid our hypothesis driven understanding of coupled human-natural systems--bridging differences in theory, hypothesis generation, modeling, and modes of inference (National Research Council, 2014). The IM3 MSD research focus and questions require a deep integration across disciplines, where new modes of analysis can emerge that rapidly synthesize and exploit advances for making decision relevant insights that at minimum acknowledge uncertainty and more ideally promote a rigorous quantitative mapping of its effects on the generality of claimed scientific insights. More broadly, diverse scientific disciplines engaged in the science of coupled human-natural systems, ranging from natural sciences to engineering and economics, employ a diversity of numerical computer models to study and understand their underlying systems of focus. The utility of these computer models hinges on their ability to represent the underlying real systems with sufficient fidelity and enable the inference of novel insights. This is particularly challenging in the case of coupled human-natural systems where there exists a multitude of interdependent human and natural processes taking place that could potentially be represented. These processes usually translate into modeled representations that are highly complex, non-linear, and exhibit strong interactions and threshold behaviors (Elsawah et al., 2020; Haimes, 2018; Helbing, 2013). Model complexity and detail have also been increasing as a result of our improving understanding of these processes, the availability of data, and the rapid growth in computing power (Saltelli et al., 2019). As model complexity grows, modelers need to specify a lot more information than before: additional model inputs and relationships as more processes are represented, higher resolution data as more observations are collected, new coupling relationships and interactions as models are put together to answer multisector questions (e.g., the land-water-energy nexus). Typically, not all of this information is well known, nor is the impact of these many uncertainties on model outputs well understood. It is further especially difficult to distinguish the effects of individual as well as interacting sources of uncertainty when modeling coupled systems with multisector and multiscale dynamics (Wirtz and Nowak, 2017). - -Given the challenge and opportunity posed by the disciplinary diversity of IM3, we utilized a team-wide survey to allow the project’s membership to provide their views on how their areas typically address uncertainty, emphasizing key literature examples and domain-specific reviews. Our synthesis of this survey information in Figure 1 summaries the team’s perspectives, enabling a summary of the commonalities and differences for how different disciplinary areas are typically addressing uncertainty. Figure 1 highlights the non-trivial challenge posed by seeking to carefully consider uncertainty across an MSD focused transdisciplinary team. There are significant differences across the team’s contributing disciplines in terms of the methodological approaches and tools used in the treatment of uncertainty. The horizontal axis of the figure represents a conceptual continuum of methodological approaches, ranging from deterministic (no uncertainty) modeling to the theoretical case of fully engaging in modeling all sources of uncertainty. The vertical axis of plot maps the analysis tools that are used in the disciplines’ literature, spanning error-driven historical analyses to full uncertainty quantification. Given that Figure 1 is a conceptual illustration, the mapping of each discipline’s boundaries is not meant to imply exactness. They encompass the scope of feedback attained in the team-wide survey responses. The color circles designate specific sources of uncertainty that could be considered. Within the mapped disciplinary approaches, the color circles distinguish those sources of uncertainty that are addressed in the bodies of literature reported by respondents. Note the complete absence of grey circles designating that at present few if any studies report results for understanding how model coupling relationships shape uncertainty. We can briefly distinguish the key terms of uncertainty quantification (UQ) and uncertainty characterization (UC). UQ refers to the formal focus on the full specification of likelihoods as well as distributional forms necessary to infer the joint probabilistic response across all modeled factors of interest (Cooke, 1991). Alternatively, uncertainty characterization as defined here, refers to exploratory modeling of alternative hypotheses for the co-evolutionary dynamics of influences, stressors, as well as path dependent changes in the form and function of modelled systems (Moallemi et al., 2020a; Walker et al., 2003). Uncertain factors are any model component which is affected by uncertainty: inputs, resolution levels, coupling relationships, model relationships and parameters. When a model has been established as a sufficiently accurate representation of the system some of these factors may reflect elements of the real-world system that the model represents (for example, a population level parameter would reflect a sufficiently accurate representation of the population level in the system under study). As discussed in later sections, the choice of UQ or UC depends on the specific goals of studies, the availability of data, the types of uncertainties (e.g., well-characterized or deep), the complexity of underlying models as well as the computational limits. Deep uncertainty (as opposed to well-characterized) refers to situations where expert opinions consulted on a decision do not know or cannot agree on system boundaries, or the outcomes of interest and their relative importance, or the prior probability distribution for the various uncertain factors present (Kwakkel et al., 2016; W. E. Walker et al., 2013). - -.. figure:: _static/figure1_state_of_the_science.png - :alt: Figure 1 - :width: 700px - :align: center - - State-of-the-art in different modeling communities, as reported in the survey distributed to IM3 teams. Deterministic Historical Evaluation: model evaluation under fully determined conditions defined using historical observations; Local Sensitivity Analysis: model evaluation performed by varying uncertain factors around specific reference values; Global Sensitivity Analysis: model evaluation performed by varying uncertain factors throughout their entire feasible value space; Uncertainty Characterization: model evaluation under alternative factor hypotheses to explore their implications for model output uncertainty; Uncertainty Quantification: representation of model output uncertainty using probability distributions; Traditional statistical inference: use of analysis results to describe deterministic or probabilistic outcomes resulting from the presence of uncertainty; Narrative scenarios: use of a limited decision-relevant number of scenarios to describe (sets of) changing system outcomes; Exploratory modeling for scenario discovery: use of large ensembles of uncertain conditions to discover decision-relevant combinations of uncertain factors - -At present, there is no singular guide for confronting the computational and conceptual challenges of the multi-model, transdisciplinary workflows that characterize ambitious projects such as IM3 (Saltelli et al., 2015). The primary aim of this text is to begin to address this gap and provide guidance for facing these challenges. Chapter 2 provides an overview of diagnostic modeling and the different perspectives for how we should evaluate our models. Chapter 3 the basic methods and concepts for sensitivity analysis. Chapter 4 delves into more technical applications of sensitivity analysis to support diagnostic model evaluation and exploratory modeling. Chapter 5 transitions to an overview of the key concepts and tools for UQ. Chapter 6 transitions to the use of UQ to capture risks and extremes in MSD systems. Chapter 7 provides concluding remarks across the UC and UQ topics covered in this text. The appendices of this text include a glossary of the key concepts as well as example test cases and scripts to showcase various UC and UQ related tools. diff --git a/docs/build/html/_sources/2.0_diagnostic_modeling_overview_and_perspectives.rst.txt b/docs/build/html/_sources/2.0_diagnostic_modeling_overview_and_perspectives.rst.txt deleted file mode 100644 index ade82a2..0000000 --- a/docs/build/html/_sources/2.0_diagnostic_modeling_overview_and_perspectives.rst.txt +++ /dev/null @@ -1,21 +0,0 @@ -Diagnostic Modeling Overview and Perspectives -============================================= - -This text prescribes a formal model diagnostic approach to IM3 computational experimentation that is a deliberative and iterative combination of state-of-the-art UC and global sensitivity analysis techniques that progresses from observed history-based fidelity evaluations to forward looking resilience and vulnerability inferences (Gupta et al., 2008; Hadjimichael et al., 2020). - - -Perspectives on diagnostic model evaluation -------------------------------------------- - -When we judge or diagnose models the terms “verification and validation” are commonly used. However, their appropriateness in the context of numerical models representing complex coupled human-natural systems is questionable (Beven, 2002; Oreskes et al., 1994). The core issue relates to the fact that these systems are often not fully known or perfectly implemented when modeled. Rather, they are defined within specific system framings and boundary conditions in an evolving learning process with the goal of making continual progress towards attaining higher levels of fidelity. For example, observations used to evaluate the fidelity of parameterized processes are often measured at a finer resolution than is represented in the model and then must be scaled up for the evaluation. In other cases, numerical models may neglect or simplify system processes because the data is not available or the physical mechanisms are not fully known. If sufficient agreement between prediction and observation is not achieved, it is challenging to know whether these types of modeling choices are the cause, or if other issues, such as deficiencies in the input parameters and/or other modeling assumptions are the true cause of errors. Even if there is high agreement between prediction and observation, the model cannot necessarily be considered validated, as it is always possible that the right values were produced for the wrong reasons. For example, low error can stem from a situation where different errors in underlying assumptions or parameters cancel each other out (“compensatory errors”). Furthermore, coupled human-natural system models are often subject to “equifinality”, a situation where multiple parameterized formulations can produce similar outputs or equally acceptable representations of the observed data. There is therefore no uniquely “true” or validated model, and the common practice of selecting “the best” deterministic calibration set is more of an assumption than a finding (Beven, 1993; Beven and Binley, 1992). The situation becomes even more tenuous when observational data is limited in its scope and/or quality to be insufficient to distinguish model representations or their performance differences. -These limitations on model verification undermine any purely positivist treatment of model validity: that a model should correctly and precisely represent reality to be valid. Under this perspective, closely related to empiricism, statistical tests should be used to compare the model’s output with observations and only through empirical verification can a model or theory be deemed credible. A criticism to this viewpoint (besides the aforementioned challenges for model verification) is that it reduces the justification of a model to the single criterion of predictive ability and accuracy (Barlas and Carpenter, 1990). Authors have argued that this ignores the explanatory power held in models and other procedures, which can also advance scientific knowledge (Toulmin, 1977). These views gave rise to relativist perspectives of science, which instead place more value on model utility in terms of fitness for a specific purpose or inquiry, rather than representational accuracy and predictive ability (Kleindorfer et al., 1998). This viewpoint appears to be most prevalent among practitioners seeking decision relevant insights (i.e., inspire new views vs. predict future conditions). The relativist perspective argues for the use of models as heuristics that can enhance our understanding and conceptions of system behaviors or possibilities (Eker et al., 2018). In contrast, natural sciences favor a positivist perspective, emphasizing similarity between simulation and observation even in application contexts where it is clear that projections are being made for conditions that have never been observed and the system of focus will have evolved structurally beyond the model representation being employed (e.g., decadal to centennial evolution of human-natural systems). - -These differences in prevalent perspectives are mirrored in how model validation is defined by the two camps: From the relativist perspective, validation is seen as a process of incremental “confidence building” in a model as a mechanism for insight (Barlas, 1996), whereas in natural sciences validation is framed as a way to classify a model as having an acceptable representation of physical reality (Oreskes et al., 1994). Even though the relativist viewpoint does not dismiss the importance of representational accuracy, it does place it within a larger process of establishing confidence through a variety of tools. These tools, not necessarily quantitative, include communicating information between practitioners and modelers, interpreting a multitude of model outputs, and contrasting preferences and viewpoints. - -On the technical side of the argument, differing views on the methodology of model validation appear as early as in the 1960’s. (Naylor and Finger, 1967) argue that model validation should not be limited to a single metric or test of performance (e.g., a single error metric), but should rather be extended to multiple tests that reflect different aspects of a model’s structure and behavior. This and similar arguments are made in literature to this day (Beven, 2018; Gupta et al., 2012, 2008; Kumar, 2011; Nearing et al., 2020) and are primarily founded on two premises. First, that even though modelers widely recognize that their models are abstractions of the truth, they still make truth claims based on traditional performance metrics that measure the divergence of their model from observation (Nearing et al., 2020). Second, that the natural systems mimicked by the models contain many processes that exhibit significant heterogeneity at various temporal and spatial scales. This heterogeneity is lost when a single performance measure is used, as a result of the inherent loss of process information occurring when transitioning from a highly dimensional and interactive system to the dimension of a single metric (Beven, 2002). These arguments are further elaborated in section 4.1 Understanding Errors. - -Multiple authors have proposed that, instead, the evaluation of several model performance signatures (characteristics) should be considered to identify model structural errors and achieve a sufficient assessment of model performance (Gupta et al., 1998). There is however a point of departure here, especially when models are used to produce inferences that can inform decisions. When agencies and practitioners use models of their systems for public decisions, those models have already met sufficient conditions for credibility (e.g., acceptable representational fidelity), but may face broader tests on their salience and legitimacy in informing negotiated decisions (Cash et al., 2003; Eker et al., 2018; White et al., 2010). This presents a new challenge to model validation, that of selecting decision-relevant performance metrics, reflective of the system’s stakeholders' viewpoints, so that the most consequential uncertainties are identified and addressed (Saltelli and Funtowicz, 2014). For complex multisector models at the intersection of climatic, hydrologic, agricultural, energy, or other processes, the output space is made up of a multitude of states and variables, with very different levels of salience to the system's stakeholders and to their goals being achieved. This is further complicated when such systems are also institutionally and dynamically complex. As a result, a broader set of qualitative and quantitative performance metrics is necessary to evaluate models of such complex systems, one that embraces the plurality of value systems, agencies and perspectives present. For IM3, even though the goal is to develop better projections of future vulnerability and resilience in co-evolving human-natural systems and not to provide decision support per se, it is critical for our multisector, multiscale model evaluation processes to represent stakeholders’ adaptive decision processes credibly. - -As a final point, when a model is used in a projection mode, its results are also subject to additional uncertainty, as there is no guarantee that the model’s functionality and predictive ability will stay the same as the baseline, where the verification and validation tests were conducted. This challenge requires an additional expansion of the scope of model evaluation: a broader set of uncertain conditions needs to be explored, spanning beyond historical observation and exploring a wide range of unprecedented conditions. This perspective on modeling, termed exploratory (Bankes, 1993), views models as computational experiments that can be used to explore vast ensembles of potential scenarios so as to identify those with consequential effects. Exploratory modeling literature explicitly orients experiments toward stakeholder consequences and decision-relevant inferences and shifts the focus from predicting future conditions to discovering which conditions lead to undesirable or desirable consequences. - -This evolution in modeling perspectives can be mirrored by the IM3 family of models in a progression from evaluating models relative to observed history to advanced formalized analyses to make inferences on multisector, multiscale vulnerabilities and resilience. Exploratory modeling approaches can help fashion experiments with large numbers of alternative hypotheses on the co-evolutionary dynamics of influences, stressors, as well as path-dependent changes in the form and function of human-natural systems (Weaver et al., 2013). The aim of this text is to therefore guide the reader through the use of sensitivity analysis and uncertainty methods across these perspectives on diagnostic and exploratory modeling. diff --git a/docs/build/html/_sources/ebook.rst.txt b/docs/build/html/_sources/ebook.rst.txt deleted file mode 100644 index e1d8664..0000000 --- a/docs/build/html/_sources/ebook.rst.txt +++ /dev/null @@ -1,10 +0,0 @@ -ebook package -============= - -Module contents ---------------- - -.. automodule:: ebook - :members: - :undoc-members: - :show-inheritance: diff --git a/docs/build/html/_sources/index.rst.txt b/docs/build/html/_sources/index.rst.txt index 689e624..137c196 100644 --- a/docs/build/html/_sources/index.rst.txt +++ b/docs/build/html/_sources/index.rst.txt @@ -18,8 +18,10 @@ Addressing Uncertainty in MultiSector Dynamics Research 5_uncertainty_quantification_the_basics 6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes 7_conclusion - 8_references - 9_glossary + 8_glossary + +.. bibliography:: + :style: plain Indices and tables diff --git a/docs/build/html/_static/underscore-1.12.0.js b/docs/build/html/_static/underscore-1.12.0.js deleted file mode 100644 index 3af6352..0000000 --- a/docs/build/html/_static/underscore-1.12.0.js +++ /dev/null @@ -1,2027 +0,0 @@ -(function (global, factory) { - typeof exports === 'object' && typeof module !== 'undefined' ? module.exports = factory() : - typeof define === 'function' && define.amd ? define('underscore', factory) : - (global = global || self, (function () { - var current = global._; - var exports = global._ = factory(); - exports.noConflict = function () { global._ = current; return exports; }; - }())); -}(this, (function () { - // Underscore.js 1.12.0 - // https://underscorejs.org - // (c) 2009-2020 Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors - // Underscore may be freely distributed under the MIT license. - - // Current version. - var VERSION = '1.12.0'; - - // Establish the root object, `window` (`self`) in the browser, `global` - // on the server, or `this` in some virtual machines. We use `self` - // instead of `window` for `WebWorker` support. - var root = typeof self == 'object' && self.self === self && self || - typeof global == 'object' && global.global === global && global || - Function('return this')() || - {}; - - // Save bytes in the minified (but not gzipped) version: - var ArrayProto = Array.prototype, ObjProto = Object.prototype; - var SymbolProto = typeof Symbol !== 'undefined' ? Symbol.prototype : null; - - // Create quick reference variables for speed access to core prototypes. - var push = ArrayProto.push, - slice = ArrayProto.slice, - toString = ObjProto.toString, - hasOwnProperty = ObjProto.hasOwnProperty; - - // Modern feature detection. - var supportsArrayBuffer = typeof ArrayBuffer !== 'undefined', - supportsDataView = typeof DataView !== 'undefined'; - - // All **ECMAScript 5+** native function implementations that we hope to use - // are declared here. - var nativeIsArray = Array.isArray, - nativeKeys = Object.keys, - nativeCreate = Object.create, - nativeIsView = supportsArrayBuffer && ArrayBuffer.isView; - - // Create references to these builtin functions because we override them. - var _isNaN = isNaN, - _isFinite = isFinite; - - // Keys in IE < 9 that won't be iterated by `for key in ...` and thus missed. - var hasEnumBug = !{toString: null}.propertyIsEnumerable('toString'); - var nonEnumerableProps = ['valueOf', 'isPrototypeOf', 'toString', - 'propertyIsEnumerable', 'hasOwnProperty', 'toLocaleString']; - - // The largest integer that can be represented exactly. - var MAX_ARRAY_INDEX = Math.pow(2, 53) - 1; - - // Some functions take a variable number of arguments, or a few expected - // arguments at the beginning and then a variable number of values to operate - // on. This helper accumulates all remaining arguments past the function’s - // argument length (or an explicit `startIndex`), into an array that becomes - // the last argument. Similar to ES6’s "rest parameter". - function restArguments(func, startIndex) { - startIndex = startIndex == null ? func.length - 1 : +startIndex; - return function() { - var length = Math.max(arguments.length - startIndex, 0), - rest = Array(length), - index = 0; - for (; index < length; index++) { - rest[index] = arguments[index + startIndex]; - } - switch (startIndex) { - case 0: return func.call(this, rest); - case 1: return func.call(this, arguments[0], rest); - case 2: return func.call(this, arguments[0], arguments[1], rest); - } - var args = Array(startIndex + 1); - for (index = 0; index < startIndex; index++) { - args[index] = arguments[index]; - } - args[startIndex] = rest; - return func.apply(this, args); - }; - } - - // Is a given variable an object? - function isObject(obj) { - var type = typeof obj; - return type === 'function' || type === 'object' && !!obj; - } - - // Is a given value equal to null? - function isNull(obj) { - return obj === null; - } - - // Is a given variable undefined? - function isUndefined(obj) { - return obj === void 0; - } - - // Is a given value a boolean? - function isBoolean(obj) { - return obj === true || obj === false || toString.call(obj) === '[object Boolean]'; - } - - // Is a given value a DOM element? - function isElement(obj) { - return !!(obj && obj.nodeType === 1); - } - - // Internal function for creating a `toString`-based type tester. - function tagTester(name) { - var tag = '[object ' + name + ']'; - return function(obj) { - return toString.call(obj) === tag; - }; - } - - var isString = tagTester('String'); - - var isNumber = tagTester('Number'); - - var isDate = tagTester('Date'); - - var isRegExp = tagTester('RegExp'); - - var isError = tagTester('Error'); - - var isSymbol = tagTester('Symbol'); - - var isArrayBuffer = tagTester('ArrayBuffer'); - - var isFunction = tagTester('Function'); - - // Optimize `isFunction` if appropriate. Work around some `typeof` bugs in old - // v8, IE 11 (#1621), Safari 8 (#1929), and PhantomJS (#2236). - var nodelist = root.document && root.document.childNodes; - if (typeof /./ != 'function' && typeof Int8Array != 'object' && typeof nodelist != 'function') { - isFunction = function(obj) { - return typeof obj == 'function' || false; - }; - } - - var isFunction$1 = isFunction; - - var hasObjectTag = tagTester('Object'); - - // In IE 10 - Edge 13, `DataView` has string tag `'[object Object]'`. - // In IE 11, the most common among them, this problem also applies to - // `Map`, `WeakMap` and `Set`. - var hasStringTagBug = ( - supportsDataView && hasObjectTag(new DataView(new ArrayBuffer(8))) - ), - isIE11 = (typeof Map !== 'undefined' && hasObjectTag(new Map)); - - var isDataView = tagTester('DataView'); - - // In IE 10 - Edge 13, we need a different heuristic - // to determine whether an object is a `DataView`. - function ie10IsDataView(obj) { - return obj != null && isFunction$1(obj.getInt8) && isArrayBuffer(obj.buffer); - } - - var isDataView$1 = (hasStringTagBug ? ie10IsDataView : isDataView); - - // Is a given value an array? - // Delegates to ECMA5's native `Array.isArray`. - var isArray = nativeIsArray || tagTester('Array'); - - // Internal function to check whether `key` is an own property name of `obj`. - function has(obj, key) { - return obj != null && hasOwnProperty.call(obj, key); - } - - var isArguments = tagTester('Arguments'); - - // Define a fallback version of the method in browsers (ahem, IE < 9), where - // there isn't any inspectable "Arguments" type. - (function() { - if (!isArguments(arguments)) { - isArguments = function(obj) { - return has(obj, 'callee'); - }; - } - }()); - - var isArguments$1 = isArguments; - - // Is a given object a finite number? - function isFinite$1(obj) { - return !isSymbol(obj) && _isFinite(obj) && !isNaN(parseFloat(obj)); - } - - // Is the given value `NaN`? - function isNaN$1(obj) { - return isNumber(obj) && _isNaN(obj); - } - - // Predicate-generating function. Often useful outside of Underscore. - function constant(value) { - return function() { - return value; - }; - } - - // Common internal logic for `isArrayLike` and `isBufferLike`. - function createSizePropertyCheck(getSizeProperty) { - return function(collection) { - var sizeProperty = getSizeProperty(collection); - return typeof sizeProperty == 'number' && sizeProperty >= 0 && sizeProperty <= MAX_ARRAY_INDEX; - } - } - - // Internal helper to generate a function to obtain property `key` from `obj`. - function shallowProperty(key) { - return function(obj) { - return obj == null ? void 0 : obj[key]; - }; - } - - // Internal helper to obtain the `byteLength` property of an object. - var getByteLength = shallowProperty('byteLength'); - - // Internal helper to determine whether we should spend extensive checks against - // `ArrayBuffer` et al. - var isBufferLike = createSizePropertyCheck(getByteLength); - - // Is a given value a typed array? - var typedArrayPattern = /\[object ((I|Ui)nt(8|16|32)|Float(32|64)|Uint8Clamped|Big(I|Ui)nt64)Array\]/; - function isTypedArray(obj) { - // `ArrayBuffer.isView` is the most future-proof, so use it when available. - // Otherwise, fall back on the above regular expression. - return nativeIsView ? (nativeIsView(obj) && !isDataView$1(obj)) : - isBufferLike(obj) && typedArrayPattern.test(toString.call(obj)); - } - - var isTypedArray$1 = supportsArrayBuffer ? isTypedArray : constant(false); - - // Internal helper to obtain the `length` property of an object. - var getLength = shallowProperty('length'); - - // Internal helper to create a simple lookup structure. - // `collectNonEnumProps` used to depend on `_.contains`, but this led to - // circular imports. `emulatedSet` is a one-off solution that only works for - // arrays of strings. - function emulatedSet(keys) { - var hash = {}; - for (var l = keys.length, i = 0; i < l; ++i) hash[keys[i]] = true; - return { - contains: function(key) { return hash[key]; }, - push: function(key) { - hash[key] = true; - return keys.push(key); - } - }; - } - - // Internal helper. Checks `keys` for the presence of keys in IE < 9 that won't - // be iterated by `for key in ...` and thus missed. Extends `keys` in place if - // needed. - function collectNonEnumProps(obj, keys) { - keys = emulatedSet(keys); - var nonEnumIdx = nonEnumerableProps.length; - var constructor = obj.constructor; - var proto = isFunction$1(constructor) && constructor.prototype || ObjProto; - - // Constructor is a special case. - var prop = 'constructor'; - if (has(obj, prop) && !keys.contains(prop)) keys.push(prop); - - while (nonEnumIdx--) { - prop = nonEnumerableProps[nonEnumIdx]; - if (prop in obj && obj[prop] !== proto[prop] && !keys.contains(prop)) { - keys.push(prop); - } - } - } - - // Retrieve the names of an object's own properties. - // Delegates to **ECMAScript 5**'s native `Object.keys`. - function keys(obj) { - if (!isObject(obj)) return []; - if (nativeKeys) return nativeKeys(obj); - var keys = []; - for (var key in obj) if (has(obj, key)) keys.push(key); - // Ahem, IE < 9. - if (hasEnumBug) collectNonEnumProps(obj, keys); - return keys; - } - - // Is a given array, string, or object empty? - // An "empty" object has no enumerable own-properties. - function isEmpty(obj) { - if (obj == null) return true; - // Skip the more expensive `toString`-based type checks if `obj` has no - // `.length`. - var length = getLength(obj); - if (typeof length == 'number' && ( - isArray(obj) || isString(obj) || isArguments$1(obj) - )) return length === 0; - return getLength(keys(obj)) === 0; - } - - // Returns whether an object has a given set of `key:value` pairs. - function isMatch(object, attrs) { - var _keys = keys(attrs), length = _keys.length; - if (object == null) return !length; - var obj = Object(object); - for (var i = 0; i < length; i++) { - var key = _keys[i]; - if (attrs[key] !== obj[key] || !(key in obj)) return false; - } - return true; - } - - // If Underscore is called as a function, it returns a wrapped object that can - // be used OO-style. This wrapper holds altered versions of all functions added - // through `_.mixin`. Wrapped objects may be chained. - function _(obj) { - if (obj instanceof _) return obj; - if (!(this instanceof _)) return new _(obj); - this._wrapped = obj; - } - - _.VERSION = VERSION; - - // Extracts the result from a wrapped and chained object. - _.prototype.value = function() { - return this._wrapped; - }; - - // Provide unwrapping proxies for some methods used in engine operations - // such as arithmetic and JSON stringification. - _.prototype.valueOf = _.prototype.toJSON = _.prototype.value; - - _.prototype.toString = function() { - return String(this._wrapped); - }; - - // Internal function to wrap or shallow-copy an ArrayBuffer, - // typed array or DataView to a new view, reusing the buffer. - function toBufferView(bufferSource) { - return new Uint8Array( - bufferSource.buffer || bufferSource, - bufferSource.byteOffset || 0, - getByteLength(bufferSource) - ); - } - - // We use this string twice, so give it a name for minification. - var tagDataView = '[object DataView]'; - - // Internal recursive comparison function for `_.isEqual`. - function eq(a, b, aStack, bStack) { - // Identical objects are equal. `0 === -0`, but they aren't identical. - // See the [Harmony `egal` proposal](https://wiki.ecmascript.org/doku.php?id=harmony:egal). - if (a === b) return a !== 0 || 1 / a === 1 / b; - // `null` or `undefined` only equal to itself (strict comparison). - if (a == null || b == null) return false; - // `NaN`s are equivalent, but non-reflexive. - if (a !== a) return b !== b; - // Exhaust primitive checks - var type = typeof a; - if (type !== 'function' && type !== 'object' && typeof b != 'object') return false; - return deepEq(a, b, aStack, bStack); - } - - // Internal recursive comparison function for `_.isEqual`. - function deepEq(a, b, aStack, bStack) { - // Unwrap any wrapped objects. - if (a instanceof _) a = a._wrapped; - if (b instanceof _) b = b._wrapped; - // Compare `[[Class]]` names. - var className = toString.call(a); - if (className !== toString.call(b)) return false; - // Work around a bug in IE 10 - Edge 13. - if (hasStringTagBug && className == '[object Object]' && isDataView$1(a)) { - if (!isDataView$1(b)) return false; - className = tagDataView; - } - switch (className) { - // These types are compared by value. - case '[object RegExp]': - // RegExps are coerced to strings for comparison (Note: '' + /a/i === '/a/i') - case '[object String]': - // Primitives and their corresponding object wrappers are equivalent; thus, `"5"` is - // equivalent to `new String("5")`. - return '' + a === '' + b; - case '[object Number]': - // `NaN`s are equivalent, but non-reflexive. - // Object(NaN) is equivalent to NaN. - if (+a !== +a) return +b !== +b; - // An `egal` comparison is performed for other numeric values. - return +a === 0 ? 1 / +a === 1 / b : +a === +b; - case '[object Date]': - case '[object Boolean]': - // Coerce dates and booleans to numeric primitive values. Dates are compared by their - // millisecond representations. Note that invalid dates with millisecond representations - // of `NaN` are not equivalent. - return +a === +b; - case '[object Symbol]': - return SymbolProto.valueOf.call(a) === SymbolProto.valueOf.call(b); - case '[object ArrayBuffer]': - case tagDataView: - // Coerce to typed array so we can fall through. - return deepEq(toBufferView(a), toBufferView(b), aStack, bStack); - } - - var areArrays = className === '[object Array]'; - if (!areArrays && isTypedArray$1(a)) { - var byteLength = getByteLength(a); - if (byteLength !== getByteLength(b)) return false; - if (a.buffer === b.buffer && a.byteOffset === b.byteOffset) return true; - areArrays = true; - } - if (!areArrays) { - if (typeof a != 'object' || typeof b != 'object') return false; - - // Objects with different constructors are not equivalent, but `Object`s or `Array`s - // from different frames are. - var aCtor = a.constructor, bCtor = b.constructor; - if (aCtor !== bCtor && !(isFunction$1(aCtor) && aCtor instanceof aCtor && - isFunction$1(bCtor) && bCtor instanceof bCtor) - && ('constructor' in a && 'constructor' in b)) { - return false; - } - } - // Assume equality for cyclic structures. The algorithm for detecting cyclic - // structures is adapted from ES 5.1 section 15.12.3, abstract operation `JO`. - - // Initializing stack of traversed objects. - // It's done here since we only need them for objects and arrays comparison. - aStack = aStack || []; - bStack = bStack || []; - var length = aStack.length; - while (length--) { - // Linear search. Performance is inversely proportional to the number of - // unique nested structures. - if (aStack[length] === a) return bStack[length] === b; - } - - // Add the first object to the stack of traversed objects. - aStack.push(a); - bStack.push(b); - - // Recursively compare objects and arrays. - if (areArrays) { - // Compare array lengths to determine if a deep comparison is necessary. - length = a.length; - if (length !== b.length) return false; - // Deep compare the contents, ignoring non-numeric properties. - while (length--) { - if (!eq(a[length], b[length], aStack, bStack)) return false; - } - } else { - // Deep compare objects. - var _keys = keys(a), key; - length = _keys.length; - // Ensure that both objects contain the same number of properties before comparing deep equality. - if (keys(b).length !== length) return false; - while (length--) { - // Deep compare each member - key = _keys[length]; - if (!(has(b, key) && eq(a[key], b[key], aStack, bStack))) return false; - } - } - // Remove the first object from the stack of traversed objects. - aStack.pop(); - bStack.pop(); - return true; - } - - // Perform a deep comparison to check if two objects are equal. - function isEqual(a, b) { - return eq(a, b); - } - - // Retrieve all the enumerable property names of an object. - function allKeys(obj) { - if (!isObject(obj)) return []; - var keys = []; - for (var key in obj) keys.push(key); - // Ahem, IE < 9. - if (hasEnumBug) collectNonEnumProps(obj, keys); - return keys; - } - - // Since the regular `Object.prototype.toString` type tests don't work for - // some types in IE 11, we use a fingerprinting heuristic instead, based - // on the methods. It's not great, but it's the best we got. - // The fingerprint method lists are defined below. - function ie11fingerprint(methods) { - var length = getLength(methods); - return function(obj) { - if (obj == null) return false; - // `Map`, `WeakMap` and `Set` have no enumerable keys. - var keys = allKeys(obj); - if (getLength(keys)) return false; - for (var i = 0; i < length; i++) { - if (!isFunction$1(obj[methods[i]])) return false; - } - // If we are testing against `WeakMap`, we need to ensure that - // `obj` doesn't have a `forEach` method in order to distinguish - // it from a regular `Map`. - return methods !== weakMapMethods || !isFunction$1(obj[forEachName]); - }; - } - - // In the interest of compact minification, we write - // each string in the fingerprints only once. - var forEachName = 'forEach', - hasName = 'has', - commonInit = ['clear', 'delete'], - mapTail = ['get', hasName, 'set']; - - // `Map`, `WeakMap` and `Set` each have slightly different - // combinations of the above sublists. - var mapMethods = commonInit.concat(forEachName, mapTail), - weakMapMethods = commonInit.concat(mapTail), - setMethods = ['add'].concat(commonInit, forEachName, hasName); - - var isMap = isIE11 ? ie11fingerprint(mapMethods) : tagTester('Map'); - - var isWeakMap = isIE11 ? ie11fingerprint(weakMapMethods) : tagTester('WeakMap'); - - var isSet = isIE11 ? ie11fingerprint(setMethods) : tagTester('Set'); - - var isWeakSet = tagTester('WeakSet'); - - // Retrieve the values of an object's properties. - function values(obj) { - var _keys = keys(obj); - var length = _keys.length; - var values = Array(length); - for (var i = 0; i < length; i++) { - values[i] = obj[_keys[i]]; - } - return values; - } - - // Convert an object into a list of `[key, value]` pairs. - // The opposite of `_.object` with one argument. - function pairs(obj) { - var _keys = keys(obj); - var length = _keys.length; - var pairs = Array(length); - for (var i = 0; i < length; i++) { - pairs[i] = [_keys[i], obj[_keys[i]]]; - } - return pairs; - } - - // Invert the keys and values of an object. The values must be serializable. - function invert(obj) { - var result = {}; - var _keys = keys(obj); - for (var i = 0, length = _keys.length; i < length; i++) { - result[obj[_keys[i]]] = _keys[i]; - } - return result; - } - - // Return a sorted list of the function names available on the object. - function functions(obj) { - var names = []; - for (var key in obj) { - if (isFunction$1(obj[key])) names.push(key); - } - return names.sort(); - } - - // An internal function for creating assigner functions. - function createAssigner(keysFunc, defaults) { - return function(obj) { - var length = arguments.length; - if (defaults) obj = Object(obj); - if (length < 2 || obj == null) return obj; - for (var index = 1; index < length; index++) { - var source = arguments[index], - keys = keysFunc(source), - l = keys.length; - for (var i = 0; i < l; i++) { - var key = keys[i]; - if (!defaults || obj[key] === void 0) obj[key] = source[key]; - } - } - return obj; - }; - } - - // Extend a given object with all the properties in passed-in object(s). - var extend = createAssigner(allKeys); - - // Assigns a given object with all the own properties in the passed-in - // object(s). - // (https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object/assign) - var extendOwn = createAssigner(keys); - - // Fill in a given object with default properties. - var defaults = createAssigner(allKeys, true); - - // Create a naked function reference for surrogate-prototype-swapping. - function ctor() { - return function(){}; - } - - // An internal function for creating a new object that inherits from another. - function baseCreate(prototype) { - if (!isObject(prototype)) return {}; - if (nativeCreate) return nativeCreate(prototype); - var Ctor = ctor(); - Ctor.prototype = prototype; - var result = new Ctor; - Ctor.prototype = null; - return result; - } - - // Creates an object that inherits from the given prototype object. - // If additional properties are provided then they will be added to the - // created object. - function create(prototype, props) { - var result = baseCreate(prototype); - if (props) extendOwn(result, props); - return result; - } - - // Create a (shallow-cloned) duplicate of an object. - function clone(obj) { - if (!isObject(obj)) return obj; - return isArray(obj) ? obj.slice() : extend({}, obj); - } - - // Invokes `interceptor` with the `obj` and then returns `obj`. - // The primary purpose of this method is to "tap into" a method chain, in - // order to perform operations on intermediate results within the chain. - function tap(obj, interceptor) { - interceptor(obj); - return obj; - } - - // Normalize a (deep) property `path` to array. - // Like `_.iteratee`, this function can be customized. - function toPath(path) { - return isArray(path) ? path : [path]; - } - _.toPath = toPath; - - // Internal wrapper for `_.toPath` to enable minification. - // Similar to `cb` for `_.iteratee`. - function toPath$1(path) { - return _.toPath(path); - } - - // Internal function to obtain a nested property in `obj` along `path`. - function deepGet(obj, path) { - var length = path.length; - for (var i = 0; i < length; i++) { - if (obj == null) return void 0; - obj = obj[path[i]]; - } - return length ? obj : void 0; - } - - // Get the value of the (deep) property on `path` from `object`. - // If any property in `path` does not exist or if the value is - // `undefined`, return `defaultValue` instead. - // The `path` is normalized through `_.toPath`. - function get(object, path, defaultValue) { - var value = deepGet(object, toPath$1(path)); - return isUndefined(value) ? defaultValue : value; - } - - // Shortcut function for checking if an object has a given property directly on - // itself (in other words, not on a prototype). Unlike the internal `has` - // function, this public version can also traverse nested properties. - function has$1(obj, path) { - path = toPath$1(path); - var length = path.length; - for (var i = 0; i < length; i++) { - var key = path[i]; - if (!has(obj, key)) return false; - obj = obj[key]; - } - return !!length; - } - - // Keep the identity function around for default iteratees. - function identity(value) { - return value; - } - - // Returns a predicate for checking whether an object has a given set of - // `key:value` pairs. - function matcher(attrs) { - attrs = extendOwn({}, attrs); - return function(obj) { - return isMatch(obj, attrs); - }; - } - - // Creates a function that, when passed an object, will traverse that object’s - // properties down the given `path`, specified as an array of keys or indices. - function property(path) { - path = toPath$1(path); - return function(obj) { - return deepGet(obj, path); - }; - } - - // Internal function that returns an efficient (for current engines) version - // of the passed-in callback, to be repeatedly applied in other Underscore - // functions. - function optimizeCb(func, context, argCount) { - if (context === void 0) return func; - switch (argCount == null ? 3 : argCount) { - case 1: return function(value) { - return func.call(context, value); - }; - // The 2-argument case is omitted because we’re not using it. - case 3: return function(value, index, collection) { - return func.call(context, value, index, collection); - }; - case 4: return function(accumulator, value, index, collection) { - return func.call(context, accumulator, value, index, collection); - }; - } - return function() { - return func.apply(context, arguments); - }; - } - - // An internal function to generate callbacks that can be applied to each - // element in a collection, returning the desired result — either `_.identity`, - // an arbitrary callback, a property matcher, or a property accessor. - function baseIteratee(value, context, argCount) { - if (value == null) return identity; - if (isFunction$1(value)) return optimizeCb(value, context, argCount); - if (isObject(value) && !isArray(value)) return matcher(value); - return property(value); - } - - // External wrapper for our callback generator. Users may customize - // `_.iteratee` if they want additional predicate/iteratee shorthand styles. - // This abstraction hides the internal-only `argCount` argument. - function iteratee(value, context) { - return baseIteratee(value, context, Infinity); - } - _.iteratee = iteratee; - - // The function we call internally to generate a callback. It invokes - // `_.iteratee` if overridden, otherwise `baseIteratee`. - function cb(value, context, argCount) { - if (_.iteratee !== iteratee) return _.iteratee(value, context); - return baseIteratee(value, context, argCount); - } - - // Returns the results of applying the `iteratee` to each element of `obj`. - // In contrast to `_.map` it returns an object. - function mapObject(obj, iteratee, context) { - iteratee = cb(iteratee, context); - var _keys = keys(obj), - length = _keys.length, - results = {}; - for (var index = 0; index < length; index++) { - var currentKey = _keys[index]; - results[currentKey] = iteratee(obj[currentKey], currentKey, obj); - } - return results; - } - - // Predicate-generating function. Often useful outside of Underscore. - function noop(){} - - // Generates a function for a given object that returns a given property. - function propertyOf(obj) { - if (obj == null) return noop; - return function(path) { - return get(obj, path); - }; - } - - // Run a function **n** times. - function times(n, iteratee, context) { - var accum = Array(Math.max(0, n)); - iteratee = optimizeCb(iteratee, context, 1); - for (var i = 0; i < n; i++) accum[i] = iteratee(i); - return accum; - } - - // Return a random integer between `min` and `max` (inclusive). - function random(min, max) { - if (max == null) { - max = min; - min = 0; - } - return min + Math.floor(Math.random() * (max - min + 1)); - } - - // A (possibly faster) way to get the current timestamp as an integer. - var now = Date.now || function() { - return new Date().getTime(); - }; - - // Internal helper to generate functions for escaping and unescaping strings - // to/from HTML interpolation. - function createEscaper(map) { - var escaper = function(match) { - return map[match]; - }; - // Regexes for identifying a key that needs to be escaped. - var source = '(?:' + keys(map).join('|') + ')'; - var testRegexp = RegExp(source); - var replaceRegexp = RegExp(source, 'g'); - return function(string) { - string = string == null ? '' : '' + string; - return testRegexp.test(string) ? string.replace(replaceRegexp, escaper) : string; - }; - } - - // Internal list of HTML entities for escaping. - var escapeMap = { - '&': '&', - '<': '<', - '>': '>', - '"': '"', - "'": ''', - '`': '`' - }; - - // Function for escaping strings to HTML interpolation. - var _escape = createEscaper(escapeMap); - - // Internal list of HTML entities for unescaping. - var unescapeMap = invert(escapeMap); - - // Function for unescaping strings from HTML interpolation. - var _unescape = createEscaper(unescapeMap); - - // By default, Underscore uses ERB-style template delimiters. Change the - // following template settings to use alternative delimiters. - var templateSettings = _.templateSettings = { - evaluate: /<%([\s\S]+?)%>/g, - interpolate: /<%=([\s\S]+?)%>/g, - escape: /<%-([\s\S]+?)%>/g - }; - - // When customizing `_.templateSettings`, if you don't want to define an - // interpolation, evaluation or escaping regex, we need one that is - // guaranteed not to match. - var noMatch = /(.)^/; - - // Certain characters need to be escaped so that they can be put into a - // string literal. - var escapes = { - "'": "'", - '\\': '\\', - '\r': 'r', - '\n': 'n', - '\u2028': 'u2028', - '\u2029': 'u2029' - }; - - var escapeRegExp = /\\|'|\r|\n|\u2028|\u2029/g; - - function escapeChar(match) { - return '\\' + escapes[match]; - } - - // JavaScript micro-templating, similar to John Resig's implementation. - // Underscore templating handles arbitrary delimiters, preserves whitespace, - // and correctly escapes quotes within interpolated code. - // NB: `oldSettings` only exists for backwards compatibility. - function template(text, settings, oldSettings) { - if (!settings && oldSettings) settings = oldSettings; - settings = defaults({}, settings, _.templateSettings); - - // Combine delimiters into one regular expression via alternation. - var matcher = RegExp([ - (settings.escape || noMatch).source, - (settings.interpolate || noMatch).source, - (settings.evaluate || noMatch).source - ].join('|') + '|$', 'g'); - - // Compile the template source, escaping string literals appropriately. - var index = 0; - var source = "__p+='"; - text.replace(matcher, function(match, escape, interpolate, evaluate, offset) { - source += text.slice(index, offset).replace(escapeRegExp, escapeChar); - index = offset + match.length; - - if (escape) { - source += "'+\n((__t=(" + escape + "))==null?'':_.escape(__t))+\n'"; - } else if (interpolate) { - source += "'+\n((__t=(" + interpolate + "))==null?'':__t)+\n'"; - } else if (evaluate) { - source += "';\n" + evaluate + "\n__p+='"; - } - - // Adobe VMs need the match returned to produce the correct offset. - return match; - }); - source += "';\n"; - - // If a variable is not specified, place data values in local scope. - if (!settings.variable) source = 'with(obj||{}){\n' + source + '}\n'; - - source = "var __t,__p='',__j=Array.prototype.join," + - "print=function(){__p+=__j.call(arguments,'');};\n" + - source + 'return __p;\n'; - - var render; - try { - render = new Function(settings.variable || 'obj', '_', source); - } catch (e) { - e.source = source; - throw e; - } - - var template = function(data) { - return render.call(this, data, _); - }; - - // Provide the compiled source as a convenience for precompilation. - var argument = settings.variable || 'obj'; - template.source = 'function(' + argument + '){\n' + source + '}'; - - return template; - } - - // Traverses the children of `obj` along `path`. If a child is a function, it - // is invoked with its parent as context. Returns the value of the final - // child, or `fallback` if any child is undefined. - function result(obj, path, fallback) { - path = toPath$1(path); - var length = path.length; - if (!length) { - return isFunction$1(fallback) ? fallback.call(obj) : fallback; - } - for (var i = 0; i < length; i++) { - var prop = obj == null ? void 0 : obj[path[i]]; - if (prop === void 0) { - prop = fallback; - i = length; // Ensure we don't continue iterating. - } - obj = isFunction$1(prop) ? prop.call(obj) : prop; - } - return obj; - } - - // Generate a unique integer id (unique within the entire client session). - // Useful for temporary DOM ids. - var idCounter = 0; - function uniqueId(prefix) { - var id = ++idCounter + ''; - return prefix ? prefix + id : id; - } - - // Start chaining a wrapped Underscore object. - function chain(obj) { - var instance = _(obj); - instance._chain = true; - return instance; - } - - // Internal function to execute `sourceFunc` bound to `context` with optional - // `args`. Determines whether to execute a function as a constructor or as a - // normal function. - function executeBound(sourceFunc, boundFunc, context, callingContext, args) { - if (!(callingContext instanceof boundFunc)) return sourceFunc.apply(context, args); - var self = baseCreate(sourceFunc.prototype); - var result = sourceFunc.apply(self, args); - if (isObject(result)) return result; - return self; - } - - // Partially apply a function by creating a version that has had some of its - // arguments pre-filled, without changing its dynamic `this` context. `_` acts - // as a placeholder by default, allowing any combination of arguments to be - // pre-filled. Set `_.partial.placeholder` for a custom placeholder argument. - var partial = restArguments(function(func, boundArgs) { - var placeholder = partial.placeholder; - var bound = function() { - var position = 0, length = boundArgs.length; - var args = Array(length); - for (var i = 0; i < length; i++) { - args[i] = boundArgs[i] === placeholder ? arguments[position++] : boundArgs[i]; - } - while (position < arguments.length) args.push(arguments[position++]); - return executeBound(func, bound, this, this, args); - }; - return bound; - }); - - partial.placeholder = _; - - // Create a function bound to a given object (assigning `this`, and arguments, - // optionally). - var bind = restArguments(function(func, context, args) { - if (!isFunction$1(func)) throw new TypeError('Bind must be called on a function'); - var bound = restArguments(function(callArgs) { - return executeBound(func, bound, context, this, args.concat(callArgs)); - }); - return bound; - }); - - // Internal helper for collection methods to determine whether a collection - // should be iterated as an array or as an object. - // Related: https://people.mozilla.org/~jorendorff/es6-draft.html#sec-tolength - // Avoids a very nasty iOS 8 JIT bug on ARM-64. #2094 - var isArrayLike = createSizePropertyCheck(getLength); - - // Internal implementation of a recursive `flatten` function. - function flatten(input, depth, strict, output) { - output = output || []; - if (!depth && depth !== 0) { - depth = Infinity; - } else if (depth <= 0) { - return output.concat(input); - } - var idx = output.length; - for (var i = 0, length = getLength(input); i < length; i++) { - var value = input[i]; - if (isArrayLike(value) && (isArray(value) || isArguments$1(value))) { - // Flatten current level of array or arguments object. - if (depth > 1) { - flatten(value, depth - 1, strict, output); - idx = output.length; - } else { - var j = 0, len = value.length; - while (j < len) output[idx++] = value[j++]; - } - } else if (!strict) { - output[idx++] = value; - } - } - return output; - } - - // Bind a number of an object's methods to that object. Remaining arguments - // are the method names to be bound. Useful for ensuring that all callbacks - // defined on an object belong to it. - var bindAll = restArguments(function(obj, keys) { - keys = flatten(keys, false, false); - var index = keys.length; - if (index < 1) throw new Error('bindAll must be passed function names'); - while (index--) { - var key = keys[index]; - obj[key] = bind(obj[key], obj); - } - return obj; - }); - - // Memoize an expensive function by storing its results. - function memoize(func, hasher) { - var memoize = function(key) { - var cache = memoize.cache; - var address = '' + (hasher ? hasher.apply(this, arguments) : key); - if (!has(cache, address)) cache[address] = func.apply(this, arguments); - return cache[address]; - }; - memoize.cache = {}; - return memoize; - } - - // Delays a function for the given number of milliseconds, and then calls - // it with the arguments supplied. - var delay = restArguments(function(func, wait, args) { - return setTimeout(function() { - return func.apply(null, args); - }, wait); - }); - - // Defers a function, scheduling it to run after the current call stack has - // cleared. - var defer = partial(delay, _, 1); - - // Returns a function, that, when invoked, will only be triggered at most once - // during a given window of time. Normally, the throttled function will run - // as much as it can, without ever going more than once per `wait` duration; - // but if you'd like to disable the execution on the leading edge, pass - // `{leading: false}`. To disable execution on the trailing edge, ditto. - function throttle(func, wait, options) { - var timeout, context, args, result; - var previous = 0; - if (!options) options = {}; - - var later = function() { - previous = options.leading === false ? 0 : now(); - timeout = null; - result = func.apply(context, args); - if (!timeout) context = args = null; - }; - - var throttled = function() { - var _now = now(); - if (!previous && options.leading === false) previous = _now; - var remaining = wait - (_now - previous); - context = this; - args = arguments; - if (remaining <= 0 || remaining > wait) { - if (timeout) { - clearTimeout(timeout); - timeout = null; - } - previous = _now; - result = func.apply(context, args); - if (!timeout) context = args = null; - } else if (!timeout && options.trailing !== false) { - timeout = setTimeout(later, remaining); - } - return result; - }; - - throttled.cancel = function() { - clearTimeout(timeout); - previous = 0; - timeout = context = args = null; - }; - - return throttled; - } - - // When a sequence of calls of the returned function ends, the argument - // function is triggered. The end of a sequence is defined by the `wait` - // parameter. If `immediate` is passed, the argument function will be - // triggered at the beginning of the sequence instead of at the end. - function debounce(func, wait, immediate) { - var timeout, previous, args, result, context; - - var later = function() { - var passed = now() - previous; - if (wait > passed) { - timeout = setTimeout(later, wait - passed); - } else { - timeout = null; - if (!immediate) result = func.apply(context, args); - // This check is needed because `func` can recursively invoke `debounced`. - if (!timeout) args = context = null; - } - }; - - var debounced = restArguments(function(_args) { - context = this; - args = _args; - previous = now(); - if (!timeout) { - timeout = setTimeout(later, wait); - if (immediate) result = func.apply(context, args); - } - return result; - }); - - debounced.cancel = function() { - clearTimeout(timeout); - timeout = args = context = null; - }; - - return debounced; - } - - // Returns the first function passed as an argument to the second, - // allowing you to adjust arguments, run code before and after, and - // conditionally execute the original function. - function wrap(func, wrapper) { - return partial(wrapper, func); - } - - // Returns a negated version of the passed-in predicate. - function negate(predicate) { - return function() { - return !predicate.apply(this, arguments); - }; - } - - // Returns a function that is the composition of a list of functions, each - // consuming the return value of the function that follows. - function compose() { - var args = arguments; - var start = args.length - 1; - return function() { - var i = start; - var result = args[start].apply(this, arguments); - while (i--) result = args[i].call(this, result); - return result; - }; - } - - // Returns a function that will only be executed on and after the Nth call. - function after(times, func) { - return function() { - if (--times < 1) { - return func.apply(this, arguments); - } - }; - } - - // Returns a function that will only be executed up to (but not including) the - // Nth call. - function before(times, func) { - var memo; - return function() { - if (--times > 0) { - memo = func.apply(this, arguments); - } - if (times <= 1) func = null; - return memo; - }; - } - - // Returns a function that will be executed at most one time, no matter how - // often you call it. Useful for lazy initialization. - var once = partial(before, 2); - - // Returns the first key on an object that passes a truth test. - function findKey(obj, predicate, context) { - predicate = cb(predicate, context); - var _keys = keys(obj), key; - for (var i = 0, length = _keys.length; i < length; i++) { - key = _keys[i]; - if (predicate(obj[key], key, obj)) return key; - } - } - - // Internal function to generate `_.findIndex` and `_.findLastIndex`. - function createPredicateIndexFinder(dir) { - return function(array, predicate, context) { - predicate = cb(predicate, context); - var length = getLength(array); - var index = dir > 0 ? 0 : length - 1; - for (; index >= 0 && index < length; index += dir) { - if (predicate(array[index], index, array)) return index; - } - return -1; - }; - } - - // Returns the first index on an array-like that passes a truth test. - var findIndex = createPredicateIndexFinder(1); - - // Returns the last index on an array-like that passes a truth test. - var findLastIndex = createPredicateIndexFinder(-1); - - // Use a comparator function to figure out the smallest index at which - // an object should be inserted so as to maintain order. Uses binary search. - function sortedIndex(array, obj, iteratee, context) { - iteratee = cb(iteratee, context, 1); - var value = iteratee(obj); - var low = 0, high = getLength(array); - while (low < high) { - var mid = Math.floor((low + high) / 2); - if (iteratee(array[mid]) < value) low = mid + 1; else high = mid; - } - return low; - } - - // Internal function to generate the `_.indexOf` and `_.lastIndexOf` functions. - function createIndexFinder(dir, predicateFind, sortedIndex) { - return function(array, item, idx) { - var i = 0, length = getLength(array); - if (typeof idx == 'number') { - if (dir > 0) { - i = idx >= 0 ? idx : Math.max(idx + length, i); - } else { - length = idx >= 0 ? Math.min(idx + 1, length) : idx + length + 1; - } - } else if (sortedIndex && idx && length) { - idx = sortedIndex(array, item); - return array[idx] === item ? idx : -1; - } - if (item !== item) { - idx = predicateFind(slice.call(array, i, length), isNaN$1); - return idx >= 0 ? idx + i : -1; - } - for (idx = dir > 0 ? i : length - 1; idx >= 0 && idx < length; idx += dir) { - if (array[idx] === item) return idx; - } - return -1; - }; - } - - // Return the position of the first occurrence of an item in an array, - // or -1 if the item is not included in the array. - // If the array is large and already in sort order, pass `true` - // for **isSorted** to use binary search. - var indexOf = createIndexFinder(1, findIndex, sortedIndex); - - // Return the position of the last occurrence of an item in an array, - // or -1 if the item is not included in the array. - var lastIndexOf = createIndexFinder(-1, findLastIndex); - - // Return the first value which passes a truth test. - function find(obj, predicate, context) { - var keyFinder = isArrayLike(obj) ? findIndex : findKey; - var key = keyFinder(obj, predicate, context); - if (key !== void 0 && key !== -1) return obj[key]; - } - - // Convenience version of a common use case of `_.find`: getting the first - // object containing specific `key:value` pairs. - function findWhere(obj, attrs) { - return find(obj, matcher(attrs)); - } - - // The cornerstone for collection functions, an `each` - // implementation, aka `forEach`. - // Handles raw objects in addition to array-likes. Treats all - // sparse array-likes as if they were dense. - function each(obj, iteratee, context) { - iteratee = optimizeCb(iteratee, context); - var i, length; - if (isArrayLike(obj)) { - for (i = 0, length = obj.length; i < length; i++) { - iteratee(obj[i], i, obj); - } - } else { - var _keys = keys(obj); - for (i = 0, length = _keys.length; i < length; i++) { - iteratee(obj[_keys[i]], _keys[i], obj); - } - } - return obj; - } - - // Return the results of applying the iteratee to each element. - function map(obj, iteratee, context) { - iteratee = cb(iteratee, context); - var _keys = !isArrayLike(obj) && keys(obj), - length = (_keys || obj).length, - results = Array(length); - for (var index = 0; index < length; index++) { - var currentKey = _keys ? _keys[index] : index; - results[index] = iteratee(obj[currentKey], currentKey, obj); - } - return results; - } - - // Internal helper to create a reducing function, iterating left or right. - function createReduce(dir) { - // Wrap code that reassigns argument variables in a separate function than - // the one that accesses `arguments.length` to avoid a perf hit. (#1991) - var reducer = function(obj, iteratee, memo, initial) { - var _keys = !isArrayLike(obj) && keys(obj), - length = (_keys || obj).length, - index = dir > 0 ? 0 : length - 1; - if (!initial) { - memo = obj[_keys ? _keys[index] : index]; - index += dir; - } - for (; index >= 0 && index < length; index += dir) { - var currentKey = _keys ? _keys[index] : index; - memo = iteratee(memo, obj[currentKey], currentKey, obj); - } - return memo; - }; - - return function(obj, iteratee, memo, context) { - var initial = arguments.length >= 3; - return reducer(obj, optimizeCb(iteratee, context, 4), memo, initial); - }; - } - - // **Reduce** builds up a single result from a list of values, aka `inject`, - // or `foldl`. - var reduce = createReduce(1); - - // The right-associative version of reduce, also known as `foldr`. - var reduceRight = createReduce(-1); - - // Return all the elements that pass a truth test. - function filter(obj, predicate, context) { - var results = []; - predicate = cb(predicate, context); - each(obj, function(value, index, list) { - if (predicate(value, index, list)) results.push(value); - }); - return results; - } - - // Return all the elements for which a truth test fails. - function reject(obj, predicate, context) { - return filter(obj, negate(cb(predicate)), context); - } - - // Determine whether all of the elements pass a truth test. - function every(obj, predicate, context) { - predicate = cb(predicate, context); - var _keys = !isArrayLike(obj) && keys(obj), - length = (_keys || obj).length; - for (var index = 0; index < length; index++) { - var currentKey = _keys ? _keys[index] : index; - if (!predicate(obj[currentKey], currentKey, obj)) return false; - } - return true; - } - - // Determine if at least one element in the object passes a truth test. - function some(obj, predicate, context) { - predicate = cb(predicate, context); - var _keys = !isArrayLike(obj) && keys(obj), - length = (_keys || obj).length; - for (var index = 0; index < length; index++) { - var currentKey = _keys ? _keys[index] : index; - if (predicate(obj[currentKey], currentKey, obj)) return true; - } - return false; - } - - // Determine if the array or object contains a given item (using `===`). - function contains(obj, item, fromIndex, guard) { - if (!isArrayLike(obj)) obj = values(obj); - if (typeof fromIndex != 'number' || guard) fromIndex = 0; - return indexOf(obj, item, fromIndex) >= 0; - } - - // Invoke a method (with arguments) on every item in a collection. - var invoke = restArguments(function(obj, path, args) { - var contextPath, func; - if (isFunction$1(path)) { - func = path; - } else { - path = toPath$1(path); - contextPath = path.slice(0, -1); - path = path[path.length - 1]; - } - return map(obj, function(context) { - var method = func; - if (!method) { - if (contextPath && contextPath.length) { - context = deepGet(context, contextPath); - } - if (context == null) return void 0; - method = context[path]; - } - return method == null ? method : method.apply(context, args); - }); - }); - - // Convenience version of a common use case of `_.map`: fetching a property. - function pluck(obj, key) { - return map(obj, property(key)); - } - - // Convenience version of a common use case of `_.filter`: selecting only - // objects containing specific `key:value` pairs. - function where(obj, attrs) { - return filter(obj, matcher(attrs)); - } - - // Return the maximum element (or element-based computation). - function max(obj, iteratee, context) { - var result = -Infinity, lastComputed = -Infinity, - value, computed; - if (iteratee == null || typeof iteratee == 'number' && typeof obj[0] != 'object' && obj != null) { - obj = isArrayLike(obj) ? obj : values(obj); - for (var i = 0, length = obj.length; i < length; i++) { - value = obj[i]; - if (value != null && value > result) { - result = value; - } - } - } else { - iteratee = cb(iteratee, context); - each(obj, function(v, index, list) { - computed = iteratee(v, index, list); - if (computed > lastComputed || computed === -Infinity && result === -Infinity) { - result = v; - lastComputed = computed; - } - }); - } - return result; - } - - // Return the minimum element (or element-based computation). - function min(obj, iteratee, context) { - var result = Infinity, lastComputed = Infinity, - value, computed; - if (iteratee == null || typeof iteratee == 'number' && typeof obj[0] != 'object' && obj != null) { - obj = isArrayLike(obj) ? obj : values(obj); - for (var i = 0, length = obj.length; i < length; i++) { - value = obj[i]; - if (value != null && value < result) { - result = value; - } - } - } else { - iteratee = cb(iteratee, context); - each(obj, function(v, index, list) { - computed = iteratee(v, index, list); - if (computed < lastComputed || computed === Infinity && result === Infinity) { - result = v; - lastComputed = computed; - } - }); - } - return result; - } - - // Sample **n** random values from a collection using the modern version of the - // [Fisher-Yates shuffle](https://en.wikipedia.org/wiki/Fisher–Yates_shuffle). - // If **n** is not specified, returns a single random element. - // The internal `guard` argument allows it to work with `_.map`. - function sample(obj, n, guard) { - if (n == null || guard) { - if (!isArrayLike(obj)) obj = values(obj); - return obj[random(obj.length - 1)]; - } - var sample = isArrayLike(obj) ? clone(obj) : values(obj); - var length = getLength(sample); - n = Math.max(Math.min(n, length), 0); - var last = length - 1; - for (var index = 0; index < n; index++) { - var rand = random(index, last); - var temp = sample[index]; - sample[index] = sample[rand]; - sample[rand] = temp; - } - return sample.slice(0, n); - } - - // Shuffle a collection. - function shuffle(obj) { - return sample(obj, Infinity); - } - - // Sort the object's values by a criterion produced by an iteratee. - function sortBy(obj, iteratee, context) { - var index = 0; - iteratee = cb(iteratee, context); - return pluck(map(obj, function(value, key, list) { - return { - value: value, - index: index++, - criteria: iteratee(value, key, list) - }; - }).sort(function(left, right) { - var a = left.criteria; - var b = right.criteria; - if (a !== b) { - if (a > b || a === void 0) return 1; - if (a < b || b === void 0) return -1; - } - return left.index - right.index; - }), 'value'); - } - - // An internal function used for aggregate "group by" operations. - function group(behavior, partition) { - return function(obj, iteratee, context) { - var result = partition ? [[], []] : {}; - iteratee = cb(iteratee, context); - each(obj, function(value, index) { - var key = iteratee(value, index, obj); - behavior(result, value, key); - }); - return result; - }; - } - - // Groups the object's values by a criterion. Pass either a string attribute - // to group by, or a function that returns the criterion. - var groupBy = group(function(result, value, key) { - if (has(result, key)) result[key].push(value); else result[key] = [value]; - }); - - // Indexes the object's values by a criterion, similar to `_.groupBy`, but for - // when you know that your index values will be unique. - var indexBy = group(function(result, value, key) { - result[key] = value; - }); - - // Counts instances of an object that group by a certain criterion. Pass - // either a string attribute to count by, or a function that returns the - // criterion. - var countBy = group(function(result, value, key) { - if (has(result, key)) result[key]++; else result[key] = 1; - }); - - // Split a collection into two arrays: one whose elements all pass the given - // truth test, and one whose elements all do not pass the truth test. - var partition = group(function(result, value, pass) { - result[pass ? 0 : 1].push(value); - }, true); - - // Safely create a real, live array from anything iterable. - var reStrSymbol = /[^\ud800-\udfff]|[\ud800-\udbff][\udc00-\udfff]|[\ud800-\udfff]/g; - function toArray(obj) { - if (!obj) return []; - if (isArray(obj)) return slice.call(obj); - if (isString(obj)) { - // Keep surrogate pair characters together. - return obj.match(reStrSymbol); - } - if (isArrayLike(obj)) return map(obj, identity); - return values(obj); - } - - // Return the number of elements in a collection. - function size(obj) { - if (obj == null) return 0; - return isArrayLike(obj) ? obj.length : keys(obj).length; - } - - // Internal `_.pick` helper function to determine whether `key` is an enumerable - // property name of `obj`. - function keyInObj(value, key, obj) { - return key in obj; - } - - // Return a copy of the object only containing the allowed properties. - var pick = restArguments(function(obj, keys) { - var result = {}, iteratee = keys[0]; - if (obj == null) return result; - if (isFunction$1(iteratee)) { - if (keys.length > 1) iteratee = optimizeCb(iteratee, keys[1]); - keys = allKeys(obj); - } else { - iteratee = keyInObj; - keys = flatten(keys, false, false); - obj = Object(obj); - } - for (var i = 0, length = keys.length; i < length; i++) { - var key = keys[i]; - var value = obj[key]; - if (iteratee(value, key, obj)) result[key] = value; - } - return result; - }); - - // Return a copy of the object without the disallowed properties. - var omit = restArguments(function(obj, keys) { - var iteratee = keys[0], context; - if (isFunction$1(iteratee)) { - iteratee = negate(iteratee); - if (keys.length > 1) context = keys[1]; - } else { - keys = map(flatten(keys, false, false), String); - iteratee = function(value, key) { - return !contains(keys, key); - }; - } - return pick(obj, iteratee, context); - }); - - // Returns everything but the last entry of the array. Especially useful on - // the arguments object. Passing **n** will return all the values in - // the array, excluding the last N. - function initial(array, n, guard) { - return slice.call(array, 0, Math.max(0, array.length - (n == null || guard ? 1 : n))); - } - - // Get the first element of an array. Passing **n** will return the first N - // values in the array. The **guard** check allows it to work with `_.map`. - function first(array, n, guard) { - if (array == null || array.length < 1) return n == null || guard ? void 0 : []; - if (n == null || guard) return array[0]; - return initial(array, array.length - n); - } - - // Returns everything but the first entry of the `array`. Especially useful on - // the `arguments` object. Passing an **n** will return the rest N values in the - // `array`. - function rest(array, n, guard) { - return slice.call(array, n == null || guard ? 1 : n); - } - - // Get the last element of an array. Passing **n** will return the last N - // values in the array. - function last(array, n, guard) { - if (array == null || array.length < 1) return n == null || guard ? void 0 : []; - if (n == null || guard) return array[array.length - 1]; - return rest(array, Math.max(0, array.length - n)); - } - - // Trim out all falsy values from an array. - function compact(array) { - return filter(array, Boolean); - } - - // Flatten out an array, either recursively (by default), or up to `depth`. - // Passing `true` or `false` as `depth` means `1` or `Infinity`, respectively. - function flatten$1(array, depth) { - return flatten(array, depth, false); - } - - // Take the difference between one array and a number of other arrays. - // Only the elements present in just the first array will remain. - var difference = restArguments(function(array, rest) { - rest = flatten(rest, true, true); - return filter(array, function(value){ - return !contains(rest, value); - }); - }); - - // Return a version of the array that does not contain the specified value(s). - var without = restArguments(function(array, otherArrays) { - return difference(array, otherArrays); - }); - - // Produce a duplicate-free version of the array. If the array has already - // been sorted, you have the option of using a faster algorithm. - // The faster algorithm will not work with an iteratee if the iteratee - // is not a one-to-one function, so providing an iteratee will disable - // the faster algorithm. - function uniq(array, isSorted, iteratee, context) { - if (!isBoolean(isSorted)) { - context = iteratee; - iteratee = isSorted; - isSorted = false; - } - if (iteratee != null) iteratee = cb(iteratee, context); - var result = []; - var seen = []; - for (var i = 0, length = getLength(array); i < length; i++) { - var value = array[i], - computed = iteratee ? iteratee(value, i, array) : value; - if (isSorted && !iteratee) { - if (!i || seen !== computed) result.push(value); - seen = computed; - } else if (iteratee) { - if (!contains(seen, computed)) { - seen.push(computed); - result.push(value); - } - } else if (!contains(result, value)) { - result.push(value); - } - } - return result; - } - - // Produce an array that contains the union: each distinct element from all of - // the passed-in arrays. - var union = restArguments(function(arrays) { - return uniq(flatten(arrays, true, true)); - }); - - // Produce an array that contains every item shared between all the - // passed-in arrays. - function intersection(array) { - var result = []; - var argsLength = arguments.length; - for (var i = 0, length = getLength(array); i < length; i++) { - var item = array[i]; - if (contains(result, item)) continue; - var j; - for (j = 1; j < argsLength; j++) { - if (!contains(arguments[j], item)) break; - } - if (j === argsLength) result.push(item); - } - return result; - } - - // Complement of zip. Unzip accepts an array of arrays and groups - // each array's elements on shared indices. - function unzip(array) { - var length = array && max(array, getLength).length || 0; - var result = Array(length); - - for (var index = 0; index < length; index++) { - result[index] = pluck(array, index); - } - return result; - } - - // Zip together multiple lists into a single array -- elements that share - // an index go together. - var zip = restArguments(unzip); - - // Converts lists into objects. Pass either a single array of `[key, value]` - // pairs, or two parallel arrays of the same length -- one of keys, and one of - // the corresponding values. Passing by pairs is the reverse of `_.pairs`. - function object(list, values) { - var result = {}; - for (var i = 0, length = getLength(list); i < length; i++) { - if (values) { - result[list[i]] = values[i]; - } else { - result[list[i][0]] = list[i][1]; - } - } - return result; - } - - // Generate an integer Array containing an arithmetic progression. A port of - // the native Python `range()` function. See - // [the Python documentation](https://docs.python.org/library/functions.html#range). - function range(start, stop, step) { - if (stop == null) { - stop = start || 0; - start = 0; - } - if (!step) { - step = stop < start ? -1 : 1; - } - - var length = Math.max(Math.ceil((stop - start) / step), 0); - var range = Array(length); - - for (var idx = 0; idx < length; idx++, start += step) { - range[idx] = start; - } - - return range; - } - - // Chunk a single array into multiple arrays, each containing `count` or fewer - // items. - function chunk(array, count) { - if (count == null || count < 1) return []; - var result = []; - var i = 0, length = array.length; - while (i < length) { - result.push(slice.call(array, i, i += count)); - } - return result; - } - - // Helper function to continue chaining intermediate results. - function chainResult(instance, obj) { - return instance._chain ? _(obj).chain() : obj; - } - - // Add your own custom functions to the Underscore object. - function mixin(obj) { - each(functions(obj), function(name) { - var func = _[name] = obj[name]; - _.prototype[name] = function() { - var args = [this._wrapped]; - push.apply(args, arguments); - return chainResult(this, func.apply(_, args)); - }; - }); - return _; - } - - // Add all mutator `Array` functions to the wrapper. - each(['pop', 'push', 'reverse', 'shift', 'sort', 'splice', 'unshift'], function(name) { - var method = ArrayProto[name]; - _.prototype[name] = function() { - var obj = this._wrapped; - if (obj != null) { - method.apply(obj, arguments); - if ((name === 'shift' || name === 'splice') && obj.length === 0) { - delete obj[0]; - } - } - return chainResult(this, obj); - }; - }); - - // Add all accessor `Array` functions to the wrapper. - each(['concat', 'join', 'slice'], function(name) { - var method = ArrayProto[name]; - _.prototype[name] = function() { - var obj = this._wrapped; - if (obj != null) obj = method.apply(obj, arguments); - return chainResult(this, obj); - }; - }); - - // Named Exports - - var allExports = { - __proto__: null, - VERSION: VERSION, - restArguments: restArguments, - isObject: isObject, - isNull: isNull, - isUndefined: isUndefined, - isBoolean: isBoolean, - isElement: isElement, - isString: isString, - isNumber: isNumber, - isDate: isDate, - isRegExp: isRegExp, - isError: isError, - isSymbol: isSymbol, - isArrayBuffer: isArrayBuffer, - isDataView: isDataView$1, - isArray: isArray, - isFunction: isFunction$1, - isArguments: isArguments$1, - isFinite: isFinite$1, - isNaN: isNaN$1, - isTypedArray: isTypedArray$1, - isEmpty: isEmpty, - isMatch: isMatch, - isEqual: isEqual, - isMap: isMap, - isWeakMap: isWeakMap, - isSet: isSet, - isWeakSet: isWeakSet, - keys: keys, - allKeys: allKeys, - values: values, - pairs: pairs, - invert: invert, - functions: functions, - methods: functions, - extend: extend, - extendOwn: extendOwn, - assign: extendOwn, - defaults: defaults, - create: create, - clone: clone, - tap: tap, - get: get, - has: has$1, - mapObject: mapObject, - identity: identity, - constant: constant, - noop: noop, - toPath: toPath, - property: property, - propertyOf: propertyOf, - matcher: matcher, - matches: matcher, - times: times, - random: random, - now: now, - escape: _escape, - unescape: _unescape, - templateSettings: templateSettings, - template: template, - result: result, - uniqueId: uniqueId, - chain: chain, - iteratee: iteratee, - partial: partial, - bind: bind, - bindAll: bindAll, - memoize: memoize, - delay: delay, - defer: defer, - throttle: throttle, - debounce: debounce, - wrap: wrap, - negate: negate, - compose: compose, - after: after, - before: before, - once: once, - findKey: findKey, - findIndex: findIndex, - findLastIndex: findLastIndex, - sortedIndex: sortedIndex, - indexOf: indexOf, - lastIndexOf: lastIndexOf, - find: find, - detect: find, - findWhere: findWhere, - each: each, - forEach: each, - map: map, - collect: map, - reduce: reduce, - foldl: reduce, - inject: reduce, - reduceRight: reduceRight, - foldr: reduceRight, - filter: filter, - select: filter, - reject: reject, - every: every, - all: every, - some: some, - any: some, - contains: contains, - includes: contains, - include: contains, - invoke: invoke, - pluck: pluck, - where: where, - max: max, - min: min, - shuffle: shuffle, - sample: sample, - sortBy: sortBy, - groupBy: groupBy, - indexBy: indexBy, - countBy: countBy, - partition: partition, - toArray: toArray, - size: size, - pick: pick, - omit: omit, - first: first, - head: first, - take: first, - initial: initial, - last: last, - rest: rest, - tail: rest, - drop: rest, - compact: compact, - flatten: flatten$1, - without: without, - uniq: uniq, - unique: uniq, - union: union, - intersection: intersection, - difference: difference, - unzip: unzip, - transpose: unzip, - zip: zip, - object: object, - range: range, - chunk: chunk, - mixin: mixin, - 'default': _ - }; - - // Default Export - - // Add all of the Underscore functions to the wrapper object. - var _$1 = mixin(allExports); - // Legacy Node.js API. - _$1._ = _$1; - - return _$1; - -}))); -//# sourceMappingURL=underscore.js.map diff --git a/docs/build/html/ebook.html b/docs/build/html/ebook.html deleted file mode 100644 index 581e9e5..0000000 --- a/docs/build/html/ebook.html +++ /dev/null @@ -1,108 +0,0 @@ - - - - - - - - - ebook package — Addressing Uncertainty in MultiSector Dynamics Research v0.1.0 documentation - - - - - - - - - - - - - - - - - -
-
-
- - -
- -
-

ebook package

-
-

Module contents

-
-
- - -
- -
-
- -
-
-
- ©2021, Battelle Memorial Institute. - - | - Powered by Sphinx 4.0.2 - & Alabaster 0.7.12 - - | - Page source -
- - - - - - \ No newline at end of file diff --git a/docs/build/html/examples.html b/docs/build/html/examples.html index cdbb377..bf256a7 100644 --- a/docs/build/html/examples.html +++ b/docs/build/html/examples.html @@ -88,8 +88,14 @@

Addressing Uncertainty in MultiSector Dyna

Navigation
diff --git a/docs/build/html/genindex.html b/docs/build/html/genindex.html index a596918..7a13d67 100644 --- a/docs/build/html/genindex.html +++ b/docs/build/html/genindex.html @@ -141,8 +141,7 @@

Navigation

  • Uncertainty Quantification: The Basics
  • Uncertainty Quantification: A Tool For Capturing Risks & Extremes
  • Conclusion
    • -
  • References
    • -
  • Glossary
    • +
  • Glossary
    • diff --git a/docs/build/html/index.html b/docs/build/html/index.html index 9cc8194..175105b 100644 --- a/docs/build/html/index.html +++ b/docs/build/html/index.html @@ -111,10 +111,48 @@

    Addressing Uncertainty in MultiSector Dynamics Research
  • Conclusion
    • -
  • References
    • -
  • Glossary
    • +
  • Glossary

    • +

    +
    1
    +

    Roger Cooke and others. Experts in uncertainty: opinion and subjective probability in science. Oxford University Press on Demand, 1991.

    +
    +
    2
    +

    National Research Council and others. Convergence: Facilitating transdisciplinary integration of life sciences, physical sciences, engineering, and beyond. National Academies Press, 2014.

    +
    +
    3
    +

    Sondoss Elsawah, Tatiana Filatova, Anthony J Jakeman, Albert J Kettner, Moira L Zellner, Ioannis N Athanasiadis, Serena H Hamilton, Robert L Axtell, Daniel G Brown, Jonathan M Gilligan, and others. Eight grand challenges in socio-environmental systems modeling. Socio-Environmental Systems Modelling, 2:16226–16226, 2020.

    +
    +
    4
    +

    Saul I Gass and Carl M Harris. Encyclopedia of operations research and management science. Journal of the Operational Research Society, 48(7):759–760, 1997.

    +
    +
    5
    +

    Yacov Y Haimes. Risk modeling of interdependent complex systems of systems: theory and practice. Risk analysis, 38(1):84–98, 2018.

    +
    +
    6
    +

    Dirk Helbing. Globally networked risks and how to respond. Nature, 497(7447):51–59, 2013.

    +
    +
    7
    +

    Jan H Kwakkel, Warren E Walker, and Marjolijn Haasnoot. Coping with the wickedness of public policy problems: approaches for decision making under deep uncertainty. 2016.

    +
    +
    8
    +

    Enayat A Moallemi, Jan Kwakkel, Fjalar J de Haan, and Brett A Bryan. Exploratory modeling for analyzing coupled human-natural systems under uncertainty. Global Environmental Change, 65:102186, 2020.

    +
    +
    9
    +

    Andrea Saltelli, Ksenia Aleksankina, William Becker, Pamela Fennell, Federico Ferretti, Niels Holst, Sushan Li, and Qiongli Wu. Why so many published sensitivity analyses are false: a systematic review of sensitivity analysis practices. Environmental modelling & software, 114:29–39, 2019.

    +
    +
    10
    +

    Andrea Saltelli, Philip B Stark, William Becker, and Pawel Stano. Climate models as economic guides scientific challenge or quixotic quest? Issues in Science and Technology, 31(3):79–84, 2015.

    +
    +
    11
    +

    Warren E Walker, Poul Harremoës, Jan Rotmans, Jeroen P Van Der Sluijs, Marjolein BA Van Asselt, Peter Janssen, and Martin P Krayer von Krauss. Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support. Integrated assessment, 4(1):5–17, 2003.

    +
    +
    12
    +

    Daniel Wirtz and Wolfgang Nowak. The rocky road to extended simulation frameworks covering uncertainty, inversion, optimization and control. Environmental Modelling & Software, 93:180–192, 2017.

    +
    +
    +

    Indices and tables

    diff --git a/docs/build/html/modules.html b/docs/build/html/modules.html index bc1eef1..e532389 100644 --- a/docs/build/html/modules.html +++ b/docs/build/html/modules.html @@ -72,9 +72,14 @@

    Addressing Uncertainty in MultiSector Dyna

    Navigation
    diff --git a/docs/build/html/objects.inv b/docs/build/html/objects.inv index 9b824ac..3c31c6b 100644 Binary files a/docs/build/html/objects.inv and b/docs/build/html/objects.inv differ diff --git a/docs/build/html/preface.html b/docs/build/html/preface.html index 76ab18e..c32056e 100644 --- a/docs/build/html/preface.html +++ b/docs/build/html/preface.html @@ -15,8 +15,6 @@ - - @@ -72,23 +70,21 @@

    Addressing Uncertainty in MultiSector Dyna

    Navigation

    -
    diff --git a/docs/build/html/search.html b/docs/build/html/search.html index 0186f26..1d9c826 100644 --- a/docs/build/html/search.html +++ b/docs/build/html/search.html @@ -90,8 +90,7 @@

    Navigation

  • Uncertainty Quantification: The Basics
  • Uncertainty Quantification: A Tool For Capturing Risks & Extremes
  • Conclusion
    • -
  • References
    • -
  • Glossary
    • +
  • Glossary
    • diff --git a/docs/build/html/searchindex.js b/docs/build/html/searchindex.js index b33cfde..81cb872 100644 --- a/docs/build/html/searchindex.js +++ b/docs/build/html/searchindex.js @@ -1 +1 @@ -Search.setIndex({docnames:["1_introduction","2.1_overview_of_model_diagnostics","2.2_perspectives_on_diagnostic_model_evaluation","2_diagnostic_modeling_overview_and_perspectives","3.1_global_versus_local_sensitivity","3.2_why_perform_sensitivity_analysis","3.3_sensitivity_analysis_applications_for_model_evaluation_and_fidelity_testing","3.4_sensitivity_analysis_applications_for_exploratory_modeling_and_scenario_discovery","3.5.1_one_at_a_time_oat","3.5.2_full_fractional_factorial_sampling","3.5.3_latin_hypercube_sampling","3.5.4_low_discrepancy_sequences","3.5.5_other_types_of_sampling","3.5.6_synthetic_generation_of_input_time_series","3.5_design_of_experiments","3.6.1_derivative_based_methods","3.6.2_elementary_effect_methods","3.6.3_regression_based_methods","3.6.4_regional_sensitivity_analysis","3.6.5_variance_based_methods","3.6.6_analysis_of_variance_anova","3.6.7_moment_independent_density_based_methods","3.6_sensitivity_analysis_methods","3.7_how_to_choose_a_sensitivity_analysis_method_model_traits_and_dimensionality","3.8_software_toolkits","3_sensitivity_analysis_the_basics","4.1_understanding_errors_what_is_controlling_model_performance","4.2_consequential_dynamics_what_is_controlling_model_behaviors_of_interest","4.3_consequential_scenarios_what_is_controlling_consequential_outcomes","4_sensitivity_analysis_diagnostic_and_exploratory_modeling","5.1_why_is_uncertainty_quantification_important_for_understanding_multisector_system_dynamics","5.2_uncertainty_quantification_for_exploratory_modeling","5.3_bayesian_uncertainty_quantification","5.4_uncertainty_quantification_under_deep_uncertainty","5.5_integrating_model_diagnostics_and_uncertainty_quantification","5_uncertainty_quantification_the_basics","6.1_understanding_risk_how_probable_are_extreme_events","6.2_understanding_tails_statistical_modeling_of_extreme_events","6.3_how_to_choose_an_appropriate_method","6.4_how_to_select_a_prior_distribution","6.5_posterior_predictive_checking","6.6_model_selection_and_comparison","6.7.1_scenario_discovery","6.7.2_pre_calibration_glue","6.7.3.1_metropolis_hastings","6.7.3.2_gibbs_sampling","6.7.3.3_hamiltonian_monte_carlo","6.7.3_markov_chain_monte_carlo","6.7.4_particle_based_methods","6.7_what_are_common_methods","6.8.1_markov_chain_monte_carlo_with_the_true_model","6.8.2_markov_chain_monte_carlo_with_surrogate_models","6.8_what_are_example_software_implementations","6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes","7_conclusion","8_references","9_glossary","examples","index","modules","nanites","nanites.tests","preface"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":3,"sphinx.domains.rst":2,"sphinx.domains.std":2,sphinx:56},filenames:["1_introduction.rst","2.1_overview_of_model_diagnostics.rst","2.2_perspectives_on_diagnostic_model_evaluation.rst","2_diagnostic_modeling_overview_and_perspectives.rst","3.1_global_versus_local_sensitivity.rst","3.2_why_perform_sensitivity_analysis.rst","3.3_sensitivity_analysis_applications_for_model_evaluation_and_fidelity_testing.rst","3.4_sensitivity_analysis_applications_for_exploratory_modeling_and_scenario_discovery.rst","3.5.1_one_at_a_time_oat.rst","3.5.2_full_fractional_factorial_sampling.rst","3.5.3_latin_hypercube_sampling.rst","3.5.4_low_discrepancy_sequences.rst","3.5.5_other_types_of_sampling.rst","3.5.6_synthetic_generation_of_input_time_series.rst","3.5_design_of_experiments.rst","3.6.1_derivative_based_methods.rst","3.6.2_elementary_effect_methods.rst","3.6.3_regression_based_methods.rst","3.6.4_regional_sensitivity_analysis.rst","3.6.5_variance_based_methods.rst","3.6.6_analysis_of_variance_anova.rst","3.6.7_moment_independent_density_based_methods.rst","3.6_sensitivity_analysis_methods.rst","3.7_how_to_choose_a_sensitivity_analysis_method_model_traits_and_dimensionality.rst","3.8_software_toolkits.rst","3_sensitivity_analysis_the_basics.rst","4.1_understanding_errors_what_is_controlling_model_performance.rst","4.2_consequential_dynamics_what_is_controlling_model_behaviors_of_interest.rst","4.3_consequential_scenarios_what_is_controlling_consequential_outcomes.rst","4_sensitivity_analysis_diagnostic_and_exploratory_modeling.rst","5.1_why_is_uncertainty_quantification_important_for_understanding_multisector_system_dynamics.rst","5.2_uncertainty_quantification_for_exploratory_modeling.rst","5.3_bayesian_uncertainty_quantification.rst","5.4_uncertainty_quantification_under_deep_uncertainty.rst","5.5_integrating_model_diagnostics_and_uncertainty_quantification.rst","5_uncertainty_quantification_the_basics.rst","6.1_understanding_risk_how_probable_are_extreme_events.rst","6.2_understanding_tails_statistical_modeling_of_extreme_events.rst","6.3_how_to_choose_an_appropriate_method.rst","6.4_how_to_select_a_prior_distribution.rst","6.5_posterior_predictive_checking.rst","6.6_model_selection_and_comparison.rst","6.7.1_scenario_discovery.rst","6.7.2_pre_calibration_glue.rst","6.7.3.1_metropolis_hastings.rst","6.7.3.2_gibbs_sampling.rst","6.7.3.3_hamiltonian_monte_carlo.rst","6.7.3_markov_chain_monte_carlo.rst","6.7.4_particle_based_methods.rst","6.7_what_are_common_methods.rst","6.8.1_markov_chain_monte_carlo_with_the_true_model.rst","6.8.2_markov_chain_monte_carlo_with_surrogate_models.rst","6.8_what_are_example_software_implementations.rst","6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes.rst","7_conclusion.rst","8_references.rst","9_glossary.rst","examples.rst","index.rst","modules.rst","nanites.rst","nanites.tests.rst","preface.rst"],objects:{"":{nanites:[60,0,0,"-"]},"nanites.model":{sum_ints:[60,1,1,""]},"nanites.tests":{test_model:[61,0,0,"-"]},"nanites.tests.test_model":{TestModel:[61,2,1,""]},"nanites.tests.test_model.TestModel":{test_sum_ints:[61,3,1,""]},nanites:{model:[60,0,0,"-"],tests:[61,0,0,"-"]}},objnames:{"0":["py","module","Python module"],"1":["py","function","Python function"],"2":["py","class","Python class"],"3":["py","method","Python method"]},objtypes:{"0":"py:module","1":"py:function","2":"py:class","3":"py:method"},terms:{"0":[54,55],"00":55,"0000287":55,"0000626":55,"0000701":55,"0000915":55,"00035":55,"00073":55,"00082":55,"001":55,"00121":55,"0015":55,"002":55,"0023":55,"00270":55,"00294":55,"003":55,"00353":55,"00401706":55,"0041":55,"0043":55,"0049124104268644":55,"005":55,"00553":55,"006":55,"0060":55,"007":55,"008":55,"009":55,"01":55,"010":55,"011":55,"012":55,"013":55,"0130":55,"014":55,"0143624417705937":55,"0148":55,"0156574":55,"01621459":55,"0167":55,"017":55,"0170052":55,"018":55,"01802":55,"01816001":55,"019":55,"0190115":55,"0190641":55,"02":55,"020":55,"02037":55,"021":55,"0223":55,"0234":55,"02443":55,"02508060":55,"026":55,"02802":55,"0290":55,"03":55,"030":55,"030234210x497726":55,"0309":55,"032":55,"033":55,"0378":55,"04":55,"04016050":55,"04018007":55,"04134":55,"0469":55,"0493":55,"05":[47,49,53,55],"05252":55,"057":55,"06":55,"074019":55,"07811":55,"0796":55,"08":55,"08019":55,"082081699":55,"0855":55,"0870":55,"09":55,"0921":55,"0986":55,"1":[0,2,3,12,14,25,47,49,53,54,55,57],"10":55,"100":55,"1000":[23,25],"1002":55,"1007":55,"1008":55,"1008935410038":55,"10143":55,"1015":55,"1016":55,"10162":55,"102009":55,"1021":55,"102186":55,"1023":55,"1026":55,"1029":55,"1038":55,"10409":55,"104551":55,"10475138":55,"10476423":55,"10476572":55,"10483310":55,"10484804":55,"10485594":55,"1049":55,"1061":55,"1063":55,"107":55,"1071":55,"1073":55,"1076":55,"108":55,"1080":55,"1087":55,"1088":55,"109":55,"1092":55,"1098":55,"1099":55,"11":55,"1100705":55,"1109":55,"111":55,"1111":55,"1115":55,"1116":55,"112":55,"1126":55,"113":55,"11362":55,"1137":55,"11373":55,"114":55,"1151915":55,"1153":55,"117":55,"1175":55,"1176344136":55,"1177":55,"1177011136":55,"1177729694":55,"119":55,"1198":55,"12":55,"1209":55,"12117":55,"1214":55,"123":55,"12348":55,"125":55,"126":55,"1268522":55,"127":55,"1273":55,"1278":55,"12804":55,"1287":55,"12888":55,"129":55,"1295":55,"13":55,"130":55,"1337":55,"1339":55,"1347":55,"135":55,"1354":55,"1355819619840672":55,"1360":55,"1371":55,"138":55,"139":55,"1392":55,"1397":55,"14":55,"140":55,"1400":55,"141":55,"1413":55,"1414":55,"142":55,"143":55,"1432":55,"144":55,"14420":55,"145":55,"1467":55,"1469":55,"14693062":55,"148":55,"1482":55,"1483":55,"1490":55,"1493":55,"15":55,"1508":55,"1509":55,"1510":55,"1517":55,"1520":55,"1539":55,"154":55,"155":55,"1593":55,"16":55,"1600":55,"161":55,"1619048":55,"162":55,"16226":55,"1623":55,"16466":55,"1657":55,"1658":55,"166":55,"1667":55,"1672":55,"1680571":55,"1699114":55,"16m1090648":55,"17":55,"1708":55,"172":55,"174":55,"1748":55,"1763":55,"18":[54,55],"180":55,"18174":55,"183":55,"1884324":55,"189":55,"19":55,"190":55,"1907826116":55,"191":55,"19188":55,"192":55,"194":55,"1943":55,"1949":[14,25,55],"1950":55,"1951":55,"1953":55,"1960":[2,3,14,25,55],"1961":[9,14,25,55],"1963":55,"1964":[35,55],"1967":[2,3,11,14,25,55],"1968":[11,14,25,55],"197":55,"1970":55,"1972":55,"1973":[12,14,25,55],"1974":55,"1975":55,"1976":[11,14,25,55],"1977":[2,3,35,55],"1978":55,"1979":[10,14,25,55],"1980":[6,25,55],"1984":55,"1988":[13,14,25,55],"1989":55,"1990":[2,3,55],"1991":[0,12,14,25,55],"1992":[2,3,47,49,53,55],"1993":[2,3,5,10,14,25,55],"1994":[2,3,11,14,25,55],"1995":[11,14,25,55],"1996":[2,3,55],"1998":[2,3,55],"1999":[12,13,14,25,55],"1_metropolis_hast":[],"1km":0,"2":[0,1,3,23,25,54,55],"20":55,"200":55,"2000":55,"2001":[55,57],"2002":[2,3,6,12,14,25,55],"2002a":[23,25,57],"2003":[0,2,3,55],"2004":[4,6,25,55],"2005":55,"2006":[4,14,25,55],"2007":[4,25,55],"2008":[1,2,3,6,9,10,11,13,14,15,22,25,47,49,53,55],"2008wr006822":55,"2009":55,"2010":[2,3,4,5,6,8,11,14,25,55],"2010wr009769":55,"2011":[2,3,6,23,25,47,49,53,55],"2011wr011044":55,"2012":[2,3,55],"20124":55,"2013":[0,2,3,13,14,25,35,54,55],"2013wr014063":55,"2014":[0,2,3,4,6,14,25,55],"2014wr016827":55,"2015":[0,11,13,14,25,55],"2016":[0,4,6,12,13,14,15,22,25,55],"2017":[0,9,13,14,25,55],"2017ef000607":55,"2017ef000701":55,"2017gl074606":55,"2018":[0,2,3,13,14,25,55],"2018ef001082":55,"2019":[0,1,3,4,13,14,25,35,55],"201907826":55,"2019ef001340":55,"2019wr024918":55,"202":55,"2020":[0,2,3,55],"2020a":[0,55],"2020a16226":55,"2020b":[35,55],"2020wr028079":55,"203":55,"2044":55,"208":55,"21":55,"210":55,"2112":55,"2127":55,"21296":55,"214":55,"2140":55,"2142":55,"215":55,"217":55,"219":55,"22":55,"220":55,"223":55,"228":55,"2289594":55,"23":55,"2304394":55,"2307":55,"231":55,"232":55,"2334940":55,"2344614":55,"235":55,"2351":55,"236":55,"2372":55,"239":55,"24":55,"240":55,"242":55,"245":55,"2465":55,"248":55,"2484":55,"25":[7,25,55],"250":55,"253":55,"260":55,"261":55,"263":55,"268":55,"27":55,"271":55,"279":55,"28":55,"280":55,"2823":55,"2839":55,"289":55,"29":55,"293":55,"297":55,"298":55,"2988185":55,"2_gibbs_sampl":[],"2d":[23,25],"2k":[9,14,25],"2nd":55,"3":[0,4,7,25,55],"30":55,"303":55,"304":55,"305":55,"307":55,"31":55,"311":55,"314":55,"3152":55,"319":55,"32":[9,14,25,55],"321":55,"327":55,"329":55,"33":55,"332":55,"335":55,"3360060305":55,"337":55,"3390":55,"34":55,"3402":55,"341":55,"3437":55,"35":55,"351":55,"355":55,"357":55,"3571":55,"3594":55,"366":55,"37":55,"370":55,"3758":55,"38":55,"3802":55,"381":55,"3813":55,"384":55,"3873":55,"3878":55,"3880":55,"3894":55,"39":55,"393":55,"395":55,"396":55,"3_hamiltonian_monte_carlo":[],"4":[0,2,3,6,25,55],"402":55,"406":55,"409":55,"41":55,"410":55,"411":55,"418":55,"423":55,"425":55,"426":55,"4260060203":55,"43":[9,14,25,55],"435":55,"436":55,"438":55,"44":55,"4419":55,"443":55,"449":55,"455":55,"457":55,"458":55,"4580163":55,"4603":55,"461":55,"4620":55,"464":55,"47":55,"4754":55,"4757":55,"47587":55,"4767596":55,"478":55,"48":55,"485":55,"48760":55,"488":55,"49":55,"494":55,"496":55,"497":55,"5":[0,23,25,55],"50":55,"504":55,"51":55,"511":55,"5147":55,"519":55,"5194":55,"520":55,"526":55,"53":55,"531":55,"5382":55,"5405":55,"545":55,"5452":55,"548":55,"55":55,"554":55,"555":55,"5553":55,"559":55,"56":55,"57":55,"573":55,"574":55,"583":55,"585":55,"589":55,"59":55,"591":55,"597":55,"6":[0,9,10,14,25,55],"60":55,"602":55,"617":55,"619":55,"62":55,"623":55,"627":55,"63":55,"639":55,"64":[9,14,25,55],"641":55,"642":55,"643":55,"644":55,"646":55,"6491":55,"6497":55,"65":55,"66":55,"669":55,"67":55,"68":55,"681":55,"689":55,"69":55,"6924":55,"6_5":55,"7":[0,55],"702":55,"709":55,"70928":55,"71":55,"7152":55,"716":55,"721":55,"723":55,"73":55,"7309":55,"7313":55,"733":55,"737":55,"74":55,"741":55,"75":55,"750":55,"75007":55,"751":55,"76":55,"760":55,"763":55,"77":55,"773":55,"778":55,"787":55,"79":55,"793":55,"795":55,"796":55,"7_1140":55,"7b00506":55,"8":[23,25,55],"80":55,"803":55,"8086":55,"8091":55,"811":55,"812":55,"817":55,"8317":55,"84":55,"85":55,"86":55,"863":55,"869":55,"879":55,"88":55,"887":55,"89":55,"8_13":55,"9":55,"90":55,"90028":55,"90040":55,"90115":55,"90144":55,"90172":55,"91":55,"9137":55,"92":55,"9269":55,"93":55,"9326":55,"94":55,"9405":55,"9416":55,"9458":55,"94jc00572":55,"95":55,"953":55,"954":55,"955":55,"95rg00287":55,"963":55,"9685623":55,"97":55,"971":55,"972":55,"978":55,"9780784480717":55,"979":55,"97wr03495":55,"98":55,"985":55,"9868":55,"99":55,"997":55,"9_1":55,"9_10":55,"abstract":[2,3,54],"b\u00fcrkner":55,"bob\u00e9":55,"case":[0,2,3,4,6,9,14,23,25,47,49,53,54,55,61],"class":61,"do":[0,1,3,23,25,47,49,53],"extr\u00eam":55,"final":[2,3,6,25],"function":[0,2,3,14,25,35,54,55,60],"gonz\u00e1lez":55,"gr\u00fcbler":55,"harremo\u00eb":55,"import":[0,2,3,4,5,8,9,14,23,25,55,57,58],"int":[55,60],"j\u00e4ger":55,"j\u00e9z\u00e9quel":55,"long":[0,47,49,53,55],"new":[0,2,3,5,25,55],"pr\u00e9cipit":55,"public":[2,3,13,14,25,55],"return":[15,22,25,55,60,61],"short":0,"th\u00e9oriqu":55,"transient":[47,49,53],"true":[2,3,6,25,58],"vikstr\u00f6m":55,"while":[0,5,6,8,10,12,13,14,25,47,49,53],A:[2,3,9,11,13,14,55,58],And:58,As:[0,1,2,3,6,7,9,14,25,35,47,49,53,54],At:[0,12,58],Be:[55,57],For:[2,3,4,5,6,7,9,12,13,14,15,22,23,25,47,49,52,58],If:[2,3,4,9,14,25],In:[2,3,4,5,6,8,9,10,14,22,23,25,35,49,53,54],Is:58,It:[0,5,8,13,14,25,47,49,53],Its:[14,25],Nearing:[2,3,55],No:[14,25,55],OF:55,ONE:55,On:[2,3,5,25,55],One:[5,12,47,49,53,55,58],Such:[23,25],THE:55,The:[0,1,2,3,6,7,8,9,10,11,12,13,14,23,47,49,52,53,54,55,57,58],Their:55,There:[0,2,3,12,14,25,49,52,53],These:[0,2,3,11,14,15,22,25],To:[10,14,58],With:[10,14,25],_:[15,22,25],_i:[15,22,25],_static:57,aacb3d:55,aba2423:55,abil:[0,2,3],abl:[47,49,53],about:[5,6,9,14,25,35,47,49,53,55,62],abov:[0,7,13,14,25],abramowitz:55,absenc:0,abusorrah:55,ac:55,acad:55,academi:55,accept:[2,3,55],account:[4,9,14,25],accur:[0,5,7,25],accuraci:[2,3,7,25],achiev:[2,3],acknowledg:0,across:[0,2,3,8,9,10,14,25,47,49,53,54,55],act:35,activ:[14,25,55],ad:[23,25],adapt:[1,2,3,55],add:[54,57],addit:[0,2,3,4,5,7,14,25,54,55],address:[0,1,2,3,10,14,23,25,54,55],adequaci:55,adlouni:55,adrian:55,adv:55,advanc:[0,2,3,5,23,25,54,55],advantag:[11,14,25,49,53],aert:55,affect:[0,14,15,22,25],affin:55,afford:[9,14,23,25],aforement:[2,3],agenc:[2,3],agent:55,aghakouchak:55,agre:0,agreement:[2,3,55],agricultur:[2,3],agronomi:55,aguilar:55,aic:55,aid:[0,55],aim:[0,2,3,6,25],akaik:55,al:[0,1,2,3,4,6,9,10,11,12,13,14,15,22,23,25,35,47,49,53,54],alabdulwahab:55,albrecht:55,alcock:55,aleksankina:55,algorithm:[47,49,53,55],all:[0,6,7,8,9,10,12,14,25,35,47,49,53,55],allei:55,allow:[0,1,3,9,14,25,55],alon:57,along:[12,14,22,25,55],alreadi:[2,3],also:[0,2,3,5,6,9,10,12,14,22,23,25,54],altern:[0,1,2,3,4,5,9,14,25,35,54,55],although:54,alwai:[2,3],am:55,ambigu:55,ambiti:[0,54],american:55,among:[2,3],amount:[12,14,25,47,49,53],amplifi:55,amplitud:[12,14,25],an:[0,1,2,3,4,5,9,10,14,25,52,54,55,58],anal:55,analchem:55,analys:[0,1,2,3,4,6,23,25,49,53,54,55],analysi:[0,1,2,3,4,10,12,13,14,49,53,54,55,58],analyst:[7,9,14,23,25],analyt:[14,25,55],analyz:[8,14,25,55],anderson:[6,25,55],andr:55,andrieu:55,ani:[0,1,2,3,4,5,10,11,14,23,25,60],ann:55,annan:55,annoni:[4,8,14,25,55],anomali:55,anoth:[11,14,25,47,49,53],anova:58,answer:[0,1,3,14,25],antarct:55,antarctica:55,ao:55,aom:55,appear:[2,3,5,25],appendic:[0,54],appl:55,appli:[4,5,6,8,9,14,22,23,25],applic:[0,2,3,4,5,12,14,23,54,55,58],apport:[4,25],apprais:55,approach:[0,2,3,4,8,11,12,13,14,25,47,49,52,53,54,55],appropri:[2,3,4,6,9,10,14,23,25,47,49,54,58],approxim:[11,14,25,47,49,53,55],ar:[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15,22,23,25,35,47,54,55,58],architectur:55,area:[0,47,49,53,55],argu:[2,3],argument:[2,3],around:[0,4,15,22,25],arrai:[10,14,25,55],art:[0,3,47,49,53],artifici:55,arxiv150502350:55,arxiv:55,asap:55,asc:55,ash:55,ask:[23,25],askei:55,asm:55,aspect:[2,3,55],asselt:55,assess:[2,3,5,14,23,25,47,49,52,53,55],assimil:55,assist:55,assoc:55,associ:[49,53],assum:[8,9,14,15,22,25],assumpt:[1,2,3,5,7,14,23,25,35],astro:55,asymptot:[47,49,53,55],athanasiadi:55,atlant:55,atmo:55,attain:[0,2,3,11,14,25],attract:[10,14,25],audienc:[],auer:55,author:[2,3],autocorrel:[47,49,53,55],autom:55,avail:[0,2,3,13,14,25],averag:[49,53],avoid:55,awar:[6,23,25],axi:0,axiom:55,axtel:55,b:[4,6,9,14,25,55,60],back:[6,25],bader:55,bai:55,bakker:55,baldassarr:55,bam:55,bangladesh:55,bank:[2,3,5,25,55],bar:[23,25],barla:[2,3,55],barro:55,base:[0,2,3,4,5,6,14,23,52,54,55,58,61],baselin:[2,3],basi:[1,3,23,25,55],basic:[0,58],basin:55,basto:55,batcheld:55,bathymetri:55,batouli:55,bawa:55,bay:55,bayesian:[52,53,55,58],becaus:[2,3,10,14,23,25],becker:55,beckman:55,becom:[2,3,6,14,25],beek:[13,14,25,55],been:[0,1,2,3,4,5,6,7,9,12,14,25,47,49,53,54],befor:[0,14,25],begin:[0,47,49,53],behavior:[0,1,2,3,5,6,14,25,35,54,55,58],being:[2,3,9,10,14,23,25],belitz:55,below:[0,6,23,25,47,49,52,53],bennett:55,berlin:55,besid:[2,3],best:[2,3,55],betancourt:55,better:[2,3,13,14,23,25,52,53,54,55],between:[0,1,2,3,4,9,10,11,14,23,25],bevacqua:55,beven:[2,3,55],beyond:[2,3,12,13,14,25,49,53,55],bhar:[13,14,25,55],bia:[9,14,25,47,49,53,55],bias:[4,10,14,25,55],bic:55,bick:55,bilir:55,binlei:[2,3,55],biologi:55,biomateri:55,biometrika:55,bloemen:55,blog:57,boca:55,bodi:[0,13,14,25],bone:55,book:[],borgomeo:[13,14,25,55],borgonovo:[4,6,15,22,25,55],both:[49,53,54],bottom:[7,25,55],botzen:55,boukouvala:55,boundari:[0,2,3,55],boussouf:55,bouwer:55,bouziota:55,box:[9,14,25,55],boyd:55,br:55,brandimart:55,bridg:0,briefli:0,broad:[4,25],broader:[2,3],broadli:[0,49,53,54],brook:55,broomel:55,brown:55,bryan:55,bryant:[5,25,55],buchanan:55,budescu:55,build:[2,3,12,14,25,52,53,55],built:[52,53],bull:55,burden:[6,9,14,25],burdensom:[9,14,25],burn:[47,49,53],burnham:55,butler:55,c:[4,9,10,14,15,22,25,52,53,55],cabrero:55,cageo:55,calcul:[14,25,55],calibr:[2,3,5,6,25,55,58],call:[23,25,47,49,53],calvin:55,cambridg:55,camco:55,cameron:55,camp:[2,3],campaign:[6,25],campolongo:55,can:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,15,22,23,25,35,47,49,53,54,55],cancel:[2,3],candid:[9,14,25],cannot:[0,2,3,4,9,10,14,23,25],cao:55,capabl:[0,54],captur:[0,13,14,25,54,58],carbon:55,care:[0,55],carefulli:0,cariboni:55,carlier:55,carlin:55,carlo:[1,3,6,11,14,25,55,58],carpent:[2,3,55],cash:[2,3,55],categori:[4,5,25,47,49,53],caus:[2,3,5,6,14,25],caveat:[23,25],centenni:[2,3],central:0,centuri:[13,14,25,55],chain:[55,58],challeng:[0,2,3,8,14,25,49,53,54,55],cham:55,chan:55,chanc:55,chang:[0,2,3,5,6,7,8,13,14,25,35,54,55],chao:55,chaotic:55,chapman:55,chapter:0,charackli:55,character:[0,1,3,11,13,14,25,35,54,55],characterist:[2,3,14,25],chart:[1,3],chatterje:55,che:55,cheap:[23,25],check:58,chem:55,chemic:55,chemistri:55,chen:55,chiaromont:55,chichest:55,choic:[0,1,2,3,14,25,35,47,49,53,54,55],choos:58,chosen:[4,14,25],ciccazzo:55,circl:0,circuit:55,circul:55,citat:54,cite:[13,14,25],citi:[14,25],civ:55,claim:[0,2,3],clarifi:[14,25,55],clark:55,classifi:[1,2,3,49,53],clear:[2,3,23,25],clim:55,climat:[2,3,13,14,25,55],close:[2,3,14,25],cluster:[11,14,25],co:[0,2,3,35,54,55],coast:55,coastal:55,code:[52,53,55],coeffici:55,coerc:55,coher:55,cole:55,colei:55,collabor:55,collect:[0,55],collin:55,color:0,combin:[0,3,9,14,25,55],come:[1,3,13,14,25],commod:[13,14,25],common:[0,2,3,6,10,14,25,52,58],commonli:[2,3,5,9,14,25],commun:[0,2,3,55],compar:[2,3,9,10,14,25,47,49,53,55],comparison:[13,14,22,25,35,55,58],compartment:55,compchemeng:55,compensatori:[2,3],complet:0,complex:[0,2,3,4,5,6,14,23,25,54,55],complic:[2,3],compon:[0,5,6,25],compos:[12,14,25],compound:[0,13,14,25,55],comprehens:[9,14,25,54,55],comput:[0,2,3,5,6,8,9,14,23,25,47,49,52,53,54,55],computation:[9,14,23,25],concentr:55,concept:[0,2,3,54],conceptu:[0,1,3,49,53,54,55],conclud:[0,47,49,53],conclus:[6,23,25,58],condit:[0,2,3,14,25,47,49,53,54],conduct:[2,3,4,14,25,49,53],confer:55,confid:[2,3],confirm:55,confound:[9,14,25],confront:0,connor:55,conov:55,consequ:[2,3,54,55],consequenti:[1,2,3,5,7,13,14,25,35,54,58],consid:[0,2,3,4,5,6,7,9,13,14,23,25,35,54],consider:[4,25,52,53],consist:[0,1,3,14,25,35,54],constrain:[10,14,23,25],constraint:[14,25,55],construct:[47,49,53,55],consult:0,consum:[5,25],contain:[2,3,54],content:59,context:[2,3,5,6,13,14,25,55],contigu:0,continent:0,continu:[2,3,9,14,25],continuum:0,contour:[4,25],contrast:[2,3,4,25],contribut:[0,6,25,55,57],control:[1,3,5,10,14,25,55,58],conveni:[4,25],converg:[23,25,47,49,53,55],convers:[6,25],cook:[0,55],cope:55,copula:55,core:[2,3,11,14,25],cornford:55,corones:55,correctli:[2,3],correl:[10,13,14,25],correspond:[9,14,25],cost:[5,8,9,14,25],costli:[4,25],could:[0,6,14,25,54],council:[0,55],coupl:[0,1,2,3,14,25,54,55],cover:[0,55],coverag:[10,14,25],cozannet:55,crc:55,creation:55,credenc:[5,25],credibl:[2,3,55],cremaschi:55,criteria:[11,14,25,55],criterion:[2,3,55],critic:[2,3,8,14,25,54,55],crombecq:55,cross:55,crucial:[6,25],crutzen:55,csrg:55,cube:55,cukier:[12,14,25,55],current:55,currin:55,cut:[6,25],cutoff:[6,25],cycl:[54,55],d:[9,10,14,23,25,55],da:55,dac:55,daedalu:55,dai:[2,3],dalal:[11,14,25,55],damag:55,data:[0,1,2,3,5,14,25,35,54,55],dauaek:55,de:55,dead:55,deal:[9,14,25],decad:[2,3],decarbon:55,deci:55,decid:[9,14,25],decis:[0,1,2,3,5,7,25,55],decompos:[1,3],deconto:55,deem:[2,3,47,49,53],deep:[0,13,14,25,49,53,54,55,58],deepli:[35,55],defens:[6,25],defici:[2,3],defin:[0,1,2,3,14,25,35,55],definit:[4,22,25,54],degre:[6,23,25],del:55,delib:[3,54],deliv:[23,25],delsa:55,delv:0,demand:[6,7,13,14,25,55],demarta:55,den:55,denot:[6,12,14,25,57],densiti:58,departur:[2,3],depend:[0,2,3,4,6,10,23,35,47,49,53,54],depth:54,der:55,deriv:[10,14,58],describ:[0,22,25,35,49,53],descript:55,design:[0,6,9,10,11,54,55,58],desir:[2,3],despit:[14,25],dessai:55,detail:[0,5,6,12,14,22,25,54],detc2003:55,detect:55,determin:[0,47,49,53],determinist:[0,2,3,55],dev:55,develop:[0,2,3,5,12,13,14,25,54,55],deviat:[11,14,25],dhaen:55,di:55,diagnos:[2,3,47,49,53],diagnost:[0,6,25,47,49,53,54,55,58],dickson:55,dietari:[7,25],differ:[0,2,3,4,5,8,12,14,25,35,47,49,53],differenti:55,difficult:0,digit:55,dimens:[2,3,11,14,25,55],dimension:[2,3,4,10,14,49,53,55,58],dioxid:55,direct:[1,3,5,12,14,25],directli:[47,49,53],disadvantag:[14,25,49,53],disast:55,discard:[47,49,53],disciplin:[0,54],disciplinari:[0,54],discontinu:[23,25],discov:[0,2,3,5,7,14,25],discoveri:[0,5,55,58],discredit:[5,25],discrep:[55,58],discret:[9,10,14,25],discuss:[0,4,13,14,22,25,47,49,52,53,54],dismiss:[2,3],dispers:[47,49,53],dispos:55,disrupt:55,distinguish:[0,2,3,10,14,25,54],distribut:[0,1,3,9,11,12,14,25,35,47,49,55,58],diverg:[2,3],divers:[0,10,14,25,54],divid:[10,14,25],dnf:55,doc:57,doctrin:55,doe:[2,3,4,25,55],doi:55,dokken:55,domain:[0,14,25,55],dominantli:[1,3],dorazio:55,doucet:55,draw:[47,49,53],drawn:[5,14,25],dream:55,drive:55,driven:[0,55],drought:[0,13,14,25,55],dubiou:[5,25],due:[4,25,47,49,53],dunson:55,duong:55,dure:[14,25],dyn:55,dynam:[0,2,3,47,49,53,54,55],e0156574:55,e0170052:55,e0190115:55,e0190641:55,e1278:55,e19100555:55,e2019wr024918:55,e2020wr028079:55,e2508:55,e:[0,1,2,3,4,5,6,7,8,9,11,13,14,25,52,53,54,55,57],e_:57,eaba2423:55,each:[0,2,3,4,6,8,9,10,12,14,22,23,25,49,52,53],eam:55,earli:[2,3],earth:55,eas:[8,14,25],eason:55,ebi:55,ebook:57,ecklei:55,ecolog:55,econ:55,econom:[0,55],ed:55,edinburgh:55,edit:55,edmond:55,edward:[6,25,55],effect:[0,2,3,4,5,6,8,9,10,14,47,49,53,55,57,58],effici:[14,25,55],effort:[5,25,52,53],eight:55,either:[1,3,5,14,25],ejor:55,eker:[2,3,55],el:55,elabor:[2,3,6,7,14,25],electr:55,electron:55,element:[0,14,25,55],elementari:58,elev:55,elicit:[14,25],elimin:[6,11,14,25],ellsberg:55,elsawah:[0,55],elsewher:[1,3],elucid:[6,25],embark:[14,25],embrac:[2,3,55],emerg:0,emiss:55,emphas:[0,2,3,54],empir:[1,2,3,4,25],empiric:[2,3],emploi:[0,2,3,11,14,25],emul:[49,52,53,55],enabl:[0,52,53],encompass:0,encourag:[13,14,25],encyclopedia:55,end:[14,25],energi:[0,2,3,7,13,14,25,55],eng:55,engag:0,engin:[0,55],england:55,english:55,enhanc:[1,2,3,55],enough:[7,25],ensembl:[0,1,2,3,55],ensur:[11,14,25,47,49,53],entir:[0,4,14,25],entropi:55,env:55,environ:55,environment:55,environmetr:55,envsoft:55,eo:55,eoq:55,epistem:[9,14,25,55],equal:[2,3,10,14,25],equat:55,equifin:[2,3,55],equival:[7,25,55],error:[0,2,3,7,11,14,25,47,49,53,55,58],escap:[47,49,53],esm:55,especi:[0,2,3,13,14,25,35],ess:[47,49,53],essai:55,establish:[0,2,3,5,7,25],estim:[9,11,13,14,23,25,47,49,53,55],estrada:55,et:[0,1,2,3,4,6,9,10,11,12,13,14,15,22,23,25,35,47,49,53,54,55],etc:[13,14,25],eur:55,euro:55,european:55,eutroph:55,evalu:[0,5,7,14,15,22,23,47,49,53,54,55,58],evan:55,even:[2,3,11,14,25],evenli:[11,14,25],evensen:55,event:[13,14,25,55,58],eventu:[47,49,53],ever:[14,25],everi:[10,14,25],evid:[6,25,55],evolut:[0,2,3],evolutionari:[0,2,3,35,54],evolv:[0,2,3],exacerb:55,exact:0,exactli:35,examin:[10,13,14,25],exampl:[0,2,3,6,7,9,10,12,13,14,23,25,47,49,54,58],execut:[6,25],exert:[14,25],exhibit:[0,2,3],exist:[0,12,13,14,25,47,49,52,53],expans:[2,3],expect:[57,61],expens:[23,25],experi:[1,2,3,4,6,9,54,55,58],experiment:[3,10,14,25,54],expert:[0,14,25,55],explain:[1,3,6,25],explanatori:[2,3],explicitli:[2,3],exploit:0,explor:[0,2,3,4,8,13,14,15,22,25,35,47,49,53,54,55],exploratori:[0,1,2,3,5,13,14,49,53,54,55,58],exponenti:[9,14,25],exposur:55,extend:[2,3,12,14,25,55],extens:[5,25],extent:[47,49,53],extract:[14,25],extrapol:[5,25],extrem:[0,9,13,14,25,55,58],f:55,face:[0,2,3,6,25,55],facilit:[47,49,53,55],fact:[2,3],factor:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,25,47,49,53,55,57],factori:[55,58],faith:[7,25],falkenmark:55,fals:55,falsif:55,famili:[2,3,54],far:[11,12,14,25,47,49,53],farmer:55,fashion:[2,3,54],fast:[12,14,25,55],favor:[2,3],feasibl:[0,4,14,25,54,55],featur:[22,25,54],fed:[14,25],feedback:0,fennel:55,ferdou:55,ferretti:55,feuillard:55,few:[0,14,25],fewer:[6,11,14,25],feyen:55,fidel:[0,2,3,5,7,58],field:[14,25,55],fifth:55,fig:[4,6,7,9,10,14,23,25],figur:[0,1,3,23,25,54,55],filatova:55,fill:[10,11,14,25],filter:[6,25,55],find:[2,3],finer:[2,3],finger:[2,3,55],finit:55,first:[1,2,3,12,14,23,25,57],fischbach:55,fisher:[14,25,55],fit:[2,3,55],fix:[6,7,8,12,14,25,49,53],fl:55,flegal:[47,49,53,55],flexibl:0,flood:55,flow:55,focu:[0,1,2,3,6,7,8,14,25,35,49,52,53],focus:[0,13,14,25],follow:[5,13,14,15,22,25,57],forecast:55,forest:55,form:[0,1,2,3,5,14,25,35,54,55],formal:[0,1,2,3,14,25,47,49,53,54,55],formul:[2,3,55],fortran:[52,53],fortuin:55,forward:3,fosa:55,fossil:55,found:[2,3,5,7,14,25],foundat:[1,3,5,25,55],four:[9,14,25],fourier:[12,14,25],fourth:55,frac:[15,22,25,57],fraction:[55,58],frame:[0,2,3,5,25,54],framework:[1,3,14,25,52,53,55],francisco:55,frankignoul:[35,55],frechett:55,free:[52,53],freedman:55,freeli:[6,25],freer:55,freita:55,frequenc:55,from:[0,1,2,3,6,10,11,14,25,47,49,53,54,55],frontlin:55,fu:55,fuel:55,full:[0,47,49,53,58],fuller:55,fulli:[0,2,3],fundament:[14,25],funtowicz:[2,3,55],further:[0,2,3,6,13,14,25],furthermor:[2,3,23,25],fusion:55,futur:[1,2,3,6,25,55],g:[0,1,2,3,4,6,7,9,11,13,14,15,22,25,52,53,54,55],ga:55,gabri:55,galeotti:55,game:55,ganeshan:55,gao:55,gap:[0,11,14,25,55],garner:55,gass:55,gatelli:55,gaug:[5,6,25],gaussian:[52,53,55],gave:[2,3],gelfand:55,gelman:[47,49,53,55],geman:55,gener:[0,1,3,4,9,10,11,12,47,49,53,55,58],genova:55,gent:55,geo:55,geogr:55,geophi:55,geosci:55,geostroph:55,geri:55,get:[1,3],geyer:[47,49,53,55],ghan:55,ghanem:55,giardino:55,gibb:[55,58],gidanski:55,gilligan:55,girma:55,given:[0,6,8,14,23,25],glob:55,global:[0,1,3,6,55,58],gloenvcha:55,glossari:[0,54,58],glue:[55,58],gmd:55,goal:[0,2,3,4,25],gober:55,godsil:55,gold:[47,49,53],gomez:55,good:[47,49,53,55],goodman:55,gordon:55,gorissen:55,grand:55,grang:55,graphic:[23,25],greatest:[6,25],grei:0,grid:55,group:[5,25,55],grow:[0,7,9,14,25,49,53],growth:[0,7,9,14,25,55],gsa:[1,3],guan:55,guarante:[2,3,10,11,14,25,47,49,53],guid:[0,2,3,54,55],guidanc:[0,54],guilla:55,gupta:[2,3,55],guston:55,h:55,ha:[0,1,3,4,5,6,7,9,14,25,35,47,49,53],haan:55,haasnoot:55,hack:55,hackbarth:55,hadjimichael:[3,55],haer:55,hagan:55,haim:[0,55],hall:55,hamilton:55,hamiltonian:[55,58],hamm:[4,25,55],han:55,hand:[5,14,23,25],handbook:55,haran:55,hardli:[14,25],hargreav:55,hartin:55,harwayn:55,hasselmann:[35,55],hast:[55,58],have:[0,2,3,4,5,6,8,9,12,13,14,23,25,47,49,53,54],hazard:55,health:55,heat:0,heather:55,heavili:[0,4,25],hector:55,heidelberg:55,height:55,helb:[0,55],held:[2,3],help:[2,3,5,6,25,54,55],helton:[14,25,55],hendri:55,heng:55,here:[0,1,2,3,23,25,49,52,53],herman:[13,14,25,55],herrera:[13,14,25,55],hess:55,hester:55,heterogen:[2,3],heurist:[2,3,5,25,47,49,53],hibbard:55,high:[2,3,7,13,14,25],higher:[0,2,3,9,14,25],highli:[0,1,2,3,5,8,10,14,25,55],highlight:0,hill:55,hindcast:55,hing:[0,7,25],hinkel:55,hinton:55,hirsch:55,histor:[0,2,3,4,25],histori:[2,3,13,14,25,54,55],hit:55,hoboken:55,hoc:[23,25],hoffman:55,hole:[14,25],holland:55,holst:55,holzmann:55,hop:55,horizont:0,hornberg:[6,25,55],horton:55,host:[5,25],hot:[13,14,25],hous:55,houtekam:55,how:[0,1,2,3,4,8,14,15,22,54,55,58],howev:[2,3,4,7,12,14,25,47,49,52,53],http:55,httpcran:55,hu:55,hub:55,human:[0,1,2,3,14,25,54,55],hunter:[9,14,25,55],hurk:55,hurrel:55,hurtt:55,hurvich:55,husbi:55,hwang:55,hybrid:[49,53,55],hydro:55,hydrol:55,hydrolog:[2,3,55],hyp:55,hypercub:[9,11,55,58],hypothes:[0,2,3,35,54],hypothesi:[0,1,3,55],i:[2,3,5,6,8,11,14,15,22,25,55,57],iaij:55,ic:55,idea:[47,49,53,55],ideal:[0,1,3,14,25,47,49,53],identif:55,identifi:[1,2,3,5,6,25],iee:55,ieee:55,ignor:[2,3],ii:55,illumin:[6,25],illustr:[0,1,3,4,25,55],im3:[0,2,3,54],imag:55,immedi:[13,14,25],immers:55,impact:[0,13,14,25,55],implement:[2,3,8,14,25,55,58],impli:0,implic:[0,55],imposs:[4,6,14,25],imprecis:55,improv:[0,1,3,6,7,11,14,25,47,49,53,55],includ:[0,2,3,4,12,14,25,52,53,54],inconsequenti:[7,25],inconsist:[52,53],incorpor:[13,14,25,52,53],increas:[0,6,7,8,9,14,23,25,55],increment:[2,3],inde:[9,14,25],independ:[1,3,8,14,23,47,49,53,55,58],index:[15,22,23,25,57,58],indic:[4,6,14,23,25,55,57],individu:[0,1,3,6,8,13,14,25,57],induc:55,infer:[0,1,2,3,5,7,14,25,35,47,49,52,53,54,55],influenc:[0,1,2,3,4,8,14,25,35,54,55],influenti:[5,6,25],info:55,inform:[0,1,2,3,4,5,6,8,9,10,13,14,23,25,35,55,62],infrastruct:55,infrastructur:55,inher:[2,3,52,53],initi:[1,3,47,49,53],inlet:55,innov:[0,1,3,54],input:[0,1,2,3,4,6,12,15,22,35,55,58],inquiri:[2,3,5,25],insid:55,insight:[0,2,3,54],inspir:[2,3],inst:55,instabl:55,instal:[],instanc:[5,6,7,25,52,53],instead:[2,3,4,5,9,14,25,35],institution:[2,3,55],insuffici:[2,3,13,14,25],integ:60,integr:[0,1,3,55,58],intel:55,intens:[23,25],interact:[0,1,2,3,4,6,8,9,14,23,25],interconnect:55,interdepend:[0,55],interdiscip:55,interest:[0,1,3,5,6,8,13,14,25,58],intergovernment:55,intermedi:55,intern:55,interpret:[2,3,47,49,53],intersci:55,intersect:[2,3],interv:[9,14,25],introduc:[14,25],introduct:[54,55,58],invari:[23,25,55],invers:55,invest:[5,25,55],investig:[5,25,55],involv:[47,49,53,54],iooss:[11,14,25,55],ip:55,issu:[2,3,55],iter:[3,8,14,25,47,49,53,54,55],its:[0,1,2,3,4,5,6,8,9,14,25,47,49,53,55],itself:[6,25],j:55,jacob:55,jakeman:55,jamuna:55,janssen:55,jasra:55,jayn:55,jhydrol:55,jin:[11,14,25,55],johansson:55,john:55,johnson:55,joint:0,jone:55,journal:55,judg:[2,3],judgement:[47,49,53],julia:[52,53],justif:[2,3,35],justifi:35,k:[9,14,25,55],kai:55,kainuma:55,kalman:55,kandlikar:55,karniadaki:55,kasmalkar:55,kass:55,keep:57,kei:[0,14,25,47,49,52,53,54],keith:55,keller:55,kennedi:55,kept:[8,14,25],kershaw:55,kettner:55,kiehl:55,kind:[14,25],kingdom:55,kirsch:[13,14,25,55],kissel:55,kleindorf:[2,3,55],klufa:55,km:55,know:[0,2,3,5,25],knowledg:[2,3,55],known:[0,2,3,6,25],kolstad:55,koomei:55,kopp:55,koutsoyianni:55,kovat:55,kraan:55,kram:55,krauss:55,krei:55,kucherenko:[11,14,25,55],kullback:55,kumar:[2,3,55],kundzewicz:55,kunreuth:55,kushner:55,kwakkel:[0,55],l:55,laboratori:[14,25],lack:[47,49,53],lade:55,lamarqu:55,lamontagn:[13,14,25,55],lamperti:55,land:0,landeg:55,landscap:0,lang:55,langan:55,languag:[7,25,52,53],lant:55,larg:[0,2,3,4,13,14,25,54,55],larger:[2,3,10,14,25,47,49,53],largest:[6,25],larson:55,later:[0,5,14,25],latest:57,latin:[9,55,58],latorr:55,lawrenc:55,le:55,lead:[2,3,6,7,25],learn:[2,3,54,55],least:[5,23,25],leav:[14,25,55],lee:55,left:[6,25],legitimaci:[2,3,55],leibler:55,lempert:[5,25,55],len:54,lend:[5,25],length:[14,25,55],less:[10,14,25,35],let:[15,22,25],lett:55,lettenmai:55,leve:55,level:[0,2,3,9,10,11,14,23,25,55],levi:55,lh:[11,58],li:55,life:55,like:[5,6,23,25],likelihood:[0,35,47,49,53,55],limit:[0,2,3,4,5,8,9,11,13,14,22,23,25,49,52,53,54,55],lin:55,lind:55,lindsai:55,linear:[0,4,14,23,25,55],linearli:[8,14,25],link:55,lipscomb:55,literatur:[0,2,3,4,6,9,12,13,14,25],liu:55,loa:55,local:[0,6,8,14,15,22,55,58],locat:[8,14,25],log:55,logic:55,lond:55,longer:[5,25],look:[3,55],lorenz:55,loss:[2,3,55],lost:[2,3],lot:0,louck:[13,14,25,55],louisiana:55,low:[2,3,55,58],lower:[9,11,14,25],lukac:55,lump:55,lumpi:[11,14,25],m:[4,25,55],maccracken:55,mach:55,machin:55,made:[2,3,6,14,25],magnitud:[23,25],mahecha:55,mahowald:55,mai:[0,2,3,35,47,49,52,53],main:[8,9,14,25,54],major:[5,25],make:[0,2,3,6,7,9,14,23,25,54,55,61],makowski:55,manag:55,manceau:55,mandat:[23,25],mani:[0,2,3,5,14,25,47,49,52,53,55],maniyar:55,manner:[5,6,8,10,14,25],mao:55,map:[0,4,6,7,14,25,54],maraun:55,marchau:[35,55],margin:[47,49,53],markov:[55,58],markovian:[47,49,53],marrel:55,marsh:55,marshal:55,martin:55,martiu:55,maruri:55,massmann:[14,25,55],mastrandrea:55,masui:55,mater:55,materi:55,math:55,mathemat:[11,14,15,22,25,55],matlab:[52,53],matott:55,matthew:55,max:55,maxim:[10,14,25],maximum:[47,49,53],mcinernei:55,mckai:[10,14,25,55],mcmc:[47,49,52,53,55],mcmcpack:55,mcneil:55,mea:55,mean:[1,3,11,14,25],meant:0,measur:[2,3,5,6,9,11,14,15,22,23,25,55],mech:55,mechan:[2,3,52,53,55],mechanist:55,mechanobiolog:55,medda:[13,14,25,55],mediterranean:55,meinshausen:55,membership:0,meng:55,mentaschi:55,mention:[7,12,14,25],meridion:55,met:[2,3],metamodel:55,meteorol:55,method:[0,2,3,4,6,8,10,11,12,13,14,52,54,55,58],methodnam:61,methodol:55,methodolog:[0,2,3,14,25,55],metric:[1,2,3,5,6,7,8,14,25],metropoli:[14,25,55,58],miao:55,mice:55,microsimul:55,might:[5,13,14,25],mijer:55,milli:[13,14,25,55],mimick:[2,3],mind:[7,13,14,25],mine:[14,25],minim:[11,14,25],minimum:[0,6,14,23,25],mirror:[2,3,54],miscalcul:[4,25],misconverg:[47,49,53],miss:[8,14,23,25],mitchel:[11,14,25,55],mitig:[14,25],mix:[47,49,53,55],moallemi:[0,35,55],modal:55,mode:[0,2,3,6,25,55],model:[0,4,5,8,9,11,13,14,15,22,47,49,54,55,57,58,59],modern:[54,55],modif:[10,14,25],modifi:[10,14,25,55],modul:[55,58,59],moment:58,mon:55,monetari:[5,25],mont:[1,3,6,11,14,25,55,58],montgomeri:[9,14,25,55],moral:55,more:[0,1,2,3,4,6,7,8,9,12,14,23,25,35,47,49,52,53],moreov:[49,53,54],morgan:55,morri:[11,12,14,25,55],most:[1,2,3,4,5,6,7,10,11,14,22,23,25,35,49,53,54],mostafavi:55,motiv:[5,14,23,25],move:[12,14,25],msd:0,msd_uncertainty_ebook:57,much:[10,11,14,25],multi:[0,13,14,25,54,55],multiag:55,multidecad:55,multidimension:[11,14,25],multimodel:55,multipl:[2,3,8,14,23,25,47,49,53,55],multiscal:[0,2,3],multisector:[0,2,3],multitud:[0,2,3],must:[2,3,47,49,53],my:57,n:[4,9,10,12,14,23,25,55],nakicenov:55,narr:0,nasa:55,nat:55,natarajan:55,nation:[0,55],nativ:[52,53],natl:55,natur:[0,1,2,3,14,25,54,55],nature12047:55,nature17145:55,naylor:[2,3,55],nb:[6,25],ncrg:55,neal:55,necessari:[0,1,2,3,6,9,14,25],necessarili:[2,3],need:[0,1,2,3,7,14,25],neg:[14,25,55],negat:[8,14,25],neglect:[2,3,55],neglig:[6,9,14,25],negoti:[2,3],neill:55,nelder:55,network:55,neural:55,never:[2,3,14,25],next:[4,25],nexu:0,nhess:55,nielsen:55,nikoletopoulo:55,nj:55,nk:[9,14,25],noah:55,nois:55,nolan:55,nomin:[5,6,14,15,22,25],non:[0,6,13,14,25,47,49,53,54,55],noncommensur:55,nonlinear:[4,8,14,23,25,55],nonperiod:55,nor:[0,14,25],norfolk:55,normal:55,norton:[14,25,55],nossent:[23,25],notat:[15,22,25],note:[0,49,53,54],novel:[0,55],now:[7,25,55],nowak:[0,55],nuclear:55,number:[0,2,3,6,9,10,11,12,14,23,25,47,49,53,54],numer:[0,2,3,4,5,6,14,25,55],ny:55,o:55,oat:[15,22,58],oberstein:55,object:[0,6,25,54,55],observ:[0,2,3,6,13,14,23,25,54,55],obtain:[9,14,25,47,49,53,57],occur:[2,3],ocean:55,oddo:55,often:[2,3,4,13,14,25,55],oliv:55,onc:[12,14,25,47,49,53],one:[1,2,3,5,6,8,9,10,11,12,14,15,22,25,35,47,49,53,55],onli:[2,3,4,8,9,14,15,22,23,25,47,49,53,55],oort:55,open:[0,54],oper:[52,53,55],opinion:[0,14,25,55],oppenheim:55,opportun:0,oppos:0,opr:55,optim:[11,14,25,55],order:[6,9,14,23,25,57],oresk:[2,3,55],org:55,organ:[5,25],orgpackag:55,orient:[2,3],origin:[1,3],orlean:55,orthogon:[10,14,25,55],ortiz:55,ortolano:55,other:[2,3,4,5,6,8,9,23,47,49,53,58],otherwis:[4,25],ouarda:55,our:[0,1,2,3,5,25,54],out:[2,3,4,25,55],outcom:[0,1,3,5,6,25,35,55,58],output:[0,1,2,3,4,5,6,7,8,11,14,15,22,25,55,57],ouyang:55,over:[14,25],overal:54,overcom:[8,14,25],overlap:[10,14,25],overturn:55,overview:[0,23,52,53,54,55,58],overyear:55,oxford:55,p:[9,14,25,55],packag:[52,53,55,59],page:58,pair:[5,25],panel:[4,7,25,55],paradigm:55,paradox:55,paramet:[0,1,2,3,4,5,8,9,10,14,25,47,49,53,55,60],parameter:[1,2,3,5,25,35],parametr:[1,3,4,25,49,53],pari:55,park:[11,14,25,55],part:55,partial:[4,15,22,25],participatori:55,particl:[55,58],particular:[1,3,14,15,22,25,49,53],particularli:0,past:[13,14,25,55],path:[0,2,3,35,54,55],pathwai:55,pattern:55,pawson:55,pearson:55,peel:55,peng:55,pepper:55,per:[2,3,9,14,25],percent:57,perfectli:[2,3],perform:[0,1,2,3,4,7,9,14,15,22,23,55,58],perfunctori:55,perish:55,persist:[13,14,25],perspect:[0,7,25,49,53,54,55,58],pertin:[5,25,52,53],perturb:[4,12,14,15,22,25],petschek:55,philo:55,philosoph:[54,55],philosophi:55,phim:55,phy:55,physic:[2,3,9,14,25,55],pianosi:[6,12,14,15,22,25,55],pie:[1,3],pillo:55,pinpoint:[6,25],pip:62,place:[0,2,3,7,25],placehold:60,plan:55,platform:[14,25,52,53,54,55],plausibl:[14,25],pleas:[52,53],plischk:[4,25,55],plo:55,plot:[0,4,25,47,49,53],plot_experimental_design:57,plural:[2,3],pna:55,point:[2,3,4,11,12,14,25,55],polici:55,policymak:55,pollard:55,polynomi:55,pone:55,pool:[47,49,53],poon:55,popper:55,popul:[0,7,14,25],popular:[8,14,25,52,53],pose:0,posit:55,positivist:[2,3],possibl:[2,3,6,9,14,25,35,55],posterior:[47,49,55,58],posteriori:[47,49,53],potenti:[0,2,3,9,11,13,14,23,25,47,49,53,55],power:[0,2,3],pp:55,pract:55,practic:[2,3,4,25,47,49,53,55],practition:[2,3],pre:[6,25,58],precalibr:55,precipit:55,precis:[2,3,55],predict:[2,3,35,55,58],prefac:[],prefer:[2,3,4,25,49,52,53],preliminari:55,premis:[2,3],prescrib:3,presenc:0,present:[0,1,2,3,4,5,6,9,14,23,25,52,53,54,55],press:55,preval:[2,3],previou:35,previous:[7,25],price:[13,14,25],prieto:55,primari:0,primarili:[2,3,4,8,14,25],primer:55,prior:[0,35,49,55,58],priori:[14,25],priorit:[6,7,25,49,53],probab:55,probabilist:[0,35,55],probabl:[0,35,55,58],problem:[6,9,14,25,55],proc:55,procedur:[2,3,14,25],process:[0,2,3,5,6,8,9,13,14,25,52,53,55],processor:55,produc:[2,3,4,5,7,9,14,23,25],prog:55,program:[52,53,57],progress:[2,3,54],proj:55,project:[0,2,3,35,54,55],promot:0,propag:[1,3],properli:[4,25],properti:[0,5,11,14,25,47,49,53,55],propheci:55,propos:[2,3,4,6,9,14,25],protect:55,proven:[4,23,25],provid:[0,1,2,3,4,13,14,22,25,54,55],proxi:[47,49,53],prudent:[23,25],pruett:55,pseudo:[11,14,25],psychol:55,publish:55,pure:[2,3],purpos:[2,3,5,13,14,23,25,54],put:0,python:[52,53],q:55,qualit:[2,3],qualiti:[2,3],quantif:[0,47,49,52,55,58],quantifi:[1,3,35,49,53],quantit:[0,2,3,11,14,25,55],quantiti:[35,47,49,53],quasi:[11,14,25,55],quest:55,question:[0,1,2,3,5,14,23,25,47,49,53,54],quinn:55,quixot:55,r:[12,14,25,52,53,55],racin:55,radar:55,rafteri:55,ragulina:55,rahman:55,rainfal:55,rais:[1,3],rakovec:[4,25,55],ramallo:55,ramo:55,random:[11,14,25,55],randomli:[10,12,14,25,47,49,53],rang:[0,2,3,4,6,9,14,25],rank:[6,25],rapid:0,rapidli:[0,49,53],rare:[13,14,25],rasch:55,rasmussen:55,rate:55,rather:[2,3,7,25,49,53],ratio:[47,49,53],raton:55,ratto:55,raymond:55,re:55,reaction:55,reader:[2,3,6,12,13,14,23,25,54],readili:[14,25],real:[0,5,7,14,25],realiti:[2,3,55],realiz:55,realli:55,reason:[2,3,5,8,9,13,14,25],recal:[23,25],recent:[9,14,25,55],recherch:55,recogn:[2,3],recommend:54,reconcil:55,reconfigur:55,record:[13,14,25],reduc:[1,2,3,5,6,9,14,25,55],reduct:[6,25,47,49,53,55],reed:55,refer:[0,4,6,9,12,14,25,35,47,49,53,58],refin:[5,25],reflect:[0,2,3,7,14,23,25],regard:[23,25],regardless:[5,25,47,49,53],region:[6,7,58],regress:[23,55,58],reitan:55,rel:[0,2,3,5,25,47,49,53,54,55],relat:[0,2,3,14,25,54,55],relationship:[0,1,3,4,14,23,25],relativist:[2,3,7,25],relax:55,relev:[0,2,3,5,7,25],reli:[47,49,53],reliab:55,reliabl:55,remain:[6,25],remark:0,rep:55,repeat:[8,14,25],repetit:[12,14,25],replac:[47,49,53],report:[0,14,25,55],repres:[0,2,3,5,6,13,14,25,47,49,53,55],represent:[0,1,2,3,5,7,25,35],requir:[0,1,2,3,23,25,35,47,49,52,53],research:[0,5,14,23,25,55],reservoir:55,resili:[0,1,2,3,54,55],resolut:[0,2,3,14,25,55],resour:55,resourc:[5,25,55],respect:57,respond:[0,55],respons:[0,1,3,4,7,14,25,55],ress:55,rest:[14,25],restor:55,result:[0,1,2,3,4,5,6,9,10,11,14,25,47,49,53,55],retreat:55,rev:55,reveal:[4,5,25],revel:55,review:[0,13,14,25,49,53,54,55],riahi:55,rich:[1,3,13,14,25],ridder:55,right:[2,3,7,25],rigor:[0,54],risa:55,risbei:55,rise:[2,3,55],risk:[0,6,25,55,58],river:55,riverin:55,road:55,robert:55,robinson:55,robust:55,rocki:55,rohmer:55,root:55,rose:55,rosenbluth:55,roson:55,rotman:55,rougier:55,rovenskaya:55,roventini:55,rozenberg:55,rspa:55,rst:[],rubin:[47,49,53,55],ruckert:55,ruddel:55,run:[1,3,6,7,9,14,25,47,49,52,53],runoff:55,runtest:61,ryan:55,s00382:55,s00477:55,s00521:55,s0304:55,s0378:55,s10463:55,s10584:55,s1064827501387826:55,s11222:55,s40070:55,s41467:55,s41598:55,s43017:55,s:[0,1,2,3,4,5,6,7,8,14,15,22,25,52,53,55],s_i:[15,22,25,57],sa:[5,25],sack:55,saf:55,safe:55,safeti:55,saisana:55,salienc:[2,3,55],sallaberri:55,salmond:55,saltelli:[0,1,2,3,4,6,8,9,10,12,14,23,25,55,57],same:[2,3,5,9,10,11,14,23,25,52,53],samo:55,sampl:[1,3,8,11,15,22,23,55,58],sampler:55,san:55,sanderson:55,sarewitz:55,savag:55,scale:[0,2,3,14,15,22,23,25,47,49,53,54,55],scenario:[0,2,3,5,13,14,35,55,58],schaibli:55,scheme:[8,9,10,14,25],schepdael:[14,25,55],schlather:55,schlesing:55,schwarz:55,sci:55,sciadv:55,scienc:[0,2,3,47,49,53,55],scientif:[0,2,3,14,25,54,55],scope:[0,2,3,13,14,25,49,53],scotland:55,scott:55,screen:[6,25],script:[0,54],sdr:55,se:[2,3],sea:55,search:58,second:[1,2,3,23,25],section:[0,2,3,6,7,13,22,23,35,49,53,54],sector:[54,55],secur:55,see:[52,53,54],seed:[14,25],seek:[0,2,3,54],seen:[2,3],select:[2,3,8,12,14,23,25,49,54,55,58],sennevil:55,sensit:[0,1,2,3,8,10,12,13,14,15,49,53,54,55,57,58],sensor:[9,14,25],seq:55,sequenc:[8,55,58],sequenti:[52,53,55],ser:55,serafin:55,seri:[55,58],serv:55,servic:55,sesmo:55,set:[0,1,2,3,4,5,6,14,23,25,35,47,49,53,55],sever:[2,3,4,12,13,14,23,25,47,49,53],shaffer:55,shafii:55,shahidehpour:55,shalizi:55,shape:[0,5,6,14,25,54,55],sharp:55,sheet:55,shelf:55,sherwen:55,shewbridg:55,shibata:55,shift:[2,3],shock:0,shortag:55,should:[0,2,3,4,6,14,23,25,47,49,53,54],show:[6,25],showcas:[0,54],shown:[0,1,3],shrader:55,shuler:55,si_67_8:55,siam:55,siamasa:55,siblei:55,side:[2,3],sign:[47,49,53],signal:55,signatur:[2,3],signific:[0,2,3,5,6,25,55],significantli:[5,9,14,25],silva:55,sim_i:57,similar:[2,3,8,14,25],simpl:55,simplic:[15,22,25],simplif:[1,3,5,25],simplifi:[2,3,5,23,25],simpson:55,simul:[1,2,3,9,11,14,25,55],sin:55,singl:[2,3,15,22,25,55],singular:[0,55],site:55,situat:[0,2,3,6,25],size:[11,23,47,49,53],skew:[47,49,53],skill:[47,49,53],slope:55,slowli:55,sluij:55,small:[4,5,11,14,25,47,49,53,55],smallest:[6,25],smart:55,sminchisescu:55,smith:55,so:[2,3,5,6,12,14,25,47,49,53,55],sobol:[11,12,14,23,25,55,57],soc:55,societi:55,socio:55,sociol:55,softw:55,softwar:58,solut:57,solv:55,some:[0,5,8,14,22,25,47,49,53],someth:[11,14,25],son:55,song:55,sophist:[6,8,14,25],sorooshian:55,sourc:[0,1,3,4,25,57],space:[0,1,2,3,4,6,8,9,10,11,12,14,25,35,47,49,53,55],span:[0,2,3],spano:55,spatial:[2,3,13,14,25,55],spear:[6,25,55],specif:[0,1,2,3,4,5,6,12,14,25,35,52,53,54,55],specifi:[0,55],spectral:55,spent:[47,49,53],spiegelhalt:55,springer:55,springett:55,srikrishnan:55,sriver:55,ss:55,stabl:[5,23,25,55],stai:[2,3],stakehold:[2,3,7,25],standard:[47,49,53],stano:55,stark:55,start:[12,14,25],stat:55,statap:55,state:[0,2,3,7,14,25,55],statement:[23,25],stationar:[13,14,25,55],stationari:[47,49,53,55],statist:[0,1,2,3,11,13,14,25,55,58],statistician:[47,49,53],statistiqu:55,statm:55,steding:[13,14,25,55],stefan:55,steinschneid:[13,14,25,55],stem:[2,3],step:[1,3],stephenson:55,stern:55,still:[2,3,55],stoch:55,stochast:[13,14,25,55],stone:55,storli:55,storm:55,stouffer:55,strategi:[9,12,14,25,55],stratifi:[14,25],streamflow:[13,14,25,55],strength:[47,49,53],stress:[47,49,53],stressor:[0,2,3,13,14,25,35,54],strict:[7,25],strong:[0,13,14,25,55],strongli:54,structur:[0,1,2,3,5,9,14,25,35,49,53,55],sts257:55,studi:[0,4,5,13,14,25,47,49,53,55],style:[11,14,25],styliz:[6,25],su5030955:55,subinterv:[10,14,25],subject:[2,3,55],submit:55,submodul:59,subpackag:59,subroutin:[52,53],subsequ:[6,7,10,12,14,23,25],subset:[52,53,55],subspac:[11,14,25],substanti:[14,25,35],suckal:55,sudjianto:55,suffici:[0,2,3,5,7,13,14,25,47,49,53,55],suggest:[6,25,54],suit:[13,14,25,52,53],suitabl:[47,49,53],sum:60,sum_int:[60,61],summar:[14,25],summari:[0,55],suppli:55,support:[0,2,3,54,55],sure:[57,61],surfac:55,surg:55,surowiec:[9,14,25,55],surrog:[55,58],surround:[5,25,55],survei:[0,55],sustain:55,svahn:55,syke:55,synthes:0,synthesi:[0,23,25],synthet:[55,58],syst:55,system:[0,1,2,3,5,7,8,14,23,25,54,55,58],systemat:[47,49,53,55],t:55,tabl:[14,25],tac:55,tail:[47,49,55,58],take:[0,11,14,25],tang:[4,10,14,25,55],tarantola:[6,25,55],target:[47,49,53,55,62],task:0,team:[0,9,14,25,54],tech:55,techfor:55,technic:[0,2,3,55],techniqu:[3,4,5,6,8,10,11,12,14,23,25,54,55],technol:55,technolog:0,technometr:55,tell:55,teller:55,tellu:55,tellusa:55,temperatur:[7,9,14,25,55],tempor:[2,3,13,14,25],tend:[7,25],tenuou:[2,3],term:[0,2,3,4,7,25],terminolog:54,test:[0,1,2,3,5,9,12,14,47,49,53,54,55,58,59,60],test_model:[59,60],test_sum_int:61,testcas:61,testmodel:61,teul:55,text:[0,2,3,4,5,13,14,25,54],th:[15,22,25],than:[0,2,3,7,9,10,14,25,47,49,53],theater:55,thei:[0,2,3,4,5,9,13,14,23,25,47,49,53],them:[5,6,7,14,25],themselv:[1,3],theoret:[0,55],theori:[0,2,3,55],therefor:[2,3,4,5,6,14,25,47,49,53],thermoclin:55,thi:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,22,23,35,47,49,53,54,57],thieri:55,think:[49,53,55],third:55,thompson:55,thomson:55,thorough:54,those:[0,1,2,3,4,25,54],though:[2,3,11,14,25,47,49,53],three:[6,9,14,25,55],threshold:[0,6,25,47,49,53,55],through:[1,2,3,10,11,14,25,54],throughout:0,thu:[47,49,53],tian:[35,55],time:[12,23,47,49,53,55,58],tissu:55,tit:55,togeth:[0,1,3],tol:55,toler:[6,25],tolson:55,tommasi:55,tool:[0,1,2,3,6,13,14,25,54,58],toolkit:58,top:[7,25],topic:[0,13,14,25],toronto:55,total:[6,9,14,23,25],toulmin:[2,3,55],toward:[2,3,55],tpami:55,trace:[6,25],track:57,tradit:[0,2,3],tradition:[9,14,25],traffic:55,trait:58,trajectori:[12,14,25],tran:55,transdisciplinari:[0,54,55],transfer:55,transit:[0,2,3],translat:0,transp:55,transport:55,travi:55,trayanov:55,treat:[9,13,14,25,49,53],treatment:[0,2,3,4,14,25],treich:55,tremblai:55,trenner:55,trivial:[0,54],tropospher:55,trust:55,truth:[2,3],trygg:55,tsai:55,tsunami:55,tune:[47,49,53],turn:[14,23,25,55],two:[1,2,3,4,5,9,14,25,47,49,52,53,55,60],type:[0,2,3,6,55,58],typic:[0,9,14,23,25,47,49,52,53],typolog:55,u:55,uc:[0,1,3,35,54],uijlenhoet:55,uk:55,ulam:[14,25,55],ullah:55,ulph:55,ultim:[8,14,25,54],unaccept:[7,25],unaid:55,uncertain:[0,1,2,3,4,5,6,8,9,10,12,14,23,25,35,49,53,55],uncertainti:[0,1,2,3,4,5,6,9,13,14,25,47,49,52,54,55],under:[0,2,3,7,14,25,54,55,58],underestim:[4,25,55],underli:[0,2,3],underlin:[15,22,25],undermin:[2,3],understand:[0,2,3,5,23,25,54,55,58],understood:[0,55],undesir:[2,3,5,6,7,25],uniform:[9,11,14,25,55],uniformli:[11,12,14,25,55],unimport:[5,25],union:55,uniqu:[2,3],unit:[0,55],unittest:61,univers:[47,49,53,55],unnecessari:[6,25],unpreced:[2,3],up:[2,3,14,25,55],upon:[52,53],upper:55,uq:[0,14,25,35,52,53,54],urban:[0,55],url:55,us:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,25,35,47,49,52,53,54,55],usa:55,user:[52,53],ussr:55,usual:[0,6,12,14,25],util:[0,2,3,11,14,25],v10:55,v29i4:55,v:[6,25,55,57],v_:57,va:55,vafeidi:55,valeur:55,valid:[2,3,4,25,55],valu:[0,2,3,4,5,6,7,9,14,15,22,23,25,47,49,53,55,57,61],valuabl:[13,14,25],van:[14,25,55],vari:[0,4,6,14,23,25,47,49,53,55,57],variabl:[2,3,4,6,9,10,11,14,25,55],varianc:[6,11,14,47,49,53,55,57,58],variat:[12,14,25],varieti:[2,3,6,12,13,14,25],variou:[0,1,2,3,11,14,25,47,49,53,54,55],vast:[2,3],vavru:55,vector:[15,22,25,55],vega:55,vehtari:55,veri:[2,3,10,13,14,25,54],verif:[2,3,55],version:55,versu:[6,13,14,58],vertenstein:55,vertic:0,via:55,view:[0,2,3,7,25,54],viewpoint:[2,3],vignotto:55,vihola:55,visual:55,vivek:55,vogel:[13,14,25,55],von:55,voukouvala:55,vousdouka:55,vroman:55,vrugt:55,vs:[2,3],vulner:[0,1,2,3,54,55],vuuren:55,w:[0,55],wa:[14,25],wagen:55,wai:[2,3,5,6,14,25,55],waic:55,walker:[0,35,55],wallach:55,warm:55,warrant:[5,25],wasec:55,washington:55,wast:55,wat2:55,watanab:55,water:[0,55,57],watersh:55,wave:0,wcc:55,we:[0,1,2,3,4,5,22,25,49,52,53,54,55,62],weak:[47,49,53],wear:55,weather:[13,14,25,55],weaver:[2,3,54,55],webster:55,wedderburn:55,weert:55,welch:55,well:[0,2,3,5,6,13,14,22,25,35,47,49,53,54,55],were:[2,3,14,25],werkhoven:55,wesselink:55,west:55,westhoff:55,westra:55,what:[1,3,23,25,58],when:[0,1,2,3,5,6,7,9,10,11,13,14,23,25,35,47,49,53,54,55],whenev:[4,25],where:[0,1,2,3,4,9,12,14,15,22,23,25,35,54],wherea:[2,3],whether:[2,3],which:[0,1,2,3,4,5,6,7,8,11,14,23,25,35,47,49,53],white:[2,3,55],whither:55,whitman:55,why:[1,3,55,58],wi:55,wicked:55,wide:[0,2,3,4,9,11,14,22,25,55],wiener:55,wikman:55,wilbi:[13,14,25,55],wildl:55,wilei:55,wilk:[13,14,25,55],willem:55,wilson:55,wire:55,wirtz:[0,55],within:[0,2,3,4,6,9,11,14,25,47,49,52,53,55],without:[4,10,14,25,35,52,53],wolff:55,wong:55,wood:55,word:[4,9,14,23,25],work:[54,55],workflow:[0,54,55],world:[0,5,14,25],would:[0,6,7,8,14,23,25],wr011i005p00681:55,wr024i009p01483:55,wr99107:55,wr:55,wrapper:[52,53],wrcr:55,wrong:[2,3,55],wu:55,wutich:55,wynn:55,x1:[4,6,9,14,25],x2:[4,6,9,14,25],x3:[6,9,14,25],x:[4,6,14,15,22,25,55,57],x_i:57,xi:[6,25],xian:55,xiu:55,xn:[4,25],xx:[47,49,53],y1:[4,25],y2:[4,25],y:[4,6,14,25,55,57],yapo:55,ye:[14,25,55],year:[13,14,25],yet:[47,49,53],yield:[1,3,5,25,35],ylvisak:55,ym:[4,25],york:55,you:[1,3],your:[5,25,57],z:55,zare:55,zarekarizi:55,zaremba:[11,14,25,55],zeff:55,zellner:[35,55],zhang:55,zhao:55,zscheischler:55},titles:["Introduction","Overview of model diagnostics","Perspectives on diagnostic model evaluation","Diagnostic Modeling Overview and Perspectives","Global Versus Local Sensitivity","Why Perform Sensitivity Analysis","Sensitivity Analysis Applications for Model Evaluation and Fidelity Testing","Sensitivity Analysis Applications for Exploratory Modeling and Scenario Discovery","One-At-a-Time (OAT)","Full and Fractional Factorial Sampling","Latin Hypercube (LH) Sampling","Low-Discrepancy Sequences","Other types of sampling","Synthetic generation of input time series","Design of Experiments","Derivative-based Methods","Elementary Effect Methods","Regression-based Methods","Regional Sensitivity Analysis","Variance-based Methods","Analysis of Variance (ANOVA)","Moment-Independent (Density-Based) Methods","Sensitivity Analysis Methods","How To Choose A Sensitivity Analysis Method: Model Traits And Dimensionality","Software Toolkits","Sensitivity Analysis: The Basics","Understanding Errors: What Is Controlling Model Performance?","Consequential Dynamics: What is Controlling Model Behaviors of Interest?","Consequential Scenarios: What is Controlling Consequential Outcomes?","Sensitivity Analysis: Diagnostic & Exploratory Modeling","Why is Uncertainty Quantification Important for Understanding MultiSector System Dynamics?","Uncertainty Quantification for Exploratory Modeling","Bayesian Uncertainty Quantification","Uncertainty Quantification Under (Deep) Uncertainty","Integrating Model Diagnostics and Uncertainty Quantification","Uncertainty Quantification: The Basics","Understanding Risk: How Probable Are Extreme Events?","Understanding Tails: Statistical Modeling of Extreme Events","How to Choose an Appropriate Method?","How to Select a Prior Distribution?","Posterior Predictive Checking","Model Selection and Comparison","Scenario Discovery","Pre-calibration/GLUE","Metropolis-Hastings","Gibbs Sampling","Hamiltonian Monte Carlo","Markov Chain Monte Carlo","Particle-based Methods","What are Common Methods?","Markov Chain Monte Carlo with the True Model","Markov Chain Monte Carlo with Surrogate Models","What are Example Software Implementations?","Uncertainty Quantification: A Tool For Capturing Risks & Extremes","Conclusion","References","Glossary","Examples","Addressing Uncertainty in MultiSector Dynamics Research","nanites","nanites package","nanites.tests package","Preface"],titleterms:{"import":[30,35],"true":[50,52,53],A:[23,25,53],And:[23,25],At:[8,14,25],For:53,Is:[26,29],One:[8,14,25],The:[25,35],To:[23,25],acknowledg:62,address:58,an:[38,53],analysi:[5,6,7,18,20,22,23,25,29],anova:[20,22,25],applic:[6,7,25],appropri:[38,53],ar:[36,49,52,53],audienc:62,base:[15,17,19,21,22,25,48,49,53],basic:[25,35],bayesian:[32,35],behavior:[27,29],book:62,build:57,calibr:[43,49,53],caption:57,captur:53,carlo:[46,47,49,50,51,52,53],chain:[47,49,50,51,52,53],check:[40,53],choos:[23,25,38,53],codeblock:57,common:[49,53],comparison:[41,53],conclus:54,consequenti:[27,28,29],content:[60,61],control:[26,27,28,29],deep:[33,35],densiti:[21,22,25],depend:[14,25],deriv:[15,22,25],design:[14,25],diagnost:[1,2,3,29,34,35],dimension:[23,25],discoveri:[7,25,42,49,53],discrep:[11,14,25],distribut:[39,53],dynam:[27,29,30,35,58],ebook:[],effect:[16,22,25],elementari:[16,22,25],error:[26,29],evalu:[2,3,6,25],event:[36,37,53],exampl:[52,53,57],experi:[14,25],exploratori:[7,25,29,31,35],extrem:[36,37,53],factori:[9,14,25],fidel:[6,25],figur:57,fraction:[9,14,25],full:[9,14,25],gener:[13,14,25],gibb:[45,47,49,53],global:[4,25],glossari:56,glue:[43,49,53],hamiltonian:[46,47,49,53],hast:[44,47,49,53],how:[23,25,36,38,39,53,62],hypercub:[10,14,25],implement:[52,53],independ:[21,22,25],indic:58,input:[13,14,25],insert:57,instal:62,integr:[34,35],interest:[27,29],introduct:0,latin:[10,14,25],lh:[10,14,25],local:[4,25],low:[11,14,25],markov:[47,49,50,51,52,53],mathjax:57,method:[15,16,17,19,21,22,23,25,38,48,49,53],metropoli:[44,47,49,53],model:[1,2,3,6,7,23,25,26,27,29,31,34,35,37,41,50,51,52,53,60],modul:[60,61],moment:[21,22,25],mont:[46,47,49,50,51,52,53],msd_uncertainty_ebook:62,multisector:[30,35,58],nanit:[59,60,61],note:57,oat:[8,14,25],other:[12,14,25],out:57,outcom:[28,29],overview:[1,3,14,25],packag:[60,61,62],particl:[48,49,53],perform:[5,25,26,29],perspect:[2,3],posterior:[40,53],pre:[43,49,53],predict:[40,53],prefac:62,prior:[39,53],probabl:[36,53],python:62,quantif:[30,31,32,33,34,35,53],refer:55,region:[18,22,25],regress:[17,22,25],research:58,risk:[36,53],sampl:[9,10,12,14,25,45,47,49,53],scenario:[7,25,28,29,42,49,53],section:[14,25],select:[39,41,53],sensit:[4,5,6,7,18,22,23,25,29],sequenc:[11,14,25],seri:[13,14,25],size:[14,25],softwar:[24,25,52,53],statist:[37,53],submodul:[60,61],subpackag:60,summari:[14,25],surrog:[51,52,53],synthet:[13,14,25],system:[30,35],tabl:58,tail:[37,53],test:[6,25,57,61],test_model:61,thi:[14,25,62],time:[8,13,14,25],tool:53,toolkit:[24,25],trait:[23,25],type:[12,14,25],uncertainti:[30,31,32,33,34,35,53,58],under:[33,35],understand:[26,29,30,35,36,37,53],us:[57,62],varianc:[19,20,22,25],versu:[4,25],what:[26,27,28,29,49,52,53],why:[5,25,30,35]}}) \ No newline at end of file +Search.setIndex({docnames:["1_introduction","2.1_overview_of_model_diagnostics","2.2_perspectives_on_diagnostic_model_evaluation","2_diagnostic_modeling_overview_and_perspectives","3.1_global_versus_local_sensitivity","3.2_why_perform_sensitivity_analysis","3.3_sensitivity_analysis_applications_for_model_evaluation_and_fidelity_testing","3.4_sensitivity_analysis_applications_for_exploratory_modeling_and_scenario_discovery","3.5.1_one_at_a_time_oat","3.5.2_full_fractional_factorial_sampling","3.5.3_latin_hypercube_sampling","3.5.4_low_discrepancy_sequences","3.5.5_other_types_of_sampling","3.5.6_synthetic_generation_of_input_time_series","3.5_design_of_experiments","3.6.1_derivative_based_methods","3.6.2_elementary_effect_methods","3.6.3_regression_based_methods","3.6.4_regional_sensitivity_analysis","3.6.5_variance_based_methods","3.6.6_analysis_of_variance_anova","3.6.7_moment_independent_density_based_methods","3.6_sensitivity_analysis_methods","3.7_how_to_choose_a_sensitivity_analysis_method_model_traits_and_dimensionality","3.8_software_toolkits","3_sensitivity_analysis_the_basics","4.1_understanding_errors_what_is_controlling_model_performance","4.2_consequential_dynamics_what_is_controlling_model_behaviors_of_interest","4.3_consequential_scenarios_what_is_controlling_consequential_outcomes","4_sensitivity_analysis_diagnostic_and_exploratory_modeling","5.1_why_is_uncertainty_quantification_important_for_understanding_multisector_system_dynamics","5.2_uncertainty_quantification_for_exploratory_modeling","5.3_bayesian_uncertainty_quantification","5.4_uncertainty_quantification_under_deep_uncertainty","5.5_integrating_model_diagnostics_and_uncertainty_quantification","5_uncertainty_quantification_the_basics","6.1_understanding_risk_how_probable_are_extreme_events","6.2_understanding_tails_statistical_modeling_of_extreme_events","6.3_how_to_choose_an_appropriate_method","6.4_how_to_select_a_prior_distribution","6.5_posterior_predictive_checking","6.6_model_selection_and_comparison","6.7.1_scenario_discovery","6.7.2_pre_calibration_glue","6.7.3.1_metropolis_hastings","6.7.3.2_gibbs_sampling","6.7.3.3_hamiltonian_monte_carlo","6.7.3_markov_chain_monte_carlo","6.7.4_particle_based_methods","6.7_what_are_common_methods","6.8.1_markov_chain_monte_carlo_with_the_true_model","6.8.2_markov_chain_monte_carlo_with_surrogate_models","6.8_what_are_example_software_implementations","6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes","7_conclusion","8_glossary","examples","index","modules","nanites","nanites.tests","preface"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":3,"sphinx.domains.rst":2,"sphinx.domains.std":2,"sphinxcontrib.bibtex":7,sphinx:56},filenames:["1_introduction.rst","2.1_overview_of_model_diagnostics.rst","2.2_perspectives_on_diagnostic_model_evaluation.rst","2_diagnostic_modeling_overview_and_perspectives.rst","3.1_global_versus_local_sensitivity.rst","3.2_why_perform_sensitivity_analysis.rst","3.3_sensitivity_analysis_applications_for_model_evaluation_and_fidelity_testing.rst","3.4_sensitivity_analysis_applications_for_exploratory_modeling_and_scenario_discovery.rst","3.5.1_one_at_a_time_oat.rst","3.5.2_full_fractional_factorial_sampling.rst","3.5.3_latin_hypercube_sampling.rst","3.5.4_low_discrepancy_sequences.rst","3.5.5_other_types_of_sampling.rst","3.5.6_synthetic_generation_of_input_time_series.rst","3.5_design_of_experiments.rst","3.6.1_derivative_based_methods.rst","3.6.2_elementary_effect_methods.rst","3.6.3_regression_based_methods.rst","3.6.4_regional_sensitivity_analysis.rst","3.6.5_variance_based_methods.rst","3.6.6_analysis_of_variance_anova.rst","3.6.7_moment_independent_density_based_methods.rst","3.6_sensitivity_analysis_methods.rst","3.7_how_to_choose_a_sensitivity_analysis_method_model_traits_and_dimensionality.rst","3.8_software_toolkits.rst","3_sensitivity_analysis_the_basics.rst","4.1_understanding_errors_what_is_controlling_model_performance.rst","4.2_consequential_dynamics_what_is_controlling_model_behaviors_of_interest.rst","4.3_consequential_scenarios_what_is_controlling_consequential_outcomes.rst","4_sensitivity_analysis_diagnostic_and_exploratory_modeling.rst","5.1_why_is_uncertainty_quantification_important_for_understanding_multisector_system_dynamics.rst","5.2_uncertainty_quantification_for_exploratory_modeling.rst","5.3_bayesian_uncertainty_quantification.rst","5.4_uncertainty_quantification_under_deep_uncertainty.rst","5.5_integrating_model_diagnostics_and_uncertainty_quantification.rst","5_uncertainty_quantification_the_basics.rst","6.1_understanding_risk_how_probable_are_extreme_events.rst","6.2_understanding_tails_statistical_modeling_of_extreme_events.rst","6.3_how_to_choose_an_appropriate_method.rst","6.4_how_to_select_a_prior_distribution.rst","6.5_posterior_predictive_checking.rst","6.6_model_selection_and_comparison.rst","6.7.1_scenario_discovery.rst","6.7.2_pre_calibration_glue.rst","6.7.3.1_metropolis_hastings.rst","6.7.3.2_gibbs_sampling.rst","6.7.3.3_hamiltonian_monte_carlo.rst","6.7.3_markov_chain_monte_carlo.rst","6.7.4_particle_based_methods.rst","6.7_what_are_common_methods.rst","6.8.1_markov_chain_monte_carlo_with_the_true_model.rst","6.8.2_markov_chain_monte_carlo_with_surrogate_models.rst","6.8_what_are_example_software_implementations.rst","6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes.rst","7_conclusion.rst","8_glossary.rst","examples.rst","index.rst","modules.rst","nanites.rst","nanites.tests.rst","preface.rst"],objects:{"":{nanites:[59,0,0,"-"]},"nanites.model":{sum_ints:[59,1,1,""]},"nanites.tests":{test_model:[60,0,0,"-"]},"nanites.tests.test_model":{TestModel:[60,2,1,""]},"nanites.tests.test_model.TestModel":{test_sum_ints:[60,3,1,""]},nanites:{model:[59,0,0,"-"],tests:[60,0,0,"-"]}},objnames:{"0":["py","module","Python module"],"1":["py","function","Python function"],"2":["py","class","Python class"],"3":["py","method","Python method"]},objtypes:{"0":"py:module","1":"py:function","2":"py:class","3":"py:method"},terms:{"0":54,"05":[47,49,53],"1":[0,2,3,12,14,25,47,49,53,54,56,57],"10":[0,57],"1000":[23,25],"102186":57,"11":[0,57],"114":57,"12":[0,57],"16226":57,"17":57,"18":54,"180":57,"192":57,"1949":[14,25],"1960":[2,3,14,25],"1961":[9,14,25],"1964":35,"1967":[2,3,11,14,25],"1968":[11,14,25],"1973":[12,14,25],"1976":[11,14,25],"1977":[2,3,35],"1979":[10,14,25],"1980":[6,25],"1988":[13,14,25],"1990":[2,3],"1991":[12,14,25,57],"1992":[2,3,47,49,53],"1993":[2,3,5,10,14,25],"1994":[2,3,11,14,25],"1995":[11,14,25],"1996":[2,3],"1997":57,"1998":[2,3],"1999":[12,13,14,25],"1km":0,"2":[0,1,3,23,25,54,57],"2001":56,"2002":[2,3,6,12,14,25],"2002a":[23,25,56],"2003":[2,3,57],"2004":[4,6,25],"2006":[4,14,25],"2007":[4,25],"2008":[1,2,3,6,9,10,11,13,14,15,22,25,47,49,53],"2010":[2,3,4,5,6,8,11,14,25],"2011":[2,3,6,23,25,47,49,53],"2012":[2,3],"2013":[2,3,13,14,25,35,54,57],"2014":[2,3,4,6,14,25,57],"2015":[11,13,14,25,57],"2016":[4,6,12,13,14,15,22,25,57],"2017":[9,13,14,25,57],"2018":[2,3,13,14,25,57],"2019":[1,3,4,13,14,25,35,57],"2020":[2,3,57],"2020b":35,"25":[7,25],"29":57,"2d":[23,25],"2k":[9,14,25],"3":[0,4,7,25,57],"31":57,"32":[9,14,25],"38":57,"39":57,"4":[0,2,3,6,25,57],"43":[9,14,25],"48":57,"497":57,"5":[0,23,25,57],"51":57,"59":57,"6":[0,9,10,14,25,57],"64":[9,14,25],"65":57,"7":[0,57],"7447":57,"759":57,"760":57,"79":57,"8":[0,23,25,57],"84":57,"9":[0,57],"93":57,"98":57,"\u00eb":57,"abstract":[2,3,54],"case":[0,2,3,4,6,9,14,23,25,47,49,53,54,60],"class":60,"do":[0,1,3,23,25,47,49,53],"final":[2,3,6,25],"function":[0,2,3,14,25,35,54,59],"import":[0,2,3,4,5,8,9,14,23,25,56,57],"int":59,"long":[0,47,49,53],"new":[0,2,3,5,25],"public":[2,3,13,14,25,57],"return":[15,22,25,59,60],"short":0,"transient":[47,49,53],"true":[2,3,6,25,57],"while":[0,5,6,8,10,12,13,14,25,47,49,53],A:[2,3,9,11,13,14,57],And:57,As:[0,1,2,3,6,7,9,14,25,35,47,49,53,54],At:[0,12,57],Be:56,For:[2,3,4,5,6,7,9,12,13,14,15,22,23,25,47,49,52,57],If:[2,3,4,9,14,25],In:[2,3,4,5,6,8,9,10,14,22,23,25,35,49,53,54],Is:57,It:[0,5,8,13,14,25,47,49,53],Its:[14,25],Nearing:[2,3],No:[14,25],On:[2,3,5,25],One:[5,12,47,49,53,57],Such:[23,25],The:[0,1,2,3,6,7,8,9,10,11,12,13,14,23,47,49,52,53,54,56,57],There:[0,2,3,12,14,25,49,52,53],These:[0,2,3,11,14,15,22,25],To:[10,14,57],With:[10,14,25],_:[15,22,25],_i:[15,22,25],_static:56,abil:[0,2,3],abl:[47,49,53],about:[5,6,9,14,25,35,47,49,53,61],abov:[0,7,13,14,25],absenc:0,academi:57,accept:[2,3],account:[4,9,14,25],accur:[0,5,7,25],accuraci:[2,3,7,25],achiev:[2,3],acknowledg:0,across:[0,2,3,8,9,10,14,25,47,49,53,54],act:35,activ:[14,25],ad:[23,25],adapt:[1,2,3],add:[54,56],addit:[0,2,3,4,5,7,14,25,54],address:[0,1,2,3,10,14,23,25,54],advanc:[0,2,3,5,23,25,54],advantag:[11,14,25,49,53],affect:[0,14,15,22,25],afford:[9,14,23,25],aforement:[2,3],agenc:[2,3],agre:0,agreement:[2,3],agricultur:[2,3],aid:0,aim:[0,2,3,6,25],al:[1,2,3,4,6,9,10,11,12,13,14,15,22,23,25,35,47,49,53,54],albert:57,aleksankina:57,algorithm:[47,49,53],all:[0,6,7,8,9,10,12,14,25,35,47,49,53],allow:[0,1,3,9,14,25],alon:56,along:[12,14,22,25],alreadi:[2,3],also:[0,2,3,5,6,9,10,12,14,22,23,25,54],altern:[0,1,2,3,4,5,9,14,25,35,54],although:54,alwai:[2,3],ambiti:[0,54],among:[2,3],amount:[12,14,25,47,49,53],amplitud:[12,14,25],an:[0,1,2,3,4,5,9,10,14,25,52,54,57],analys:[0,1,2,3,4,6,23,25,49,53,54,57],analysi:[0,1,2,3,4,10,12,13,14,49,53,54,57],analyst:[7,9,14,23,25],analyt:[14,25],analyz:[8,14,25,57],anderson:[6,25],andrea:57,ani:[0,1,2,3,4,5,10,11,14,23,25,59],annoni:[4,8,14,25],anoth:[11,14,25,47,49,53],anova:57,answer:[0,1,3,14,25],anthoni:57,appear:[2,3,5,25],appendic:[0,54],appli:[4,5,6,8,9,14,22,23,25],applic:[0,2,3,4,5,12,14,23,54,57],apport:[4,25],approach:[0,2,3,4,8,11,12,13,14,25,47,49,52,53,54,57],appropri:[2,3,4,6,9,10,14,23,25,47,49,54,57],approxim:[11,14,25,47,49,53],ar:[0,1,2,3,4,5,6,7,8,9,10,12,13,14,15,22,23,25,35,47,54,57],area:[0,47,49,53],argu:[2,3],argument:[2,3],around:[0,4,15,22,25],arrai:[10,14,25],art:[0,3,47,49,53],ask:[23,25],aspect:[2,3],asselt:57,assess:[2,3,5,14,23,25,47,49,52,53,57],associ:[49,53],assum:[8,9,14,15,22,25],assumpt:[1,2,3,5,7,14,23,25,35],asymptot:[47,49,53],athanasiadi:57,attain:[0,2,3,11,14,25],attract:[10,14,25],author:[2,3],autocorrel:[47,49,53],avail:[0,2,3,13,14,25],averag:[49,53],awar:[6,23,25],axi:0,axtel:57,b:[4,6,9,14,25,57,59],ba:57,back:[6,25],bank:[2,3,5,25],bar:[23,25],barla:[2,3],base:[0,2,3,4,5,6,14,23,52,54,57,60],baselin:[2,3],basi:[1,3,23,25,57],basic:[0,57],bayesian:[52,53,57],becaus:[2,3,10,14,23,25],becker:57,becom:[2,3,6,14,25],beek:[13,14,25],been:[0,1,2,3,4,5,6,7,9,12,14,25,47,49,53,54],befor:[0,14,25],begin:[0,47,49,53],behavior:[0,1,2,3,5,6,14,25,35,54,57],being:[2,3,9,10,14,23,25],below:[0,6,23,25,47,49,52,53],besid:[2,3],best:[2,3],better:[2,3,13,14,23,25,52,53,54],between:[0,1,2,3,4,9,10,11,14,23,25],beven:[2,3],beyond:[2,3,12,13,14,25,49,53,57],bhar:[13,14,25],bia:[9,14,25,47,49,53],bias:[4,10,14,25],binlei:[2,3],blog:56,bodi:[0,13,14,25],borgomeo:[13,14,25],borgonovo:[4,6,15,22,25],both:[49,53,54],bottom:[7,25],boundari:[0,2,3],box:[9,14,25],brett:57,bridg:0,briefli:0,broad:[4,25],broader:[2,3],broadli:[0,49,53,54],brown:57,bryan:57,bryant:[5,25],build:[2,3,12,14,25,52,53],built:[52,53],burden:[6,9,14,25],burdensom:[9,14,25],burn:[47,49,53],c:[4,9,10,14,15,22,25,52,53],calcul:[14,25],calibr:[2,3,5,6,25,57],call:[23,25,47,49,53],camp:[2,3],campaign:[6,25],can:[0,1,2,3,4,5,6,7,8,9,10,11,13,14,15,22,23,25,35,47,49,53,54],cancel:[2,3],candid:[9,14,25],cannot:[0,2,3,4,9,10,14,23,25],capabl:[0,54],captur:[0,13,14,25,54,57],care:0,carefulli:0,carl:57,carlo:[1,3,6,11,14,25,57],carpent:[2,3],cash:[2,3],categori:[4,5,25,47,49,53],caus:[2,3,5,6,14,25],caveat:[23,25],centenni:[2,3],central:0,centuri:[13,14,25],chain:57,challeng:[0,2,3,8,14,25,49,53,54,57],chang:[0,2,3,5,6,7,8,13,14,25,35,54,57],chapter:0,character:[0,1,3,11,13,14,25,35,54],characterist:[2,3,14,25],chart:[1,3],cheap:[23,25],check:57,choic:[0,1,2,3,14,25,35,47,49,53,54],choos:57,chosen:[4,14,25],circl:0,citat:54,cite:[13,14,25],citi:[14,25],claim:[0,2,3],clarifi:[14,25],classifi:[1,2,3,49,53],clear:[2,3,23,25],climat:[2,3,13,14,25,57],close:[2,3,14,25],cluster:[11,14,25],co:[0,2,3,35,54],code:[52,53],collect:0,color:0,combin:[0,3,9,14,25],come:[1,3,13,14,25],commod:[13,14,25],common:[0,2,3,6,10,14,25,52,57],commonli:[2,3,5,9,14,25],commun:[0,2,3],compar:[2,3,9,10,14,25,47,49,53],comparison:[13,14,22,25,35,57],compensatori:[2,3],complet:0,complex:[0,2,3,4,5,6,14,23,25,54,57],complic:[2,3],compon:[0,5,6,25],compos:[12,14,25],compound:[0,13,14,25],comprehens:[9,14,25,54],comput:[0,2,3,5,6,8,9,14,23,25,47,49,52,53,54],computation:[9,14,23,25],concept:[0,2,3,54],conceptu:[0,1,3,49,53,54,57],conclud:[0,47,49,53],conclus:[6,23,25,57],condit:[0,2,3,14,25,47,49,53,54],conduct:[2,3,4,14,25,49,53],confid:[2,3],confound:[9,14,25],confront:0,consequ:[2,3,54],consequenti:[1,2,3,5,7,13,14,25,35,54,57],consid:[0,2,3,4,5,6,7,9,13,14,23,25,35,54],consider:[4,25,52,53],consist:[0,1,3,14,25,35,54],constrain:[10,14,23,25],constraint:[14,25],construct:[47,49,53],consult:0,consum:[5,25],contain:[2,3,54],content:58,context:[2,3,5,6,13,14,25],contigu:0,continent:0,continu:[2,3,9,14,25],continuum:0,contour:[4,25],contrast:[2,3,4,25],contribut:[0,6,25,56],control:[1,3,5,10,14,25,57],conveni:[4,25],converg:[23,25,47,49,53,57],convers:[6,25],cook:57,cope:57,core:[2,3,11,14,25],correctli:[2,3],correl:[10,13,14,25],correspond:[9,14,25],cost:[5,8,9,14,25],costli:[4,25],could:[0,6,14,25,54],council:57,coupl:[0,1,2,3,14,25,54,57],cover:[0,57],coverag:[10,14,25],credenc:[5,25],credibl:[2,3],criteria:[11,14,25],criterion:[2,3],critic:[2,3,8,14,25,54],crucial:[6,25],cukier:[12,14,25],cut:[6,25],cutoff:[6,25],cycl:54,d:[9,10,14,23,25],dai:[2,3],dalal:[11,14,25],daniel:57,data:[0,1,2,3,5,14,25,35,54],de:57,deal:[9,14,25],decad:[2,3],decid:[9,14,25],decis:[0,1,2,3,5,7,25,57],decompos:[1,3],deem:[2,3,47,49,53],deep:[0,13,14,25,49,53,54,57],deepli:35,defens:[6,25],defici:[2,3],defin:[0,1,2,3,14,25,35,57],definit:[4,22,25,54],degre:[6,23,25],delib:[3,54],deliv:[23,25],delv:0,demand:[6,7,13,14,25,57],denot:[6,12,14,25,56],densiti:57,departur:[2,3],depend:[0,2,3,4,6,10,23,35,47,49,53,54],depth:54,der:57,deriv:[10,14,57],describ:[0,22,25,35,49,53],design:[0,6,9,10,11,54,57],desir:[2,3],despit:[14,25],detail:[0,5,6,12,14,22,25,54],determin:[0,47,49,53],determinist:[0,2,3],develop:[0,2,3,5,12,13,14,25,54],deviat:[11,14,25],diagnos:[2,3,47,49,53],diagnost:[0,6,25,47,49,53,54,57],dietari:[7,25],differ:[0,2,3,4,5,8,12,14,25,35,47,49,53],difficult:0,dimens:[2,3,11,14,25],dimension:[2,3,4,10,14,49,53,57],direct:[1,3,5,12,14,25],directli:[47,49,53],dirk:57,disadvantag:[14,25,49,53],discard:[47,49,53],disciplin:[0,54],disciplinari:[0,54],discontinu:[23,25],discov:[0,2,3,5,7,14,25],discoveri:[0,5,57],discredit:[5,25],discrep:57,discret:[9,10,14,25],discuss:[0,4,13,14,22,25,47,49,52,53,54],dismiss:[2,3],dispers:[47,49,53],distinguish:[0,2,3,10,14,25,54],distribut:[0,1,3,9,11,12,14,25,35,47,49,57],diverg:[2,3],divers:[0,10,14,25,54],divid:[10,14,25],doc:56,doe:[2,3,4,25],domain:[0,14,25],dominantli:[1,3],draw:[47,49,53],drawn:[5,14,25],driven:0,drought:[0,13,14,25],dubiou:[5,25],due:[4,25,47,49,53],dure:[14,25],dynam:[0,2,3,47,49,53,54],e:[0,1,2,3,4,5,6,7,8,9,11,13,14,25,52,53,54,56,57],e_:56,each:[0,2,3,4,6,8,9,10,12,14,22,23,25,49,52,53],earli:[2,3],eas:[8,14,25],ebook:56,econom:[0,57],edward:[6,25],effect:[0,2,3,4,5,6,8,9,10,14,47,49,53,56,57],effici:[14,25],effort:[5,25,52,53],eight:57,either:[1,3,5,14,25],eker:[2,3],elabor:[2,3,6,7,14,25],element:[0,14,25],elementari:57,elicit:[14,25],elimin:[6,11,14,25],elsawah:57,elsewher:[1,3],elucid:[6,25],embark:[14,25],embrac:[2,3],emerg:0,emphas:[0,2,3,54],empir:[1,2,3,4,25],empiric:[2,3],emploi:[0,2,3,11,14,25],emul:[49,52,53],enabl:[0,52,53],enayat:57,encompass:0,encourag:[13,14,25],encyclopedia:57,end:[14,25],energi:[0,2,3,7,13,14,25],engag:0,engin:[0,57],enhanc:[1,2,3],enough:[7,25],ensembl:[0,1,2,3],ensur:[11,14,25,47,49,53],entir:[0,4,14,25],environment:57,epistem:[9,14,25],equal:[2,3,10,14,25],equifin:[2,3],equival:[7,25],error:[0,2,3,7,11,14,25,47,49,53,57],escap:[47,49,53],especi:[0,2,3,13,14,25,35],ess:[47,49,53],establish:[0,2,3,5,7,25],estim:[9,11,13,14,23,25,47,49,53],et:[1,2,3,4,6,9,10,11,12,13,14,15,22,23,25,35,47,49,53,54],etc:[13,14,25],evalu:[0,5,7,14,15,22,23,47,49,53,54,57],even:[2,3,11,14,25],evenli:[11,14,25],event:[13,14,25,57],eventu:[47,49,53],ever:[14,25],everi:[10,14,25],evid:[6,25],evolut:[0,2,3],evolutionari:[0,2,3,35,54],evolv:[0,2,3],exact:0,exactli:35,examin:[10,13,14,25],exampl:[0,2,3,6,7,9,10,12,13,14,23,25,47,49,54,57],execut:[6,25],exert:[14,25],exhibit:[0,2,3],exist:[0,12,13,14,25,47,49,52,53],expans:[2,3],expect:[56,60],expens:[23,25],experi:[1,2,3,4,6,9,54,57],experiment:[3,10,14,25,54],expert:[0,14,25,57],explain:[1,3,6,25],explanatori:[2,3],explicitli:[2,3],exploit:0,explor:[0,2,3,4,8,13,14,15,22,25,35,47,49,53,54],exploratori:[0,1,2,3,5,13,14,49,53,54,57],exponenti:[9,14,25],extend:[2,3,12,14,25,57],extens:[5,25],extent:[47,49,53],extract:[14,25],extrapol:[5,25],extrem:[0,9,13,14,25,57],face:[0,2,3,6,25],facilit:[47,49,53,57],fact:[2,3],factor:[0,1,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,25,47,49,53,56],factori:57,faith:[7,25],fals:57,famili:[2,3,54],far:[11,12,14,25,47,49,53],fashion:[2,3,54],fast:[12,14,25],favor:[2,3],feasibl:[0,4,14,25,54],featur:[22,25,54],fed:[14,25],federico:57,feedback:0,fennel:57,ferretti:57,few:[0,14,25],fewer:[6,11,14,25],fidel:[0,2,3,5,7,57],field:[14,25],fig:[4,6,7,9,10,14,23,25],figur:[0,1,3,23,25,54],filatova:57,fill:[10,11,14,25],filter:[6,25],find:[2,3],finer:[2,3],finger:[2,3],first:[1,2,3,12,14,23,25,56],fisher:[14,25],fit:[2,3],fix:[6,7,8,12,14,25,49,53],fjalar:57,flegal:[47,49,53],flexibl:0,focu:[0,1,2,3,6,7,8,14,25,35,49,52,53],focus:[0,13,14,25],follow:[5,13,14,15,22,25,56],form:[0,1,2,3,5,14,25,35,54],formal:[0,1,2,3,14,25,47,49,53,54],formul:[2,3],fortran:[52,53],forward:3,found:[2,3,5,7,14,25],foundat:[1,3,5,25],four:[9,14,25],fourier:[12,14,25],frac:[15,22,25,56],fraction:57,frame:[0,2,3,5,25,54],framework:[1,3,14,25,52,53,57],frankignoul:35,free:[52,53],freeli:[6,25],from:[0,1,2,3,6,10,11,14,25,47,49,53,54],full:[0,47,49,53,57],fulli:[0,2,3],fundament:[14,25],funtowicz:[2,3],further:[0,2,3,6,13,14,25],furthermor:[2,3,23,25],futur:[1,2,3,6,25],g:[0,1,2,3,4,6,7,9,11,13,14,15,22,25,52,53,54,57],gap:[0,11,14,25],gass:57,gaug:[5,6,25],gaussian:[52,53],gave:[2,3],gelman:[47,49,53],gener:[0,1,3,4,9,10,11,12,47,49,53,57],get:[1,3],geyer:[47,49,53],gibb:57,gilligan:57,given:[0,6,8,14,23,25],global:[0,1,3,6,57],glossari:[0,54,57],glue:57,goal:[0,2,3,4,25],gold:[47,49,53],good:[47,49,53],grand:57,graphic:[23,25],greatest:[6,25],grei:0,group:[5,25],grow:[0,7,9,14,25,49,53],growth:[0,7,9,14,25],gsa:[1,3],guarante:[2,3,10,11,14,25,47,49,53],guid:[0,2,3,54,57],guidanc:[0,54],gupta:[2,3],h:57,ha:[0,1,3,4,5,6,7,9,14,25,35,47,49,53],haan:57,haasnoot:57,hadjimichael:3,haim:57,hamilton:57,hamiltonian:57,hamm:[4,25],hand:[5,14,23,25],hardli:[14,25],harremo:57,harri:57,hasselmann:35,hast:57,have:[0,2,3,4,5,6,8,9,12,13,14,23,25,47,49,53,54],heat:0,heavili:[0,4,25],helb:57,held:[2,3],help:[2,3,5,6,25,54],helton:[14,25],here:[0,1,2,3,23,25,49,52,53],herman:[13,14,25],herrera:[13,14,25],heterogen:[2,3],heurist:[2,3,5,25,47,49,53],high:[2,3,7,13,14,25],higher:[0,2,3,9,14,25],highli:[0,1,2,3,5,8,10,14,25],highlight:0,hing:[0,7,25],histor:[0,2,3,4,25],histori:[2,3,13,14,25,54],hoc:[23,25],hole:[14,25],holst:57,horizont:0,hornberg:[6,25],host:[5,25],hot:[13,14,25],how:[0,1,2,3,4,8,14,15,22,54,57],howev:[2,3,4,7,12,14,25,47,49,52,53],human:[0,1,2,3,14,25,54,57],hunter:[9,14,25],hybrid:[49,53],hydrolog:[2,3],hypercub:[9,11,57],hypothes:[0,2,3,35,54],hypothesi:[0,1,3],i:[2,3,5,6,8,11,14,15,22,25,56,57],idea:[47,49,53],ideal:[0,1,3,14,25,47,49,53],identifi:[1,2,3,5,6,25],ignor:[2,3],illumin:[6,25],illustr:[0,1,3,4,25],im3:[0,2,3,54],immedi:[13,14,25],impact:[0,13,14,25],implement:[2,3,8,14,25,57],impli:0,implic:0,imposs:[4,6,14,25],improv:[0,1,3,6,7,11,14,25,47,49,53],includ:[0,2,3,4,12,14,25,52,53,54],inconsequenti:[7,25],inconsist:[52,53],incorpor:[13,14,25,52,53],increas:[0,6,7,8,9,14,23,25],increment:[2,3],inde:[9,14,25],independ:[1,3,8,14,23,47,49,53,57],index:[15,22,23,25,56,57],indic:[4,6,14,23,25,56],individu:[0,1,3,6,8,13,14,25,56],infer:[0,1,2,3,5,7,14,25,35,47,49,52,53,54],influenc:[0,1,2,3,4,8,14,25,35,54],influenti:[5,6,25],inform:[0,1,2,3,4,5,6,8,9,10,13,14,23,25,35,61],inher:[2,3,52,53],initi:[1,3,47,49,53],innov:[0,1,3,54],input:[0,1,2,3,4,6,12,15,22,35,57],inquiri:[2,3,5,25],insight:[0,2,3,54],inspir:[2,3],instanc:[5,6,7,25,52,53],instead:[2,3,4,5,9,14,25,35],institution:[2,3],insuffici:[2,3,13,14,25],integ:59,integr:[0,1,3,57],intens:[23,25],interact:[0,1,2,3,4,6,8,9,14,23,25],interdepend:[0,57],interest:[0,1,3,5,6,8,13,14,25,57],interpret:[2,3,47,49,53],intersect:[2,3],interv:[9,14,25],introduc:[14,25],introduct:[54,57],invari:[23,25],invers:57,invest:[5,25],investig:[5,25],involv:[47,49,53,54],ioanni:57,iooss:[11,14,25],issu:[2,3,57],iter:[3,8,14,25,47,49,53,54],its:[0,1,2,3,4,5,6,8,9,14,25,47,49,53],itself:[6,25],j:57,jakeman:57,jan:57,janssen:57,jeroen:57,jin:[11,14,25],joint:0,jonathan:57,journal:57,judg:[2,3],judgement:[47,49,53],julia:[52,53],justif:[2,3,35],justifi:35,k:[9,14,25],keep:56,kei:[0,14,25,47,49,52,53,54],kept:[8,14,25],kettner:57,kind:[14,25],kirsch:[13,14,25],kleindorf:[2,3],know:[0,2,3,5,25],knowledg:[2,3],known:[0,2,3,6,25],krauss:57,krayer:57,ksenia:57,kucherenko:[11,14,25],kumar:[2,3],kwakkel:57,l:57,laboratori:[14,25],lack:[47,49,53],lamontagn:[13,14,25],land:0,landscap:0,languag:[7,25,52,53],larg:[0,2,3,4,13,14,25,54],larger:[2,3,10,14,25,47,49,53],largest:[6,25],later:[0,5,14,25],latest:56,latin:[9,57],lead:[2,3,6,7,25],learn:[2,3,54],least:[5,23,25],leav:[14,25],left:[6,25],legitimaci:[2,3],lempert:[5,25],len:54,lend:[5,25],length:[14,25],less:[10,14,25,35],let:[15,22,25],level:[0,2,3,9,10,11,14,23,25],lh:[11,57],li:57,life:57,like:[5,6,23,25],likelihood:[0,35,47,49,53],limit:[0,2,3,4,5,8,9,11,13,14,22,23,25,49,52,53,54],linear:[0,4,14,23,25],linearli:[8,14,25],literatur:[0,2,3,4,6,9,12,13,14,25],local:[0,6,8,14,15,22,57],locat:[8,14,25],longer:[5,25],look:3,loss:[2,3],lost:[2,3],lot:0,louck:[13,14,25],low:[2,3,57],lower:[9,11,14,25],lumpi:[11,14,25],m:[4,25,57],made:[2,3,6,14,25],magnitud:[23,25],mai:[0,2,3,35,47,49,52,53],main:[8,9,14,25,54],major:[5,25],make:[0,2,3,6,7,9,14,23,25,54,57,60],manag:57,mandat:[23,25],mani:[0,2,3,5,14,25,47,49,52,53,57],manner:[5,6,8,10,14,25],map:[0,4,6,7,14,25,54],marchau:35,margin:[47,49,53],marjolein:57,marjolijn:57,markov:57,markovian:[47,49,53],martin:57,massmann:[14,25],mathemat:[11,14,15,22,25],matlab:[52,53],maxim:[10,14,25],maximum:[47,49,53],mckai:[10,14,25],mcmc:[47,49,52,53],mean:[1,3,11,14,25],meant:0,measur:[2,3,5,6,9,11,14,15,22,23,25],mechan:[2,3,52,53],medda:[13,14,25],membership:0,mention:[7,12,14,25],met:[2,3],method:[0,2,3,4,6,8,10,11,12,13,14,52,54,57],methodnam:60,methodolog:[0,2,3,14,25],metric:[1,2,3,5,6,7,8,14,25],metropoli:[14,25,57],might:[5,13,14,25],milli:[13,14,25],mimick:[2,3],mind:[7,13,14,25],mine:[14,25],minim:[11,14,25],minimum:[0,6,14,23,25],mirror:[2,3,54],miscalcul:[4,25],misconverg:[47,49,53],miss:[8,14,23,25],mitchel:[11,14,25],mitig:[14,25],mix:[47,49,53],moallemi:[35,57],mode:[0,2,3,6,25],model:[0,4,5,8,9,11,13,14,15,22,47,49,54,56,57,58],modern:54,modif:[10,14,25],modifi:[10,14,25],modul:[57,58],moira:57,moment:57,monetari:[5,25],mont:[1,3,6,11,14,25,57],montgomeri:[9,14,25],more:[0,1,2,3,4,6,7,8,9,12,14,23,25,35,47,49,52,53],moreov:[49,53,54],morri:[11,12,14,25],most:[1,2,3,4,5,6,7,10,11,14,22,23,25,35,49,53,54],motiv:[5,14,23,25],move:[12,14,25],msd:0,msd_uncertainty_ebook:56,much:[10,11,14,25],multi:[0,13,14,25,54],multidimension:[11,14,25],multipl:[2,3,8,14,23,25,47,49,53],multiscal:[0,2,3],multisector:[0,2,3],multitud:[0,2,3],must:[2,3,47,49,53],my:56,n:[4,9,10,12,14,23,25,57],narr:0,nation:57,nativ:[52,53],natur:[0,1,2,3,14,25,54,57],naylor:[2,3],nb:[6,25],necessari:[0,1,2,3,6,9,14,25],necessarili:[2,3],need:[0,1,2,3,7,14,25],neg:[14,25],negat:[8,14,25],neglect:[2,3],neglig:[6,9,14,25],negoti:[2,3],network:57,never:[2,3,14,25],next:[4,25],nexu:0,niel:57,nk:[9,14,25],nomin:[5,6,14,15,22,25],non:[0,6,13,14,25,47,49,53,54],nonlinear:[4,8,14,23,25],nor:[0,14,25],norton:[14,25],nossent:[23,25],notat:[15,22,25],note:[0,49,53,54],novel:0,now:[7,25],nowak:57,number:[0,2,3,6,9,10,11,12,14,23,25,47,49,53,54],numer:[0,2,3,4,5,6,14,25],oat:[15,22,57],object:[0,6,25,54],observ:[0,2,3,6,13,14,23,25,54],obtain:[9,14,25,47,49,53,56],occur:[2,3],often:[2,3,4,13,14,25],onc:[12,14,25,47,49,53],one:[1,2,3,5,6,8,9,10,11,12,14,15,22,25,35,47,49,53],onli:[2,3,4,8,9,14,15,22,23,25,47,49,53],open:[0,54],oper:[52,53,57],opinion:[0,14,25,57],opportun:0,oppos:0,optim:[11,14,25,57],order:[6,9,14,23,25,56],oresk:[2,3],organ:[5,25],orient:[2,3],origin:[1,3],orthogon:[10,14,25],other:[2,3,4,5,6,8,9,23,47,49,53,57],otherwis:[4,25],our:[0,1,2,3,5,25,54],out:[2,3,4,25],outcom:[0,1,3,5,6,25,35,57],output:[0,1,2,3,4,5,6,7,8,11,14,15,22,25,56],over:[14,25],overal:54,overcom:[8,14,25],overlap:[10,14,25],overview:[0,23,52,53,54,57],oxford:57,p:[9,14,25,57],packag:[52,53,58],page:57,pair:[5,25],pamela:57,panel:[4,7,25],paramet:[0,1,2,3,4,5,8,9,10,14,25,47,49,53,59],parameter:[1,2,3,5,25,35],parametr:[1,3,4,25,49,53],park:[11,14,25],partial:[4,15,22,25],particl:57,particular:[1,3,14,15,22,25,49,53],particularli:0,past:[13,14,25],path:[0,2,3,35,54],pawel:57,per:[2,3,9,14,25],percent:56,perfectli:[2,3],perform:[0,1,2,3,4,7,9,14,15,22,23,57],persist:[13,14,25],perspect:[0,7,25,49,53,54,57],pertin:[5,25,52,53],perturb:[4,12,14,15,22,25],peter:57,philip:57,philosoph:54,physic:[2,3,9,14,25,57],pianosi:[6,12,14,15,22,25],pie:[1,3],pinpoint:[6,25],pip:61,place:[0,2,3,7,25],placehold:59,platform:[14,25,52,53,54],plausibl:[14,25],pleas:[52,53],plischk:[4,25],plot:[0,4,25,47,49,53],plot_experimental_design:56,plural:[2,3],point:[2,3,4,11,12,14,25],polici:57,pool:[47,49,53],popul:[0,7,14,25],popular:[8,14,25,52,53],pose:0,positivist:[2,3],possibl:[2,3,6,9,14,25,35],posterior:[47,49,57],posteriori:[47,49,53],potenti:[0,2,3,9,11,13,14,23,25,47,49,53],poul:57,power:[0,2,3],practic:[2,3,4,25,47,49,53,57],practition:[2,3],pre:[6,25,57],precis:[2,3],predict:[2,3,35,57],prefer:[2,3,4,25,49,52,53],premis:[2,3],prescrib:3,presenc:0,present:[0,1,2,3,4,5,6,9,14,23,25,52,53,54],press:57,preval:[2,3],previou:35,previous:[7,25],price:[13,14,25],primari:0,primarili:[2,3,4,8,14,25],prior:[0,35,49,57],priori:[14,25],priorit:[6,7,25,49,53],probabilist:[0,35],probabl:[0,35,57],problem:[6,9,14,25,57],procedur:[2,3,14,25],process:[0,2,3,5,6,8,9,13,14,25,52,53],produc:[2,3,4,5,7,9,14,23,25],program:[52,53,56],progress:[2,3,54],project:[0,2,3,35,54],promot:0,propag:[1,3],properli:[4,25],properti:[0,5,11,14,25,47,49,53],propos:[2,3,4,6,9,14,25],proven:[4,23,25],provid:[0,1,2,3,4,13,14,22,25,54],proxi:[47,49,53],prudent:[23,25],pseudo:[11,14,25],publish:57,pure:[2,3],purpos:[2,3,5,13,14,23,25,54],put:0,python:[52,53],qiongli:57,qualit:[2,3],qualiti:[2,3],quantif:[0,47,49,52,57],quantifi:[1,3,35,49,53],quantit:[0,2,3,11,14,25],quantiti:[35,47,49,53],quasi:[11,14,25],quest:57,question:[0,1,2,3,5,14,23,25,47,49,53,54],quixot:57,r:[12,14,25,52,53],rais:[1,3],rakovec:[4,25],random:[11,14,25],randomli:[10,12,14,25,47,49,53],rang:[0,2,3,4,6,9,14,25],rank:[6,25],rapid:0,rapidli:[0,49,53],rare:[13,14,25],rather:[2,3,7,25,49,53],ratio:[47,49,53],reader:[2,3,6,12,13,14,23,25,54],readili:[14,25],real:[0,5,7,14,25],realiti:[2,3],reason:[2,3,5,8,9,13,14,25],recal:[23,25],recent:[9,14,25],recogn:[2,3],recommend:54,record:[13,14,25],reduc:[1,2,3,5,6,9,14,25],reduct:[6,25,47,49,53],refer:[0,4,6,9,12,14,25,35,47,49,53],refin:[5,25],reflect:[0,2,3,7,14,23,25],regard:[23,25],regardless:[5,25,47,49,53],region:[6,7,57],regress:[23,57],rel:[0,2,3,5,25,47,49,53,54],relat:[0,2,3,14,25,54],relationship:[0,1,3,4,14,23,25],relativist:[2,3,7,25],relev:[0,2,3,5,7,25],reli:[47,49,53],remain:[6,25],remark:0,repeat:[8,14,25],repetit:[12,14,25],replac:[47,49,53],report:[0,14,25],repres:[0,2,3,5,6,13,14,25,47,49,53],represent:[0,1,2,3,5,7,25,35],requir:[0,1,2,3,23,25,35,47,49,52,53],research:[0,5,14,23,25],resili:[0,1,2,3,54],resolut:[0,2,3,14,25],resourc:[5,25],respect:56,respond:[0,57],respons:[0,1,3,4,7,14,25],rest:[14,25],result:[0,1,2,3,4,5,6,9,10,11,14,25,47,49,53],reveal:[4,5,25],review:[0,13,14,25,49,53,54,57],rich:[1,3,13,14,25],right:[2,3,7,25],rigor:[0,54],rise:[2,3],risk:[0,6,25,57],road:57,robert:57,rocki:57,roger:57,rotman:57,rubin:[47,49,53],run:[1,3,6,7,9,14,25,47,49,52,53],runtest:60,s:[0,1,2,3,4,5,6,7,8,14,15,22,25,52,53,57],s_i:[15,22,25,56],sa:[5,25],salienc:[2,3],saltelli:[1,2,3,4,6,8,9,10,12,14,23,25,56,57],same:[2,3,5,9,10,11,14,23,25,52,53],sampl:[1,3,8,11,15,22,23,57],saul:57,scale:[0,2,3,14,15,22,23,25,47,49,53,54],scenario:[0,2,3,5,13,14,35,57],scheme:[8,9,10,14,25],schepdael:[14,25],scienc:[0,2,3,47,49,53,57],scientif:[0,2,3,14,25,54,57],scope:[0,2,3,13,14,25,49,53],screen:[6,25],script:[0,54],se:[2,3],search:57,second:[1,2,3,23,25],section:[0,2,3,6,7,13,22,23,35,49,53,54],sector:54,see:[52,53,54],seed:[14,25],seek:[0,2,3,54],seen:[2,3],select:[2,3,8,12,14,23,25,49,54,57],sensit:[0,1,2,3,8,10,12,13,14,15,49,53,54,56,57],sensor:[9,14,25],sequenc:[8,57],sequenti:[52,53],serena:57,seri:57,set:[0,1,2,3,4,5,6,14,23,25,35,47,49,53],sever:[2,3,4,12,13,14,23,25,47,49,53],shape:[0,5,6,14,25,54],shift:[2,3],shock:0,should:[0,2,3,4,6,14,23,25,47,49,53,54],show:[6,25],showcas:[0,54],shown:[0,1,3],side:[2,3],sign:[47,49,53],signatur:[2,3],signific:[0,2,3,5,6,25],significantli:[5,9,14,25],sim_i:56,similar:[2,3,8,14,25],simplic:[15,22,25],simplif:[1,3,5,25],simplifi:[2,3,5,23,25],simul:[1,2,3,9,11,14,25,57],singl:[2,3,15,22,25],singular:0,situat:[0,2,3,6,25],size:[11,23,47,49,53],skew:[47,49,53],skill:[47,49,53],sluij:57,small:[4,5,11,14,25,47,49,53],smallest:[6,25],so:[2,3,5,6,12,14,25,47,49,53,57],sobol:[11,12,14,23,25,56],societi:57,socio:57,softwar:57,solut:56,some:[0,5,8,14,22,25,47,49,53],someth:[11,14,25],sondoss:57,sophist:[6,8,14,25],sourc:[0,1,3,4,25,56],space:[0,1,2,3,4,6,8,9,10,11,12,14,25,35,47,49,53],span:[0,2,3],spatial:[2,3,13,14,25],spear:[6,25],specif:[0,1,2,3,4,5,6,12,14,25,35,52,53,54],specifi:0,spent:[47,49,53],stabl:[5,23,25],stai:[2,3],stakehold:[2,3,7,25],standard:[47,49,53],stano:57,stark:57,start:[12,14,25],state:[0,2,3,7,14,25],statement:[23,25],stationar:[13,14,25],stationari:[47,49,53],statist:[0,1,2,3,11,13,14,25,57],statistician:[47,49,53],steding:[13,14,25],steinschneid:[13,14,25],stem:[2,3],step:[1,3],still:[2,3],stochast:[13,14,25],strategi:[9,12,14,25],stratifi:[14,25],streamflow:[13,14,25],strength:[47,49,53],stress:[47,49,53],stressor:[0,2,3,13,14,25,35,54],strict:[7,25],strong:[0,13,14,25],strongli:54,structur:[0,1,2,3,5,9,14,25,35,49,53],studi:[0,4,5,13,14,25,47,49,53],style:[11,14,25],styliz:[6,25],subinterv:[10,14,25],subject:[2,3,57],submodul:58,subpackag:58,subroutin:[52,53],subsequ:[6,7,10,12,14,23,25],subset:[52,53],subspac:[11,14,25],substanti:[14,25,35],suffici:[0,2,3,5,7,13,14,25,47,49,53],suggest:[6,25,54],suit:[13,14,25,52,53],suitabl:[47,49,53],sum:59,sum_int:[59,60],summar:[14,25],summari:0,support:[0,2,3,54,57],sure:[56,60],surowiec:[9,14,25],surrog:57,surround:[5,25],survei:0,sushan:57,synthes:0,synthesi:[0,23,25],synthet:57,system:[0,1,2,3,5,7,8,14,23,25,54,57],systemat:[47,49,53,57],tabl:[14,25],tail:[47,49,57],take:[0,11,14,25],tang:[4,10,14,25],tarantola:[6,25],target:[47,49,53,61],task:0,tatiana:57,team:[0,9,14,25,54],technic:[0,2,3],techniqu:[3,4,5,6,8,10,11,12,14,23,25,54],technolog:[0,57],temperatur:[7,9,14,25],tempor:[2,3,13,14,25],tend:[7,25],tenuou:[2,3],term:[0,2,3,4,7,25],terminolog:54,test:[0,1,2,3,5,9,12,14,47,49,53,54,57,58,59],test_model:[58,59],test_sum_int:60,testcas:60,testmodel:60,text:[0,2,3,4,5,13,14,25,54],th:[15,22,25],than:[0,2,3,7,9,10,14,25,47,49,53],thei:[0,2,3,4,5,9,13,14,23,25,47,49,53],them:[5,6,7,14,25],themselv:[1,3],theoret:0,theori:[0,2,3,57],therefor:[2,3,4,5,6,14,25,47,49,53],thi:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,22,23,35,47,49,53,54,56],think:[49,53],thorough:54,those:[0,1,2,3,4,25,54],though:[2,3,11,14,25,47,49,53],three:[6,9,14,25],threshold:[0,6,25,47,49,53],through:[1,2,3,10,11,14,25,54],throughout:0,thu:[47,49,53],tian:35,time:[12,23,47,49,53,57],togeth:[0,1,3],toler:[6,25],tool:[0,1,2,3,6,13,14,25,54,57],toolkit:57,top:[7,25],topic:[0,13,14,25],total:[6,9,14,23,25],toulmin:[2,3],toward:[2,3],trace:[6,25],track:56,tradit:[0,2,3],tradition:[9,14,25],trait:57,trajectori:[12,14,25],transdisciplinari:[0,54,57],transit:[0,2,3],translat:0,treat:[9,13,14,25,49,53],treatment:[0,2,3,4,14,25],trivial:[0,54],truth:[2,3],tune:[47,49,53],turn:[14,23,25],two:[1,2,3,4,5,9,14,25,47,49,52,53,59],type:[0,2,3,6,57],typic:[0,9,14,23,25,47,49,52,53],uc:[0,1,3,35,54],ulam:[14,25],ultim:[8,14,25,54],unaccept:[7,25],uncertain:[0,1,2,3,4,5,6,8,9,10,12,14,23,25,35,49,53],uncertainti:[0,1,2,3,4,5,6,9,13,14,25,47,49,52,54],under:[0,2,3,7,14,25,54,57],underestim:[4,25],underli:[0,2,3],underlin:[15,22,25],undermin:[2,3],understand:[0,2,3,5,23,25,54,57],understood:0,undesir:[2,3,5,6,7,25],uniform:[9,11,14,25],uniformli:[11,12,14,25],unimport:[5,25],uniqu:[2,3],unit:0,unittest:60,univers:[47,49,53,57],unnecessari:[6,25],unpreced:[2,3],up:[2,3,14,25],upon:[52,53],uq:[0,14,25,35,52,53,54],urban:0,us:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,22,23,25,35,47,49,52,53,54],user:[52,53],usual:[0,6,12,14,25],util:[0,2,3,11,14,25],v:[6,25,56],v_:56,valid:[2,3,4,25],valu:[0,2,3,4,5,6,7,9,14,15,22,23,25,47,49,53,56,60],valuabl:[13,14,25],van:[14,25,57],vari:[0,4,6,14,23,25,47,49,53,56],variabl:[2,3,4,6,9,10,11,14,25],varianc:[6,11,14,47,49,53,56,57],variat:[12,14,25],varieti:[2,3,6,12,13,14,25],variou:[0,1,2,3,11,14,25,47,49,53,54],vast:[2,3],vector:[15,22,25],veri:[2,3,10,13,14,25,54],verif:[2,3],versu:[6,13,14,57],vertic:0,view:[0,2,3,7,25,54],viewpoint:[2,3],vogel:[13,14,25],von:57,vs:[2,3],vulner:[0,1,2,3,54],wa:[14,25],wai:[2,3,5,6,14,25],walker:[35,57],warrant:[5,25],warren:57,water:[0,56],wave:0,we:[0,1,2,3,4,5,22,25,49,52,53,54,61],weak:[47,49,53],weather:[13,14,25],weaver:[2,3,54],well:[0,2,3,5,6,13,14,22,25,35,47,49,53,54],were:[2,3,14,25],what:[1,3,23,25,57],when:[0,1,2,3,5,6,7,9,10,11,13,14,23,25,35,47,49,53,54],whenev:[4,25],where:[0,1,2,3,4,9,12,14,15,22,23,25,35,54],wherea:[2,3],whether:[2,3],which:[0,1,2,3,4,5,6,7,8,11,14,23,25,35,47,49,53],white:[2,3],why:[1,3,57],wicked:57,wide:[0,2,3,4,9,11,14,22,25],wilbi:[13,14,25],wilk:[13,14,25],william:57,wirtz:57,within:[0,2,3,4,6,9,11,14,25,47,49,52,53],without:[4,10,14,25,35,52,53],wolfgang:57,word:[4,9,14,23,25],work:54,workflow:[0,54],world:[0,5,14,25],would:[0,6,7,8,14,23,25],wrapper:[52,53],wrong:[2,3],wu:57,x1:[4,6,9,14,25],x2:[4,6,9,14,25],x3:[6,9,14,25],x:[4,6,14,15,22,25,56],x_i:56,xi:[6,25],xn:[4,25],xx:[47,49,53],y1:[4,25],y2:[4,25],y:[4,6,14,25,56,57],yacov:57,ye:[14,25],year:[13,14,25],yet:[47,49,53],yield:[1,3,5,25,35],ym:[4,25],you:[1,3],your:[5,25,56],zaremba:[11,14,25],zellner:[35,57]},titles:["Introduction","Overview of model diagnostics","Perspectives on diagnostic model evaluation","Diagnostic Modeling Overview and Perspectives","Global Versus Local Sensitivity","Why Perform Sensitivity Analysis","Sensitivity Analysis Applications for Model Evaluation and Fidelity Testing","Sensitivity Analysis Applications for Exploratory Modeling and Scenario Discovery","One-At-a-Time (OAT)","Full and Fractional Factorial Sampling","Latin Hypercube (LH) Sampling","Low-Discrepancy Sequences","Other types of sampling","Synthetic generation of input time series","Design of Experiments","Derivative-based Methods","Elementary Effect Methods","Regression-based Methods","Regional Sensitivity Analysis","Variance-based Methods","Analysis of Variance (ANOVA)","Moment-Independent (Density-Based) Methods","Sensitivity Analysis Methods","How To Choose A Sensitivity Analysis Method: Model Traits And Dimensionality","Software Toolkits","Sensitivity Analysis: The Basics","Understanding Errors: What Is Controlling Model Performance?","Consequential Dynamics: What is Controlling Model Behaviors of Interest?","Consequential Scenarios: What is Controlling Consequential Outcomes?","Sensitivity Analysis: Diagnostic & Exploratory Modeling","Why is Uncertainty Quantification Important for Understanding MultiSector System Dynamics?","Uncertainty Quantification for Exploratory Modeling","Bayesian Uncertainty Quantification","Uncertainty Quantification Under (Deep) Uncertainty","Integrating Model Diagnostics and Uncertainty Quantification","Uncertainty Quantification: The Basics","Understanding Risk: How Probable Are Extreme Events?","Understanding Tails: Statistical Modeling of Extreme Events","How to Choose an Appropriate Method?","How to Select a Prior Distribution?","Posterior Predictive Checking","Model Selection and Comparison","Scenario Discovery","Pre-calibration/GLUE","Metropolis-Hastings","Gibbs Sampling","Hamiltonian Monte Carlo","Markov Chain Monte Carlo","Particle-based Methods","What are Common Methods?","Markov Chain Monte Carlo with the True Model","Markov Chain Monte Carlo with Surrogate Models","What are Example Software Implementations?","Uncertainty Quantification: A Tool For Capturing Risks & Extremes","Conclusion","Glossary","Examples","Addressing Uncertainty in MultiSector Dynamics Research","nanites","nanites package","nanites.tests package","Preface"],titleterms:{"import":[30,35],"true":[50,52,53],A:[23,25,53],And:[23,25],At:[8,14,25],For:53,Is:[26,29],One:[8,14,25],The:[25,35],To:[23,25],acknowledg:61,address:57,an:[38,53],analysi:[5,6,7,18,20,22,23,25,29],anova:[20,22,25],applic:[6,7,25],appropri:[38,53],ar:[36,49,52,53],audienc:61,base:[15,17,19,21,22,25,48,49,53],basic:[25,35],bayesian:[32,35],behavior:[27,29],book:61,build:56,calibr:[43,49,53],caption:56,captur:53,carlo:[46,47,49,50,51,52,53],chain:[47,49,50,51,52,53],check:[40,53],choos:[23,25,38,53],codeblock:56,common:[49,53],comparison:[41,53],conclus:54,consequenti:[27,28,29],content:[59,60],control:[26,27,28,29],deep:[33,35],densiti:[21,22,25],depend:[14,25],deriv:[15,22,25],design:[14,25],diagnost:[1,2,3,29,34,35],dimension:[23,25],discoveri:[7,25,42,49,53],discrep:[11,14,25],distribut:[39,53],dynam:[27,29,30,35,57],effect:[16,22,25],elementari:[16,22,25],error:[26,29],evalu:[2,3,6,25],event:[36,37,53],exampl:[52,53,56],experi:[14,25],exploratori:[7,25,29,31,35],extrem:[36,37,53],factori:[9,14,25],fidel:[6,25],figur:56,fraction:[9,14,25],full:[9,14,25],gener:[13,14,25],gibb:[45,47,49,53],global:[4,25],glossari:55,glue:[43,49,53],hamiltonian:[46,47,49,53],hast:[44,47,49,53],how:[23,25,36,38,39,53,61],hypercub:[10,14,25],implement:[52,53],independ:[21,22,25],indic:57,input:[13,14,25],insert:56,instal:61,integr:[34,35],interest:[27,29],introduct:0,latin:[10,14,25],lh:[10,14,25],local:[4,25],low:[11,14,25],markov:[47,49,50,51,52,53],mathjax:56,method:[15,16,17,19,21,22,23,25,38,48,49,53],metropoli:[44,47,49,53],model:[1,2,3,6,7,23,25,26,27,29,31,34,35,37,41,50,51,52,53,59],modul:[59,60],moment:[21,22,25],mont:[46,47,49,50,51,52,53],msd_uncertainty_ebook:61,multisector:[30,35,57],nanit:[58,59,60],note:56,oat:[8,14,25],other:[12,14,25],out:56,outcom:[28,29],overview:[1,3,14,25],packag:[59,60,61],particl:[48,49,53],perform:[5,25,26,29],perspect:[2,3],posterior:[40,53],pre:[43,49,53],predict:[40,53],prefac:61,prior:[39,53],probabl:[36,53],python:61,quantif:[30,31,32,33,34,35,53],region:[18,22,25],regress:[17,22,25],research:57,risk:[36,53],sampl:[9,10,12,14,25,45,47,49,53],scenario:[7,25,28,29,42,49,53],section:[14,25],select:[39,41,53],sensit:[4,5,6,7,18,22,23,25,29],sequenc:[11,14,25],seri:[13,14,25],size:[14,25],softwar:[24,25,52,53],statist:[37,53],submodul:[59,60],subpackag:59,summari:[14,25],surrog:[51,52,53],synthet:[13,14,25],system:[30,35],tabl:57,tail:[37,53],test:[6,25,56,60],test_model:60,thi:[14,25,61],time:[8,13,14,25],tool:53,toolkit:[24,25],trait:[23,25],type:[12,14,25],uncertainti:[30,31,32,33,34,35,53,57],under:[33,35],understand:[26,29,30,35,36,37,53],us:[56,61],varianc:[19,20,22,25],versu:[4,25],what:[26,27,28,29,49,52,53],why:[5,25,30,35]}}) \ No newline at end of file diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.aux b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.aux index fdd5df7..e9a1a42 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.aux +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.aux @@ -26,7 +26,7 @@ \newlabel{1_introduction:introduction}{{1}{1}{Introduction}{chapter.1}{}} \newlabel{1_introduction::doc}{{1}{1}{Introduction}{chapter.1}{}} \@writefile{lof}{\contentsline {figure}{\numberline {1}{\ignorespaces State\sphinxhyphen {}of\sphinxhyphen {}the\sphinxhyphen {}art in different modeling communities, as reported in the survey distributed to IM3 teams. Deterministic Historical Evaluation: model evaluation under fully determined conditions defined using historical observations; Local Sensitivity Analysis: model evaluation performed by varying uncertain factors around specific reference values; Global Sensitivity Analysis: model evaluation performed by varying uncertain factors throughout their entire feasible value space; Uncertainty Characterization: model evaluation under alternative factor hypotheses to explore their implications for model output uncertainty; Uncertainty Quantification: representation of model output uncertainty using probability distributions; Traditional statistical inference: use of analysis results to describe deterministic or probabilistic outcomes resulting from the presence of uncertainty; Narrative scenarios: use of a limited decision\sphinxhyphen {}relevant number of scenarios to describe (sets of) changing system outcomes; Exploratory modeling for scenario discovery: use of large ensembles of uncertain conditions to discover decision\sphinxhyphen {}relevant combinations of uncertain factors}}{3}{figure.1.1}\protected@file@percent } -\newlabel{1_introduction:id1}{{1}{3}{State\sphinxhyphen {}of\sphinxhyphen {}the\sphinxhyphen {}art in different modeling communities, as reported in the survey distributed to IM3 teams. Deterministic Historical Evaluation: model evaluation under fully determined conditions defined using historical observations; Local Sensitivity Analysis: model evaluation performed by varying uncertain factors around specific reference values; Global Sensitivity Analysis: model evaluation performed by varying uncertain factors throughout their entire feasible value space; Uncertainty Characterization: model evaluation under alternative factor hypotheses to explore their implications for model output uncertainty; Uncertainty Quantification: representation of model output uncertainty using probability distributions; Traditional statistical inference: use of analysis results to describe deterministic or probabilistic outcomes resulting from the presence of uncertainty; Narrative scenarios: use of a limited decision\sphinxhyphen {}relevant number of scenarios to describe (sets of) changing system outcomes; Exploratory modeling for scenario discovery: use of large ensembles of uncertain conditions to discover decision\sphinxhyphen {}relevant combinations of uncertain factors}{figure.1.1}{}} +\newlabel{1_introduction:id9}{{1}{3}{State\sphinxhyphen {}of\sphinxhyphen {}the\sphinxhyphen {}art in different modeling communities, as reported in the survey distributed to IM3 teams. Deterministic Historical Evaluation: model evaluation under fully determined conditions defined using historical observations; Local Sensitivity Analysis: model evaluation performed by varying uncertain factors around specific reference values; Global Sensitivity Analysis: model evaluation performed by varying uncertain factors throughout their entire feasible value space; Uncertainty Characterization: model evaluation under alternative factor hypotheses to explore their implications for model output uncertainty; Uncertainty Quantification: representation of model output uncertainty using probability distributions; Traditional statistical inference: use of analysis results to describe deterministic or probabilistic outcomes resulting from the presence of uncertainty; Narrative scenarios: use of a limited decision\sphinxhyphen {}relevant number of scenarios to describe (sets of) changing system outcomes; Exploratory modeling for scenario discovery: use of large ensembles of uncertain conditions to discover decision\sphinxhyphen {}relevant combinations of uncertain factors}{figure.1.1}{}} \@writefile{toc}{\contentsline {chapter}{\numberline {2}Diagnostic Modeling Overview and Perspectives}{5}{chapter.2}\protected@file@percent } \@writefile{lof}{\addvspace {10\p@ }} \@writefile{lot}{\addvspace {10\p@ }} @@ -163,18 +163,26 @@ \@writefile{lot}{\addvspace {10\p@ }} \newlabel{7_conclusion:conclusion}{{7}{27}{Conclusion}{chapter.7}{}} \newlabel{7_conclusion::doc}{{7}{27}{Conclusion}{chapter.7}{}} -\@writefile{toc}{\contentsline {chapter}{\numberline {8}References}{29}{chapter.8}\protected@file@percent } +\@writefile{toc}{\contentsline {chapter}{\numberline {8}Glossary}{29}{chapter.8}\protected@file@percent } \@writefile{lof}{\addvspace {10\p@ }} \@writefile{lot}{\addvspace {10\p@ }} -\newlabel{8_references:references}{{8}{29}{References}{chapter.8}{}} -\newlabel{8_references::doc}{{8}{29}{References}{chapter.8}{}} -\@writefile{toc}{\contentsline {chapter}{\numberline {9}Glossary}{37}{chapter.9}\protected@file@percent } +\newlabel{8_glossary:glossary}{{8}{29}{Glossary}{chapter.8}{}} +\newlabel{8_glossary::doc}{{8}{29}{Glossary}{chapter.8}{}} +\@writefile{toc}{\contentsline {chapter}{\numberline {9}Indices and tables}{31}{chapter.9}\protected@file@percent } \@writefile{lof}{\addvspace {10\p@ }} \@writefile{lot}{\addvspace {10\p@ }} -\newlabel{9_glossary:glossary}{{9}{37}{Glossary}{chapter.9}{}} -\newlabel{9_glossary::doc}{{9}{37}{Glossary}{chapter.9}{}} -\@writefile{toc}{\contentsline {chapter}{\numberline {10}Indices and tables}{39}{chapter.10}\protected@file@percent } -\@writefile{lof}{\addvspace {10\p@ }} -\@writefile{lot}{\addvspace {10\p@ }} -\newlabel{index:indices-and-tables}{{10}{39}{Indices and tables}{chapter.10}{}} -\gdef \@abspage@last{43} +\newlabel{index:indices-and-tables}{{9}{31}{Indices and tables}{chapter.9}{}} +\bibcite{index:id11}{1} +\bibcite{index:id2}{2} +\bibcite{index:id3}{3} +\bibcite{index:id9}{4} +\bibcite{index:id4}{5} +\bibcite{index:id5}{6} +\bibcite{index:id8}{7} +\bibcite{index:id12}{8} +\bibcite{index:id6}{9} +\bibcite{index:id10}{10} +\bibcite{index:id13}{11} +\bibcite{index:id7}{12} +\@writefile{toc}{\contentsline {chapter}{Bibliography}{33}{chapter*.3}\protected@file@percent } +\gdef \@abspage@last{37} diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.fdb_latexmk b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.fdb_latexmk index 036b9b0..4e3a0f0 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.fdb_latexmk +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.fdb_latexmk @@ -1,10 +1,10 @@ # Fdb version 3 -["makeindex addressinguncertaintyinmultisectordynamicsresearch.idx"] 1622173776 "addressinguncertaintyinmultisectordynamicsresearch.idx" "addressinguncertaintyinmultisectordynamicsresearch.ind" "addressinguncertaintyinmultisectordynamicsresearch" 1622216225 - "addressinguncertaintyinmultisectordynamicsresearch.idx" 1622216224 0 d41d8cd98f00b204e9800998ecf8427e "pdflatex" +["makeindex addressinguncertaintyinmultisectordynamicsresearch.idx"] 1622672866 "addressinguncertaintyinmultisectordynamicsresearch.idx" "addressinguncertaintyinmultisectordynamicsresearch.ind" "addressinguncertaintyinmultisectordynamicsresearch" 1622672870 + "addressinguncertaintyinmultisectordynamicsresearch.idx" 1622672869 0 d41d8cd98f00b204e9800998ecf8427e "pdflatex" (generated) - "addressinguncertaintyinmultisectordynamicsresearch.ind" "addressinguncertaintyinmultisectordynamicsresearch.ilg" -["pdflatex"] 1622216223 "addressinguncertaintyinmultisectordynamicsresearch.tex" "addressinguncertaintyinmultisectordynamicsresearch.pdf" "addressinguncertaintyinmultisectordynamicsresearch" 1622216225 + "addressinguncertaintyinmultisectordynamicsresearch.ind" +["pdflatex"] 1622672868 "addressinguncertaintyinmultisectordynamicsresearch.tex" "addressinguncertaintyinmultisectordynamicsresearch.pdf" "addressinguncertaintyinmultisectordynamicsresearch" 1622672870 "/usr/local/texlive/2021/texmf-dist/fonts/enc/dvips/tex-gyre/q-ec.enc" 1529098226 2457 aaabbccba4df2a7f7371410ee4e075a3 "" "/usr/local/texlive/2021/texmf-dist/fonts/enc/dvips/tex-gyre/q-ts1.enc" 1529098226 3124 3813fd4c981d99822890a2861b0d274c "" "/usr/local/texlive/2021/texmf-dist/fonts/map/fontname/texfonts.map" 1577235249 3524 cb3e574dea2d1052e39280babc910dc8 "" @@ -140,39 +140,39 @@ "/usr/local/texlive/2021/texmf-var/fonts/map/pdftex/updmap/pdftex.map" 1616695603 5169178 a2ce6e2d73f603b690db6777d06dbccb "" "/usr/local/texlive/2021/texmf-var/web2c/pdftex/pdflatex.fmt" 1622119425 2724741 fab0e2aa17d02fc4386894d351bd4006 "" "/usr/local/texlive/2021/texmf.cnf" 1616695596 577 4c1911a7228e554c6032256d3cf5567d "" - "addressinguncertaintyinmultisectordynamicsresearch.aux" 1622216225 27207 b8b5eebe280bee8a00ef77228d788342 "pdflatex" - "addressinguncertaintyinmultisectordynamicsresearch.ind" 1622173776 0 d41d8cd98f00b204e9800998ecf8427e "makeindex addressinguncertaintyinmultisectordynamicsresearch.idx" - "addressinguncertaintyinmultisectordynamicsresearch.out" 1622216225 14478 cfdfe8e76528191e7cef8ab4eb0c584a "pdflatex" - "addressinguncertaintyinmultisectordynamicsresearch.tex" 1622216221 130282 1a5994c2f4eb6dbc9318b1e8ddf2f82b "" - "addressinguncertaintyinmultisectordynamicsresearch.toc" 1622216225 6067 eade09e14809b622e60ff2103e0150b9 "pdflatex" + "addressinguncertaintyinmultisectordynamicsresearch.aux" 1622672870 27266 8b009d4221dea1acfbbedb0b40b3f6ae "pdflatex" + "addressinguncertaintyinmultisectordynamicsresearch.ind" 1622672866 0 d41d8cd98f00b204e9800998ecf8427e "makeindex addressinguncertaintyinmultisectordynamicsresearch.idx" + "addressinguncertaintyinmultisectordynamicsresearch.out" 1622672870 14488 700f7a207d0268af2f72bdb28aa04315 "pdflatex" + "addressinguncertaintyinmultisectordynamicsresearch.tex" 1622672863 85586 02ee5757742ad2f93ff7ef2aedc53858 "" + "addressinguncertaintyinmultisectordynamicsresearch.toc" 1622672870 6053 bee423d24c216656f88a97f30597c899 "pdflatex" "figure1_state_of_the_science.png" 1622172293 57841 c8aee8d3cf5a71c8359d95609e510cf0 "" "figure2_idealized_uc.png" 1622174982 166028 44bbaea22f905869661692784fc39e0d "" "figure3_global_versus_local.png" 1622205961 395615 0d1b39495f78854678e10e9b32f50173 "" "figure4_factor_mapping.png" 1622207313 239434 1d143bcd8974f9ad4d41e540602d6318 "" "figure5_alternative_designs.png" 1622208986 66067 6abaa08d906ea0a6d872dc189f8655c5 "" - "sphinx.sty" 1622058511 13210 1fc7fddaf4a9dba457504448864afe1f "" - "sphinxhighlight.sty" 1622216221 6679 76d10c62e0f0661410b46f5db6118e26 "" - "sphinxlatexadmonitions.sty" 1622058511 6238 2d867d769abf3f72abc17ef52adff78b "" - "sphinxlatexgraphics.sty" 1622058511 4840 a9578332b6f3b35e198751fb632c9b79 "" - "sphinxlatexindbibtoc.sty" 1622058511 2066 b93f8504d02f6337fde3074b179de55e "" - "sphinxlatexlists.sty" 1622058511 3541 cdbe977296704a7f4febf62b72e20a0b "" - "sphinxlatexliterals.sty" 1622058511 35106 36d90f2f3a5de04ed4009f6607e82334 "" - "sphinxlatexnumfig.sty" 1622058511 4532 3633caf84afa1a98e1a060b7868202bb "" - "sphinxlatexobjects.sty" 1622058511 7270 a81899b12ca256f0a3b27e2783aa2cde "" - "sphinxlatexshadowbox.sty" 1622058511 3885 01355157c5580b34ec18031ea7a5e0e0 "" - "sphinxlatexstyleheadings.sty" 1622058511 3253 144b82f37e9d225d888cce45c19b8ed4 "" - "sphinxlatexstylepage.sty" 1622058511 3064 abce6c3018a84afee0afb08a431944ea "" - "sphinxlatexstyletext.sty" 1622058511 5771 a578900e23c0331bf3e01507feb96320 "" - "sphinxlatextables.sty" 1622058511 21848 2827eb0b11b99b185a8b77317d3e131c "" - "sphinxmanual.cls" 1622058511 4236 124cd90deb92742b5d3922bfc2cd70c0 "" - "sphinxmessages.sty" 1622216221 745 3f5fcd6cdd7964ed608767954a8ced6f "" - "sphinxoptionsgeometry.sty" 1622058511 2061 47bb34b8ed8a78823eb0c886abfb9f4d "" - "sphinxoptionshyperref.sty" 1622058511 1093 22dd608777751296c61a93174b0f4bb8 "" - "sphinxpackagefootnote.sty" 1622058511 13248 e2c77871396275ccf592403f8130f6f2 "" + "sphinx.sty" 1622668810 13210 1fc7fddaf4a9dba457504448864afe1f "" + "sphinxhighlight.sty" 1622672863 6679 76d10c62e0f0661410b46f5db6118e26 "" + "sphinxlatexadmonitions.sty" 1622668810 6238 2d867d769abf3f72abc17ef52adff78b "" + "sphinxlatexgraphics.sty" 1622668810 4840 a9578332b6f3b35e198751fb632c9b79 "" + "sphinxlatexindbibtoc.sty" 1622668810 2066 b93f8504d02f6337fde3074b179de55e "" + "sphinxlatexlists.sty" 1622668810 3541 cdbe977296704a7f4febf62b72e20a0b "" + "sphinxlatexliterals.sty" 1622668810 35106 36d90f2f3a5de04ed4009f6607e82334 "" + "sphinxlatexnumfig.sty" 1622668810 4532 3633caf84afa1a98e1a060b7868202bb "" + "sphinxlatexobjects.sty" 1622668810 7270 a81899b12ca256f0a3b27e2783aa2cde "" + "sphinxlatexshadowbox.sty" 1622668810 3885 01355157c5580b34ec18031ea7a5e0e0 "" + "sphinxlatexstyleheadings.sty" 1622668810 3253 144b82f37e9d225d888cce45c19b8ed4 "" + "sphinxlatexstylepage.sty" 1622668810 3064 abce6c3018a84afee0afb08a431944ea "" + "sphinxlatexstyletext.sty" 1622668810 5771 a578900e23c0331bf3e01507feb96320 "" + "sphinxlatextables.sty" 1622668810 21848 2827eb0b11b99b185a8b77317d3e131c "" + "sphinxmanual.cls" 1622668810 4236 124cd90deb92742b5d3922bfc2cd70c0 "" + "sphinxmessages.sty" 1622672863 745 3f5fcd6cdd7964ed608767954a8ced6f "" + "sphinxoptionsgeometry.sty" 1622668810 2061 47bb34b8ed8a78823eb0c886abfb9f4d "" + "sphinxoptionshyperref.sty" 1622668810 1093 22dd608777751296c61a93174b0f4bb8 "" + "sphinxpackagefootnote.sty" 1622668810 13248 e2c77871396275ccf592403f8130f6f2 "" (generated) - "addressinguncertaintyinmultisectordynamicsresearch.pdf" - "addressinguncertaintyinmultisectordynamicsresearch.idx" "addressinguncertaintyinmultisectordynamicsresearch.toc" - "addressinguncertaintyinmultisectordynamicsresearch.aux" + "addressinguncertaintyinmultisectordynamicsresearch.idx" "addressinguncertaintyinmultisectordynamicsresearch.log" + "addressinguncertaintyinmultisectordynamicsresearch.pdf" + "addressinguncertaintyinmultisectordynamicsresearch.aux" "addressinguncertaintyinmultisectordynamicsresearch.out" diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.log b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.log index 41355a1..f1c30f3 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.log +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.log @@ -1,4 +1,4 @@ -This is pdfTeX, Version 3.141592653-2.6-1.40.22 (TeX Live 2021) (preloaded format=pdflatex 2021.5.27) 28 MAY 2021 11:37 +This is pdfTeX, Version 3.141592653-2.6-1.40.22 (TeX Live 2021) (preloaded format=pdflatex 2021.5.27) 2 JUN 2021 18:27 entering extended mode restricted \write18 enabled. %&-line parsing enabled. @@ -794,7 +794,7 @@ Chapter 1. ] - + LaTeX Font Info: Trying to load font information for T1+txtt on input line 1 07. (/usr/local/texlive/2021/texmf-dist/tex/latex/txfonts/t1txtt.fd @@ -810,14 +810,14 @@ Package pdftex.def Info: figure1_state_of_the_science.png used on input line 1 ] Chapter 2. - + File: figure2_idealized_uc.png Graphic file (type png) Package pdftex.def Info: figure2_idealized_uc.png used on input line 129. (pdftex.def) Requested size: 469.75311pt x 287.44142pt. [5] [6 <./figure2_idealized_uc.png>] [7] [8] Chapter 3. - + File: figure3_global_versus_local.png Graphic file (type png) Package pdftex.def Info: figure3_global_versus_local.png used on input line 17 @@ -826,13 +826,13 @@ Package pdftex.def Info: figure3_global_versus_local.png used on input line 17 [9 ] [10 <./figure3_global_versus_local.png>] - + File: figure4_factor_mapping.png Graphic file (type png) Package pdftex.def Info: figure4_factor_mapping.png used on input line 213. (pdftex.def) Requested size: 376.40381pt x 314.06319pt. [11] [12 <./figure4_factor_mapping.png>] [13] - + File: figure5_alternative_designs.png Graphic file (type png) Package pdftex.def Info: figure5_alternative_designs.png used on input line 32 @@ -864,28 +864,28 @@ Chapter 7. ] Chapter 8. -[29] [30] [31] [32] [33] [34] [35] [36] +[29] [30 + +] Chapter 9. -[37 +[31] [32 -] [38 +] (./addressinguncertaintyinmultisectordynamicsresearch.ind) [33 ] -Chapter 10. -(./addressinguncertaintyinmultisectordynamicsresearch.ind) [39] (./addressinguncertaintyinmultisectordynamicsresearch.aux) Package rerunfilecheck Info: File `addressinguncertaintyinmultisectordynamicsre search.out' has not changed. -(rerunfilecheck) Checksum: CFDFE8E76528191E7CEF8AB4EB0C584A;14478. +(rerunfilecheck) Checksum: 700F7A207D0268AF2F72BDB28AA04315;14488. ) Here is how much of TeX's memory you used: - 13716 strings out of 478994 - 204918 string characters out of 5858184 - 711347 words of memory out of 5000000 - 30727 multiletter control sequences out of 15000+600000 + 13732 strings out of 478994 + 205140 string characters out of 5858184 + 552339 words of memory out of 5000000 + 30737 multiletter control sequences out of 15000+600000 462595 words of font info for 68 fonts, out of 8000000 for 9000 1142 hyphenation exceptions out of 8191 - 83i,17n,88p,3812b,569s stack positions out of 5000i,500n,10000p,200000b,80000s + 83i,17n,88p,3880b,569s stack positions out of 5000i,500n,10000p,200000b,80000s {/usr/local/texlive/2021/texmf-dist/fonts/enc/dvips/tex-gyre/q-ts1.enc}{/usr/ local/texlive/2021/texmf-dist/fonts/enc/dvips/tex-gyre/q-ec.enc} -Output written on addressinguncertaintyinmultisectordynamicsresearch.pdf (43 pa -ges, 1085625 bytes). +Output written on addressinguncertaintyinmultisectordynamicsresearch.pdf (37 pa +ges, 1043930 bytes). PDF statistics: - 741 PDF objects out of 1000 (max. 8388607) - 663 compressed objects within 7 object streams - 108 named destinations out of 1000 (max. 500000) + 579 PDF objects out of 1000 (max. 8388607) + 508 compressed objects within 6 object streams + 114 named destinations out of 1000 (max. 500000) 510 words of extra memory for PDF output out of 10000 (max. 10000000) diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.out b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.out index b40f912..f6d1c7e 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.out +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.out @@ -53,6 +53,6 @@ \BOOKMARK [2][-]{subsection.6.8.1}{\376\377\000M\000a\000r\000k\000o\000v\000\040\000C\000h\000a\000i\000n\000\040\000M\000o\000n\000t\000e\000\040\000C\000a\000r\000l\000o\000\040\000w\000i\000t\000h\000\040\000t\000h\000e\000\040\000T\000r\000u\000e\000\040\000M\000o\000d\000e\000l}{section.6.8}% 53 \BOOKMARK [2][-]{subsection.6.8.2}{\376\377\000M\000a\000r\000k\000o\000v\000\040\000C\000h\000a\000i\000n\000\040\000M\000o\000n\000t\000e\000\040\000C\000a\000r\000l\000o\000\040\000w\000i\000t\000h\000\040\000S\000u\000r\000r\000o\000g\000a\000t\000e\000\040\000M\000o\000d\000e\000l\000s}{section.6.8}% 54 \BOOKMARK [0][-]{chapter.7}{\376\377\000C\000o\000n\000c\000l\000u\000s\000i\000o\000n}{}% 55 -\BOOKMARK [0][-]{chapter.8}{\376\377\000R\000e\000f\000e\000r\000e\000n\000c\000e\000s}{}% 56 -\BOOKMARK [0][-]{chapter.9}{\376\377\000G\000l\000o\000s\000s\000a\000r\000y}{}% 57 -\BOOKMARK [0][-]{chapter.10}{\376\377\000I\000n\000d\000i\000c\000e\000s\000\040\000a\000n\000d\000\040\000t\000a\000b\000l\000e\000s}{}% 58 +\BOOKMARK [0][-]{chapter.8}{\376\377\000G\000l\000o\000s\000s\000a\000r\000y}{}% 56 +\BOOKMARK [0][-]{chapter.9}{\376\377\000I\000n\000d\000i\000c\000e\000s\000\040\000a\000n\000d\000\040\000t\000a\000b\000l\000e\000s}{}% 57 +\BOOKMARK [0][-]{chapter*.3}{\376\377\000B\000i\000b\000l\000i\000o\000g\000r\000a\000p\000h\000y}{}% 58 diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.pdf b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.pdf index 24b7f32..12f396d 100644 Binary files a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.pdf and b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.pdf differ diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.tex b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.tex index 32d2f26..5da7115 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.tex +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.tex @@ -60,7 +60,7 @@ \title{Addressing Uncertainty in MultiSector Dynamics Research} -\date{May 28, 2021} +\date{Jun 02, 2021} \release{v0.1.0} \author{Patrick M.\@{} Reed, ...\@{}} \newcommand{\sphinxlogo}{\vbox{}} @@ -95,20 +95,20 @@ \chapter{Introduction} \sphinxstyleemphasis{Understand the implications of uncertainty in data, observations, models, and model coupling approaches for projections of human\sphinxhyphen{}natural system dynamics.} \sphinxAtStartPar -Addressing the objectives above poses a strong transdisciplinary challenge that heavily depends on a diversity of models and, more specifically, a consistent framing for making model\sphinxhyphen{}based science inferences. The term transdisciplinary science as used here formally implies a deep integration of disciplines to aid our hypothesis driven understanding of coupled human\sphinxhyphen{}natural systems\textendash{}bridging differences in theory, hypothesis generation, modeling, and modes of inference (National Research Council, 2014). The IM3 MSD research focus and questions require a deep integration across disciplines, where new modes of analysis can emerge that rapidly synthesize and exploit advances for making decision relevant insights that at minimum acknowledge uncertainty and more ideally promote a rigorous quantitative mapping of its effects on the generality of claimed scientific insights. More broadly, diverse scientific disciplines engaged in the science of coupled human\sphinxhyphen{}natural systems, ranging from natural sciences to engineering and economics, employ a diversity of numerical computer models to study and understand their underlying systems of focus. The utility of these computer models hinges on their ability to represent the underlying real systems with sufficient fidelity and enable the inference of novel insights. This is particularly challenging in the case of coupled human\sphinxhyphen{}natural systems where there exists a multitude of interdependent human and natural processes taking place that could potentially be represented. These processes usually translate into modeled representations that are highly complex, non\sphinxhyphen{}linear, and exhibit strong interactions and threshold behaviors (Elsawah et al., 2020; Haimes, 2018; Helbing, 2013). Model complexity and detail have also been increasing as a result of our improving understanding of these processes, the availability of data, and the rapid growth in computing power (Saltelli et al., 2019). As model complexity grows, modelers need to specify a lot more information than before: additional model inputs and relationships as more processes are represented, higher resolution data as more observations are collected, new coupling relationships and interactions as models are put together to answer multisector questions (e.g., the land\sphinxhyphen{}water\sphinxhyphen{}energy nexus). Typically, not all of this information is well known, nor is the impact of these many uncertainties on model outputs well understood. It is further especially difficult to distinguish the effects of individual as well as interacting sources of uncertainty when modeling coupled systems with multisector and multiscale dynamics (Wirtz and Nowak, 2017). +Addressing the objectives above poses a strong transdisciplinary challenge that heavily depends on a diversity of models and, more specifically, a consistent framing for making model\sphinxhyphen{}based science inferences. The term transdisciplinary science as used here formally implies a deep integration of disciplines to aid our hypothesis driven understanding of coupled human\sphinxhyphen{}natural systems\textendash{}bridging differences in theory, hypothesis generation, modeling, and modes of inference {[}\hyperlink{cite.index:id2}{2}{]}. The IM3 MSD research focus and questions require a deep integration across disciplines, where new modes of analysis can emerge that rapidly synthesize and exploit advances for making decision relevant insights that at minimum acknowledge uncertainty and more ideally promote a rigorous quantitative mapping of its effects on the generality of claimed scientific insights. More broadly, diverse scientific disciplines engaged in the science of coupled human\sphinxhyphen{}natural systems, ranging from natural sciences to engineering and economics, employ a diversity of numerical computer models to study and understand their underlying systems of focus. The utility of these computer models hinges on their ability to represent the underlying real systems with sufficient fidelity and enable the inference of novel insights. This is particularly challenging in the case of coupled human\sphinxhyphen{}natural systems where there exists a multitude of interdependent human and natural processes taking place that could potentially be represented. These processes usually translate into modeled representations that are highly complex, non\sphinxhyphen{}linear, and exhibit strong interactions and threshold behaviors {[}\hyperlink{cite.index:id3}{3}, \hyperlink{cite.index:id4}{5}, \hyperlink{cite.index:id5}{6}{]}. Model complexity and detail have also been increasing as a result of our improving understanding of these processes, the availability of data, and the rapid growth in computing power {[}\hyperlink{cite.index:id6}{9}{]}. As model complexity grows, modelers need to specify a lot more information than before: additional model inputs and relationships as more processes are represented, higher resolution data as more observations are collected, new coupling relationships and interactions as models are put together to answer multisector questions (e.g., the land\sphinxhyphen{}water\sphinxhyphen{}energy nexus). Typically, not all of this information is well known, nor is the impact of these many uncertainties on model outputs well understood. It is further especially difficult to distinguish the effects of individual as well as interacting sources of uncertainty when modeling coupled systems with multisector and multiscale dynamics {[}\hyperlink{cite.index:id7}{12}{]}. \sphinxAtStartPar -Given the challenge and opportunity posed by the disciplinary diversity of IM3, we utilized a team\sphinxhyphen{}wide survey to allow the project’s membership to provide their views on how their areas typically address uncertainty, emphasizing key literature examples and domain\sphinxhyphen{}specific reviews. Our synthesis of this survey information in Figure 1 summaries the team’s perspectives, enabling a summary of the commonalities and differences for how different disciplinary areas are typically addressing uncertainty. Figure 1 highlights the non\sphinxhyphen{}trivial challenge posed by seeking to carefully consider uncertainty across an MSD focused transdisciplinary team. There are significant differences across the team’s contributing disciplines in terms of the methodological approaches and tools used in the treatment of uncertainty. The horizontal axis of the figure represents a conceptual continuum of methodological approaches, ranging from deterministic (no uncertainty) modeling to the theoretical case of fully engaging in modeling all sources of uncertainty. The vertical axis of plot maps the analysis tools that are used in the disciplines’ literature, spanning error\sphinxhyphen{}driven historical analyses to full uncertainty quantification. Given that Figure 1 is a conceptual illustration, the mapping of each discipline’s boundaries is not meant to imply exactness. They encompass the scope of feedback attained in the team\sphinxhyphen{}wide survey responses. The color circles designate specific sources of uncertainty that could be considered. Within the mapped disciplinary approaches, the color circles distinguish those sources of uncertainty that are addressed in the bodies of literature reported by respondents. Note the complete absence of grey circles designating that at present few if any studies report results for understanding how model coupling relationships shape uncertainty. We can briefly distinguish the key terms of uncertainty quantification (UQ) and uncertainty characterization (UC). UQ refers to the formal focus on the full specification of likelihoods as well as distributional forms necessary to infer the joint probabilistic response across all modeled factors of interest (Cooke, 1991). Alternatively, uncertainty characterization as defined here, refers to exploratory modeling of alternative hypotheses for the co\sphinxhyphen{}evolutionary dynamics of influences, stressors, as well as path dependent changes in the form and function of modelled systems (Moallemi et al., 2020a; Walker et al., 2003). Uncertain factors are any model component which is affected by uncertainty: inputs, resolution levels, coupling relationships, model relationships and parameters. When a model has been established as a sufficiently accurate representation of the system some of these factors may reflect elements of the real\sphinxhyphen{}world system that the model represents (for example, a population level parameter would reflect a sufficiently accurate representation of the population level in the system under study). As discussed in later sections, the choice of UQ or UC depends on the specific goals of studies, the availability of data, the types of uncertainties (e.g., well\sphinxhyphen{}characterized or deep), the complexity of underlying models as well as the computational limits. Deep uncertainty (as opposed to well\sphinxhyphen{}characterized) refers to situations where expert opinions consulted on a decision do not know or cannot agree on system boundaries, or the outcomes of interest and their relative importance, or the prior probability distribution for the various uncertain factors present (Kwakkel et al., 2016; W. E. Walker et al., 2013). +Given the challenge and opportunity posed by the disciplinary diversity of IM3, we utilized a team\sphinxhyphen{}wide survey to allow the project’s membership to provide their views on how their areas typically address uncertainty, emphasizing key literature examples and domain\sphinxhyphen{}specific reviews. Our synthesis of this survey information in Figure 1 summaries the team’s perspectives, enabling a summary of the commonalities and differences for how different disciplinary areas are typically addressing uncertainty. Figure 1 highlights the non\sphinxhyphen{}trivial challenge posed by seeking to carefully consider uncertainty across an MSD focused transdisciplinary team. There are significant differences across the team’s contributing disciplines in terms of the methodological approaches and tools used in the treatment of uncertainty. The horizontal axis of the figure represents a conceptual continuum of methodological approaches, ranging from deterministic (no uncertainty) modeling to the theoretical case of fully engaging in modeling all sources of uncertainty. The vertical axis of plot maps the analysis tools that are used in the disciplines’ literature, spanning error\sphinxhyphen{}driven historical analyses to full uncertainty quantification. Given that Figure 1 is a conceptual illustration, the mapping of each discipline’s boundaries is not meant to imply exactness. They encompass the scope of feedback attained in the team\sphinxhyphen{}wide survey responses. The color circles designate specific sources of uncertainty that could be considered. Within the mapped disciplinary approaches, the color circles distinguish those sources of uncertainty that are addressed in the bodies of literature reported by respondents. Note the complete absence of grey circles designating that at present few if any studies report results for understanding how model coupling relationships shape uncertainty. We can briefly distinguish the key terms of uncertainty quantification (UQ) and uncertainty characterization (UC). UQ refers to the formal focus on the full specification of likelihoods as well as distributional forms necessary to infer the joint probabilistic response across all modeled factors of interest {[}\hyperlink{cite.index:id11}{1}{]}. Alternatively, uncertainty characterization as defined here, refers to exploratory modeling of alternative hypotheses for the co\sphinxhyphen{}evolutionary dynamics of influences, stressors, as well as path dependent changes in the form and function of modelled systems {[}\hyperlink{cite.index:id12}{8}, \hyperlink{cite.index:id13}{11}{]}. Uncertain factors are any model component which is affected by uncertainty: inputs, resolution levels, coupling relationships, model relationships and parameters. When a model has been established as a sufficiently accurate representation of the system some of these factors may reflect elements of the real\sphinxhyphen{}world system that the model represents (for example, a population level parameter would reflect a sufficiently accurate representation of the population level in the system under study). As discussed in later sections, the choice of UQ or UC depends on the specific goals of studies, the availability of data, the types of uncertainties (e.g., well\sphinxhyphen{}characterized or deep), the complexity of underlying models as well as the computational limits. Deep uncertainty (as opposed to well\sphinxhyphen{}characterized) refers to situations where expert opinions consulted on a decision do not know or cannot agree on system boundaries, or the outcomes of interest and their relative importance, or the prior probability distribution for the various uncertain factors present {[}\hyperlink{cite.index:id9}{4}, \hyperlink{cite.index:id8}{7}{]}. \begin{figure}[htbp] \centering \capstart \noindent\sphinxincludegraphics[width=700\sphinxpxdimen]{{figure1_state_of_the_science}.png} -\caption{State\sphinxhyphen{}of\sphinxhyphen{}the\sphinxhyphen{}art in different modeling communities, as reported in the survey distributed to IM3 teams. Deterministic Historical Evaluation: model evaluation under fully determined conditions defined using historical observations; Local Sensitivity Analysis: model evaluation performed by varying uncertain factors around specific reference values; Global Sensitivity Analysis: model evaluation performed by varying uncertain factors throughout their entire feasible value space; Uncertainty Characterization: model evaluation under alternative factor hypotheses to explore their implications for model output uncertainty; Uncertainty Quantification: representation of model output uncertainty using probability distributions; Traditional statistical inference: use of analysis results to describe deterministic or probabilistic outcomes resulting from the presence of uncertainty; Narrative scenarios: use of a limited decision\sphinxhyphen{}relevant number of scenarios to describe (sets of) changing system outcomes; Exploratory modeling for scenario discovery: use of large ensembles of uncertain conditions to discover decision\sphinxhyphen{}relevant combinations of uncertain factors}\label{\detokenize{1_introduction:id1}}\end{figure} +\caption{State\sphinxhyphen{}of\sphinxhyphen{}the\sphinxhyphen{}art in different modeling communities, as reported in the survey distributed to IM3 teams. Deterministic Historical Evaluation: model evaluation under fully determined conditions defined using historical observations; Local Sensitivity Analysis: model evaluation performed by varying uncertain factors around specific reference values; Global Sensitivity Analysis: model evaluation performed by varying uncertain factors throughout their entire feasible value space; Uncertainty Characterization: model evaluation under alternative factor hypotheses to explore their implications for model output uncertainty; Uncertainty Quantification: representation of model output uncertainty using probability distributions; Traditional statistical inference: use of analysis results to describe deterministic or probabilistic outcomes resulting from the presence of uncertainty; Narrative scenarios: use of a limited decision\sphinxhyphen{}relevant number of scenarios to describe (sets of) changing system outcomes; Exploratory modeling for scenario discovery: use of large ensembles of uncertain conditions to discover decision\sphinxhyphen{}relevant combinations of uncertain factors}\label{\detokenize{1_introduction:id9}}\end{figure} \sphinxAtStartPar -At present, there is no singular guide for confronting the computational and conceptual challenges of the multi\sphinxhyphen{}model, transdisciplinary workflows that characterize ambitious projects such as IM3 (Saltelli et al., 2015). The primary aim of this text is to begin to address this gap and provide guidance for facing these challenges. Chapter 2 provides an overview of diagnostic modeling and the different perspectives for how we should evaluate our models. Chapter 3 the basic methods and concepts for sensitivity analysis. Chapter 4 delves into more technical applications of sensitivity analysis to support diagnostic model evaluation and exploratory modeling. Chapter 5 transitions to an overview of the key concepts and tools for UQ. Chapter 6 transitions to the use of UQ to capture risks and extremes in MSD systems. Chapter 7 provides concluding remarks across the UC and UQ topics covered in this text. The appendices of this text include a glossary of the key concepts as well as example test cases and scripts to showcase various UC and UQ related tools. +At present, there is no singular guide for confronting the computational and conceptual challenges of the multi\sphinxhyphen{}model, transdisciplinary workflows that characterize ambitious projects such as IM3 {[}\hyperlink{cite.index:id10}{10}{]}. The primary aim of this text is to begin to address this gap and provide guidance for facing these challenges. Chapter 2 provides an overview of diagnostic modeling and the different perspectives for how we should evaluate our models. Chapter 3 the basic methods and concepts for sensitivity analysis. Chapter 4 delves into more technical applications of sensitivity analysis to support diagnostic model evaluation and exploratory modeling. Chapter 5 transitions to an overview of the key concepts and tools for UQ. Chapter 6 transitions to the use of UQ to capture risks and extremes in MSD systems. Chapter 7 provides concluding remarks across the UC and UQ topics covered in this text. The appendices of this text include a glossary of the key concepts as well as example test cases and scripts to showcase various UC and UQ related tools. \chapter{Diagnostic Modeling Overview and Perspectives} @@ -523,232 +523,11 @@ \chapter{Conclusion} Although we distinguish the UC and UQ model diagnostics, the reader should note that we suggest an overall consistent approach to both in this text by emphasizing “exploratory modeling” (see review add citation). Although data support, model complexity, and computational limits strongly distinguish the feasibility and appropriateness of the UC and UQ diagnostic tools (e.g., see Figure 18), we overall recommend that modelers view their work through the lens of cycles of learning. Iterative and deliberative exploration of model\sphinxhyphen{}based hypotheses and inferences for transdisciplinary teams is non\sphinxhyphen{}trivial and ultimately critical for mapping where innovations or insights are most consequential. Overall, we recommend approaching modeling with an openness to the diverse disciplinary perspectives such as those mirrored by the IM3 family of models in a progression from evaluating models relative to observed history to advanced formalized analyses to make inferences on multi\sphinxhyphen{}sector, multi\sphinxhyphen{}scale vulnerabilities and resilience. Exploratory modeling approaches can help fashion experiments with large numbers of alternative hypotheses on the co\sphinxhyphen{}evolutionary dynamics of influences, stressors, as well as path\sphinxhyphen{}dependent changes in the form and function of coupled human\sphinxhyphen{}natural systems (Weaver et al., 2013). This text guides the reader through the use of sensitivity analysis and uncertainty methods across the diverse perspectives that have shaped modern diagnostic and exploratory modeling. -\chapter{References} -\label{\detokenize{8_references:references}}\label{\detokenize{8_references::doc}} -\sphinxAtStartPar -Akaike, H., 1978. On the Likelihood of a Time Series Model. J. R. Stat. Soc. Ser. Stat. 27, 217\textendash{}235. \sphinxurl{https://doi.org/10.2307/2988185} -Akaike, H., 1974. A new look at the statistical model identification. IEEE Trans. Autom. Control 19, 716\textendash{}723. \sphinxurl{https://doi.org/10.1109/TAC.1974.1100705} -Anderson, B., Borgonovo, E., Galeotti, M., Roson, R., 2014. Uncertainty in Climate Change Modeling: Can Global Sensitivity Analysis Be of Help? Risk Anal. 34, 271\textendash{}293. \sphinxurl{https://doi.org/10.1111/risa.12117} -Annan, J.D., Hargreaves, J.C., 2004. Efficient parameter estimation for a highly chaotic system. Tellus A 56, 520\textendash{}526. \sphinxurl{https://doi.org/10.1111/j.1600-0870.2004.00073.x} -Bakker, A.M.R., Wong, T.E., Ruckert, K.L., Keller, K., 2017. Sea\sphinxhyphen{}level projections representing the deeply uncertain contribution of the West Antarctic ice sheet. Sci Rep 7, 3880. \sphinxurl{https://doi.org/10.1038/s41598-017-04134-5} -Bankes, S., 1993. Exploratory Modeling for Policy Analysis. Oper. Res. 41, 435\textendash{}449. \sphinxurl{https://doi.org/10.1287/opre.41.3.435} -Barlas, Y., 1996. Formal aspects of model validity and validation in system dynamics. Syst. Dyn. Rev. J. Syst. Dyn. Soc. 12, 183\textendash{}210. -Barlas, Y., Carpenter, S., 1990. Philosophical roots of model validation: Two paradigms. Syst. Dyn. Rev. 6, 148\textendash{}166. \sphinxurl{https://doi.org/10.1002/sdr.4260060203} -Bastos, L.S., O’Hagan, A., 2009. Diagnostics for Gaussian Process Emulators. Technometrics 51, 425\textendash{}438. \sphinxurl{https://doi.org/10.1198/TECH.2009.08019} -Batouli, M., Mostafavi, A., 2018. Multiagent Simulation for Complex Adaptive Modeling of Road Infrastructure Resilience to Sea\sphinxhyphen{}Level Rise. Comput.\sphinxhyphen{}Aided Civ. Infrastruct. Eng. 33, 393\textendash{}410. \sphinxurl{https://doi.org/10.1111/mice.12348} -Bayes, T., 1763. An Essay towards Solving a Problem in the Doctrine of Chance. Philos. Trans. R. Soc. Lond. 53, 370\textendash{}418. -Betancourt, M., 2017. A Conceptual Introduction to Hamiltonian Monte Carlo. ArXiv StatME. -Beven, K., 2002. Towards a coherent philosophy for modelling the environment. Proc. R. Soc. Lond. Ser. Math. Phys. Eng. Sci. 458, 2465\textendash{}2484. \sphinxurl{https://doi.org/10.1098/rspa.2002.0986} -Beven, K., 1993. Prophecy, reality and uncertainty in distributed hydrological modelling. Adv. Water Resour., Research Perspectives in Hydrology 16, 41\textendash{}51. \sphinxurl{https://doi.org/10.1016/0309-1708(93)90028-E} -Beven, K., Binley, A., 1992. The future of distributed models: Model calibration and uncertainty prediction. Hydrol. Process. 6, 279\textendash{}298. \sphinxurl{https://doi.org/10.1002/hyp.3360060305} -Beven, K., Freer, J., 2001. Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology. J Hydrol. -Beven, K.J., 2018. On hypothesis testing in hydrology: Why falsification of models is still a really good idea. WIREs Water 5, e1278. \sphinxurl{https://doi.org/10.1002/wat2.1278} -Bobée, B., 1975. The Log Pearson type 3 distribution and its application in hydrology. Water Resour Res 11, 681\textendash{}689. \sphinxurl{https://doi.org/10.1029/WR011i005p00681} -Bobée, B., Rasmussen, P.F., 1995. Recent advances in flood frequency analysis. Rev Geophys 33, 1111\textendash{}1116. \sphinxurl{https://doi.org/10.1029/95RG00287} -Borgomeo, E., Farmer, C.L., Hall, J.W., 2015. Numerical rivers: A synthetic streamflow generator for water resources vulnerability assessments. Water Resour. Res. 51, 5382\textendash{}5405. \sphinxurl{https://doi.org/10.1002/2014WR016827} -Borgonovo, E., 2010. Sensitivity analysis with finite changes: An application to modified EOQ models. Eur. J. Oper. Res. 200, 127\textendash{}138. -Borgonovo, E., Plischke, E., 2016. Sensitivity analysis: A review of recent advances. Eur. J. Oper. Res. 248, 869\textendash{}887. \sphinxurl{https://doi.org/10.1016/j.ejor.2015.06.032} -Boukouvalas, A., Sykes, P., Cornford, D., Maruri\sphinxhyphen{}Aguilar, H., 2014. Bayesian Precalibration of a Large Stochastic Microsimulation Model. IEEE Trans Intell Transp Syst 15, 1337\textendash{}1347. \sphinxurl{https://doi.org/10.1109/tits.2014.2304394} -Box, G.E., Hunter, J.S., 1961. The 2 k—p fractional factorial designs. Technometrics 3, 311\textendash{}351. -Bryant, B.P., Lempert, R.J., 2010. Thinking inside the box: A participatory, computer\sphinxhyphen{}assisted approach to scenario discovery. Technol. Forecast. Soc. Change 77, 34\textendash{}49. \sphinxurl{https://doi.org/10.1016/j.techfore.2009.08.002} -Budescu, D.V., Broomell, S.B., Lempert, R.J., Keller, K., 2014. Aided and unaided decisions with imprecise probabilities in the domain of losses. EURO J. Decis. Process. 2, 31\textendash{}62. \sphinxurl{https://doi.org/10.1007/s40070-013-0023-4} -Bürkner, P.\sphinxhyphen{}C., Gabry, J., Vehtari, A., 2019. Approximate leave\sphinxhyphen{}future\sphinxhyphen{}out cross\sphinxhyphen{}validation for Bayesian time series models. ArXiv StatME. -Burnham, K.P., Anderson, D.R., 2004. Multimodel Inference: Understanding AIC and BIC in Model Selection. Sociol Methods Res 33, 261\textendash{}304. \sphinxurl{https://doi.org/10.1177/0049124104268644} -Burnham, K.P., Anderson, D.R., 2001. Kullback\sphinxhyphen{}Leibler information as a basis for strong inference in ecological studies. Wildl Res 28, 111\textendash{}119. \sphinxurl{https://doi.org/10.1071/wr99107} -Cao, Y., Li, B., 2019. Assessing models for estimation and methods for uncertainty quantification for spatial return levels. Environmetrics 30, e2508. -Cash, D.W., Clark, W.C., Alcock, F., Dickson, N.M., Eckley, N., Guston, D.H., Jäger, J., Mitchell, R.B., 2003. Knowledge systems for sustainable development. Proc. Natl. Acad. Sci. 100, 8086\textendash{}8091. -Ciccazzo, A., Pillo, G.D., Latorre, V., 2014. Support vector machines for surrogate modeling of electronic circuits. Neural Comput. Appl. 24, 69\textendash{}76. \sphinxurl{https://doi.org/10.1007/s00521-013-1509-5} -Coles, S., Bawa, J., Trenner, L., Dorazio, P., 2001. An introduction to statistical modeling of extreme values. Springer. -Cooke, R.M., 1991. Experts in uncertainty: Opinion and subjective probability in science, Experts in uncertainty: Opinion and subjective probability in science. Oxford University Press, New York, NY, US. -Coronese, M., Lamperti, F., Keller, K., Chiaromonte, F., Roventini, A., 2019. Evidence for sharp increase in the economic damages of extreme natural disasters. Proc Natl Acad Sci U A 201907826. \sphinxurl{https://doi.org/10.1073/pnas.1907826116} -Cukier, R.I., Fortuin, C.M., Shuler, K.E., Petschek, A.G., Schaibly, J.H., 1973. Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory. J. Chem. Phys. 59, 3873\textendash{}3878. \sphinxurl{https://doi.org/10.1063/1.1680571} -Currin, C., Mitchell, T., Morris, M., Ylvisaker, D., 1991. Bayesian Prediction of Deterministic Functions, with Applications to the Design and Analysis of Computer Experiments. J Am Stat Assoc 86, 953\textendash{}963. \sphinxurl{https://doi.org/10.1080/01621459.1991.10475138} -Dalal, I.L., Stefan, D., Harwayne\sphinxhyphen{}Gidansky, J., 2008. Low discrepancy sequences for Monte Carlo simulations on reconfigurable platforms, in: 2008 International Conference on Application\sphinxhyphen{}Specific Systems, Architectures and Processors. Presented at the 2008 International Conference on Application\sphinxhyphen{}Specific Systems, Architectures and Processors, pp. 108\textendash{}113. \sphinxurl{https://doi.org/10.1109/ASAP.2008.4580163} -DeConto, R.M., Pollard, D., 2016. Contribution of Antarctica to past and future sea\sphinxhyphen{}level rise. Nature 531, 591\textendash{}597. \sphinxurl{https://doi.org/10.1038/nature17145} -Del Moral, P., Doucet, A., Jasra, A., 2006. Sequential Monte Carlo samplers. J R Stat Soc Ser. B Stat Methodol 68, 411\textendash{}436. \sphinxurl{https://doi.org/10.1111/j.1467-9868.2006.00553.x} -Demarta, S., McNeil, A.J., 2005. The t copula and related copulas. Int. Stat. Rev. 73, 111\textendash{}129. -Doucet, A., Freitas, N., Gordon, N., 2001. An Introduction to Sequential Monte Carlo Methods. Seq. Monte Carlo Methods Pract. 3\textendash{}14. \sphinxurl{https://doi.org/10.1007/978-1-4757-3437-9\_1} -Doucet, A., Godsill, S., Andrieu, C., 2000. On sequential Monte Carlo sampling methods for Bayesian filtering. Stat Comput 10, 197\textendash{}208. \sphinxurl{https://doi.org/10.1023/A:1008935410038} -Eason, J., Cremaschi, S., 2014. Adaptive sequential sampling for surrogate model generation with artificial neural networks. Comput. Chem. Eng. 68, 220\textendash{}232. \sphinxurl{https://doi.org/10.1016/j.compchemeng.2014.05.021} -Edwards, N.R., Cameron, D., Rougier, J., 2011. Precalibrating an intermediate complexity climate model. Clim Dyn 37, 1469\textendash{}1482. \sphinxurl{https://doi.org/10.1007/s00382-010-0921-0} -Eker, S., Rovenskaya, E., Obersteiner, M., Langan, S., 2018. Practice and perspectives in the validation of resource management models. Nat. Commun. 9, 1\textendash{}10. \sphinxurl{https://doi.org/10.1038/s41467-018-07811-9} -El Adlouni, S., Bobée, B., Ouarda, T.B.M.J., 2008. On the tails of extreme event distributions in hydrology. J Hydrol 355, 16\textendash{}33. \sphinxurl{https://doi.org/10.1016/j.jhydrol.2008.02.011} -Ellsberg, D., 1961. Risk, Ambiguity, and the Savage Axioms. Q J Econ 75, 643\textendash{}669. \sphinxurl{https://doi.org/10.2307/1884324} -Elsawah, S., Filatova, T., Jakeman, A.J., Kettner, A.J., Zellner, M.L., Athanasiadis, I.N., Hamilton, S.H., Axtell, R.L., Brown, D.G., Gilligan, J.M., Janssen, M.A., Robinson, D.T., Rozenberg, J., Ullah, I.I.T., Lade, S.J., 2020. Eight grand challenges in socio\sphinxhyphen{}environmental systems modeling. Socio\sphinxhyphen{}Environ. Syst. Model. 2, 16226\textendash{}16226. \sphinxurl{https://doi.org/10.18174/sesmo.2020a16226} -Evensen, G., 1994. Sequential data assimilation with a nonlinear quasi\sphinxhyphen{}geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res. Oceans 99, 10143\textendash{}10162. \sphinxurl{https://doi.org/10.1029/94JC00572} -Ferdous, M.R., Wesselink, A., Brandimarte, L., Di Baldassarre, G., Rahman, M.M., 2019. The levee effect along the Jamuna River in Bangladesh. Water Int 44, 496\textendash{}519. \sphinxurl{https://doi.org/10.1080/02508060.2019.1619048} -Field, C.B., Barros, V.R., Dokken, D.J., Mach, K.J., Mastrandrea, M.D., Bilir, T.E., Chatterjee, M., Ebi, K.L., Estrada, Y.O., Genova, R.C., Girma, B., Kissel, E.S., Levy, A.N., MacCracken, S., Mastrandrea, P.R., White, L.L., 2014. Summary for Policymakers, in: Climate Change 2014: Impacts, Adaptation, and Vulnerability. Part A: Global and Sectoral Aspects. Contribution of Working Group II to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge, United Kingdom, and New York, NY, USA, pp. 1\textendash{}32. -Fisher, R.A., 1960. The design of experiments. Oliver and Boyd, Edinburgh, Scotland, UK. -Flegal, J.M., Haran, M., Jones, G.L., 2008. Markov Chain Monte Carlo: Can We Trust the Third Significant Figure? Stat Sci 23, 250\textendash{}260. \sphinxurl{https://doi.org/10.1214/08-STS257} -Frankignoul, C., Hasselmann, K., 1977. Stochastic climate models, Part II Application to sea\sphinxhyphen{}surface temperature anomalies and thermocline variability. Tell’Us 29, 289\textendash{}305. \sphinxurl{https://doi.org/10.3402/tellusa.v29i4.11362} -Fuller, R.W., Wong, T.E., Keller, K., 2017. Probabilistic inversion of expert assessments to inform projections about Antarctic ice sheet responses. PLoS One 12, e0190115. \sphinxurl{https://doi.org/10.1371/journal.pone.0190115} -Gao, L., Bryan, B.A., Nolan, M., Connor, J.D., Song, X., Zhao, G., 2016. Robust global sensitivity analysis under deep uncertainty via scenario analysis. Environ. Model. Softw. 76, 154\textendash{}166. \sphinxurl{https://doi.org/10.1016/j.envsoft.2015.11.001} -Gelfand, A.E., Hills, S.E., Racine\sphinxhyphen{}Poon, A., Adrian F. M. Smith, 1990. Illustration of Bayesian Inference in Normal Data Models Using Gibbs Sampling. J Am Stat Assoc 85, 972\textendash{}985. \sphinxurl{https://doi.org/10.2307/2289594} -Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B., 2013. Bayesian Data Analysis, Third. ed. CRC Press. -Gelman, A., Hwang, J., Vehtari, A., 2014. Understanding predictive information criteria for Bayesian models. Stat Comput 24, 997\textendash{}1016. \sphinxurl{https://doi.org/10.1007/s11222-013-9416-2} -Gelman, A., Meng, X.\sphinxhyphen{}L., Stern, H., 1996. Posterior predictive assessment of model fitness via realized discrepancies. Stat Sin 6, 733\textendash{}760. -Gelman, A., Rubin, D.B., 1992. Inference from Iterative Simulation Using Multiple Simulations. Stat Sci 7, 457\textendash{}511. \sphinxurl{https://doi.org/10.1214/ss/1177011136} -Gelman, A., Shalizi, C.R., 2013. Philosophy and the practice of Bayesian statistics. Br J Math Stat Psychol 66, 8\textendash{}38. \sphinxurl{https://doi.org/10.1111/j.2044-8317.2011.02037.x} -Gelman, A., Simpson, D., Betancourt, M., 2017. The Prior Can Often Only Be Understood in the Context of the Likelihood. Entropy 19, 555. \sphinxurl{https://doi.org/10.3390/e19100555} -Geman, S., Geman, D., 1984. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans Pattern Anal Mach Intell 6, 721\textendash{}741. \sphinxurl{https://doi.org/10.1109/tpami.1984.4767596} -Geyer, C.J., 2011. Introduction to Markov Chain Monte Carlo, in: Brooks, S., Gelman, A., Jones, G.L., Meng, X.\sphinxhyphen{}L. (Eds.), Handbook of Markov Chain Monte Carlo. Chapman \& Hall/CRC, Boca Raton, FL, pp. 3\textendash{}48. -Geyer, C.J., Johnson, L.T., 2013. mcmc: Markov Chain Monte Carlo. R Package Version 0 9\sphinxhyphen{}2 URL HttpCRAN R\sphinxhyphen{}Proj. Orgpackage Mcmc. -Ghanem, R.G., Spanos, P.D., 1991. Spectral stochastic finite\sphinxhyphen{}element formulation for reliability analysis. J Eng Mech 117, 2351\textendash{}2372. -Goodman, J., Weare, J., 2010. Ensemble samplers with affine invariance. Commun. Appl. Math. Comput. Sci. 5, 65\textendash{}80. \sphinxurl{https://doi.org/10.2140/camcos.2010.5.65} -Gordon, N.J., Salmond, D.J., Smith, A.F.M., 1993. Novel approach to nonlinear/non\sphinxhyphen{}Gaussian Bayesian state estimation. IEE Proc. F Radar Signal Process. 140, 107\textendash{}113. \sphinxurl{https://doi.org/10.1049/ip-f-2.1993.0015} -Gorissen, D., De Tommasi, L., Crombecq, K., Dhaene, T., 2009. Sequential modeling of a low noise amplifier with neural networks and active learning. Neural Comput. Appl. 18, 485\textendash{}494. \sphinxurl{https://doi.org/10.1007/s00521-008-0223-1} -Gupta, H.V., Clark, M.P., Vrugt, J.A., Abramowitz, G., Ye, M., 2012. Towards a comprehensive assessment of model structural adequacy. Water Resour. Res. 48. \sphinxurl{https://doi.org/10.1029/2011WR011044} -Gupta, H.V., Sorooshian, S., Yapo, P.O., 1998. Toward improved calibration of hydrologic models: Multiple and noncommensurable measures of information. Water Resour. Res. 34, 751\textendash{}763. \sphinxurl{https://doi.org/10.1029/97WR03495} -Gupta, H.V., Wagener, T., Liu, Y., 2008. Reconciling theory with observations: elements of a diagnostic approach to model evaluation. Hydrol. Process. Int. J. 22, 3802\textendash{}3813. -Hadjimichael, A., Quinn, J., Reed, P., 2020. Advancing Diagnostic Model Evaluation to Better Understand Water Shortage Mechanisms in Institutionally Complex River Basins. Water Resour. Res. 56, e2020WR028079. \sphinxurl{https://doi.org/10.1029/2020WR028079} -Haer, T., Husby, T.G., Botzen, W.J.W., Aerts, J.C.J.H., 2020. The safe development paradox: An agent\sphinxhyphen{}based model for flood risk under climate change in the European Union. Glob Env. Change 60, 102009. \sphinxurl{https://doi.org/10.1016/j.gloenvcha.2019.102009} -Haimes, Y.Y., 2018. Risk Modeling of Interdependent Complex Systems of Systems: Theory and Practice. Risk Anal. 38, 84\textendash{}98. \sphinxurl{https://doi.org/10.1111/risa.12804} -Hall, J.W., Lempert, R.J., Keller, K., Hackbarth, A., Mijere, C., McInerney, D.J., 2012. Robust climate policies under uncertainty: A comparison of robust decision making and info\sphinxhyphen{}gap methods. Risk Anal. 32, 1657\textendash{}1672. \sphinxurl{https://doi.org/10.1111/j.1539-6924.2012.01802.x} -Hamm, N.A.S., Hall, J.W., Anderson, M.G., 2006. Variance\sphinxhyphen{}based sensitivity analysis of the probability of hydrologically induced slope instability. Comput. Geosci. 32, 803\textendash{}817. \sphinxurl{https://doi.org/10.1016/j.cageo.2005.10.007} -Han, Y., Ash, K., Mao, L., Peng, Z.\sphinxhyphen{}R., 2020. An agent\sphinxhyphen{}based model for community flood adaptation under uncertain sea\sphinxhyphen{}level rise. Clim Change. \sphinxurl{https://doi.org/10.1007/s10584-020-02802-6} -Hastings, W.K., 1970. Monte Carlo Sampling Methods Using Markov Chains and Their Applications. Biometrika 57, 97\textendash{}109. \sphinxurl{https://doi.org/10.2307/2334940} -Helbing, D., 2013. Globally networked risks and how to respond. Nature 497, 51\textendash{}59. \sphinxurl{https://doi.org/10.1038/nature12047} -Helton, J.C., Johnson, J.D., Sallaberry, C.J., Storlie, C.B., 2006. Survey of sampling\sphinxhyphen{}based methods for uncertainty and sensitivity analysis. Reliab. Eng. Syst. Saf., The Fourth International Conference on Sensitivity Analysis of Model Output (SAMO 2004) 91, 1175\textendash{}1209. \sphinxurl{https://doi.org/10.1016/j.ress.2005.11.017} -Herman, J.D., Reed, P.M., Wagener, T., 2013. Time\sphinxhyphen{}varying sensitivity analysis clarifies the effects of watershed model formulation on model behavior. Water Resour. Res. 49, 1400\textendash{}1414. \sphinxurl{https://doi.org/10.1002/wrcr.20124} -Herman, J.D., Zeff, H.B., Lamontagne, J.R., Reed, P.M., Characklis, G.W., 2016. Synthetic Drought Scenario Generation to Support Bottom\sphinxhyphen{}Up Water Supply Vulnerability Assessments. J. Water Resour. Plan. Manag. 142, 04016050. \sphinxurl{https://doi.org/10.1061/(ASCE)WR.1943-5452.0000701} -Herrera, M., Natarajan, S., Coley, D.A., Kershaw, T., Ramallo\sphinxhyphen{}González, A.P., Eames, M., Fosas, D., Wood, M., 2017. A review of current and future weather data for building simulation. Build. Serv. Eng. Res. Technol. 38, 602\textendash{}627. \sphinxurl{https://doi.org/10.1177/0143624417705937} -Hoffman, M.D., Gelman, A., 2014. The No\sphinxhyphen{}U\sphinxhyphen{}Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo. J Mach Learn Res 15, 1593\textendash{}1623. -Houtekamer, P.L., Mitchell, H.L., 1998. Data Assimilation Using an Ensemble Kalman Filter Technique. Mon. Weather Rev. 126, 796\textendash{}811. \sphinxurl{https://doi.org/10.1175/1520-0493(1998})126\textless{}0796:DAUAEK\textgreater{}2.0.CO;2 -Hu, L., Keller, C.A., Long, M.S., Sherwen, T., Auer, B., Da Silva, A., Nielsen, J.E., Pawson, S., Thompson, M.A., Trayanov, A.L., Travis, K.R., Grange, S.K., Evans, M.J., Jacob, D.J., 2018. Global simulation of tropospheric chemistry at 12.5 km resolution: performance and evaluation of the GEOS\sphinxhyphen{}Chem chemical module (v10\sphinxhyphen{}1) within the NASA GEOS Earth system model (GEOS\sphinxhyphen{}5 ESM). Geosci Model Dev 11, 4603\textendash{}4620. \sphinxurl{https://doi.org/10.5194/gmd-11-4603-2018} -Hu, Y.\sphinxhyphen{}M., Hendry, M., Heng, I.S., 2014. Efficient Exploration of Multi\sphinxhyphen{}Modal Posterior Distributions. ArXiv Astro\sphinxhyphen{}PhIM. -Hurrell, J.W., Holland, M.M., Gent, P.R., Ghan, S., Kay, J.E., Kushner, P.J., Lamarque, J.\sphinxhyphen{}F., Large, W.G., Lawrence, D., Lindsay, K., Lipscomb, W.H., Long, M.C., Mahowald, N., Marsh, D.R., Neale, R.B., Rasch, P., Vavrus, S., Vertenstein, M., Bader, D., Collins, W.D., Hack, J.J., Kiehl, J., Marshall, S., 2013. The Community Earth System Model: A Framework for Collaborative Research. Bull Am Meteorol Soc 94, 1339\textendash{}1360. \sphinxurl{https://doi.org/10.1175/BAMS-D-12-00121.1} -Hurvich, C.M., Tsai, C.\sphinxhyphen{}L., 1989. Regression and time series model selection in small samples. Biometrika 76, 297\textendash{}307. -Iooss, B., Boussouf, L., Feuillard, V., Marrel, A., 2010. Numerical studies of the metamodel fitting and validation processes. Int. J. Adv. Syst. Meas. 3, 11\textendash{}21. -Jaynes, E.T., 2003. Probability Theory: The Logic of Science. Cambridge University Press. -Jin, R., Chen, W., Sudjianto, A., 2008. An Efficient Algorithm for Constructing Optimal Design of Computer Experiments. Presented at the ASME 2003 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers Digital Collection, pp. 545\textendash{}554. \sphinxurl{https://doi.org/10.1115/DETC2003/DAC-48760} -Johnson, D.R., Fischbach, J.R., Ortiz, D.S., 2013. Estimating Surge\sphinxhyphen{}Based Flood Risk with the Coastal Louisiana Risk Assessment Model. J Coast Res 109\textendash{}126. \sphinxurl{https://doi.org/10.2112/si\_67\_8} -Kandlikar, M., Risbey, J., Dessai, S., 2005. Representing and communicating deep uncertainty in climate\sphinxhyphen{}change assessments. C R Geosci 337, 443\textendash{}455. -Kasmalkar, I.G., Serafin, K.A., Miao, Y., Bick, I.A., Ortolano, L., Ouyang, D., Suckale, J., 2020. When floods hit the road: Resilience to flood\sphinxhyphen{}related traffic disruption in the San Francisco Bay Area and beyond. Sci Adv 6, eaba2423. \sphinxurl{https://doi.org/10.1126/sciadv.aba2423} -Kass, R.E., Raftery, A.E., 1995. Bayes Factors. J Am Stat Assoc 90, 773\textendash{}795. \sphinxurl{https://doi.org/10.1080/01621459.1995.10476572} -Keller, K., McInerney, D., 2008. The dynamics of learning about a climate threshold. Clim Dyn 30, 321\textendash{}332. \sphinxurl{https://doi.org/10.1007/s00382-007-0290-5} -Kennedy, M.C., O’Hagan, A., 2001. Bayesian calibration of computer models. J R Stat Soc Ser. B Stat Methodol 63, 425\textendash{}464. \sphinxurl{https://doi.org/10.1111/1467-9868.00294} -Kirsch, B.R., Characklis, G.W., Zeff, H.B., 2013. Evaluating the Impact of Alternative Hydro\sphinxhyphen{}Climate Scenarios on Transfer Agreements: Practical Improvement for Generating Synthetic Streamflows. J. Water Resour. Plan. Manag. 139, 396\textendash{}406. \sphinxurl{https://doi.org/10.1061/(ASCE)WR.1943-5452.0000287} -Kleindorfer, G.B., O’Neill, L., Ganeshan, R., 1998. Validation in simulation: Various positions in the philosophy of science. Manag. Sci. 44, 1087\textendash{}1099. -Koutsoyiannis, D., 2004. Statistics of extremes and estimation of extreme rainfall: I. Theoretical investigation/Statistiques de valeurs extrêmes et estimation de précipitations extrêmes: I. Recherche théorique. Hydrol. Sci. J. 49. -Kraan, B.C., Cooke, R.M., 2000. Uncertainty in compartmental models for hazardous materials \sphinxhyphen{} a case study. J Hazard Mater 71, 253\textendash{}268. \sphinxurl{https://doi.org/10.1016/S0304-3894(99)00082-5} -Kucherenko, S., Albrecht, D., Saltelli, A., 2015. Exploring multi\sphinxhyphen{}dimensional spaces: a Comparison of Latin Hypercube and Quasi Monte Carlo Sampling Techniques. ArXiv150502350 Stat. -Kullback, S., Leibler, R.A., 1951. On Information and Sufficiency. Ann Math Stat 22, 79\textendash{}86. \sphinxurl{https://doi.org/10.1214/aoms/1177729694} -Kumar, P., 2011. Typology of hydrologic predictability. Water Resour. Res. 47. \sphinxurl{https://doi.org/10.1029/2010WR009769} -Kwakkel, J.H., Walker, W.E., Haasnoot, M., 2016. Coping with the Wickedness of Public Policy Problems: Approaches for Decision Making under Deep Uncertainty. J. Water Resour. Plan. Manag. 142, 01816001. \sphinxurl{https://doi.org/10.1061/(ASCE)WR.1943-5452.0000626} -Lamontagne, J.R., Reed, P.M., Link, R., Calvin, K.V., Clarke, L.E., Edmonds, J.A., 2018. Large Ensemble Analytic Framework for Consequence\sphinxhyphen{}Driven Discovery of Climate Change Scenarios. Earths Future 6, 488\textendash{}504. \sphinxurl{https://doi.org/10.1002/2017EF000701} -Lamontagne, J.R., Stedinger, J.R., 2018. Generating Synthetic Streamflow Forecasts with Specified Precision. J. Water Resour. Plan. Manag. 144, 04018007. \sphinxurl{https://doi.org/10.1061/(ASCE)WR.1943-5452.0000915} -Landeg, O., Whitman, G., Walker\sphinxhyphen{}Springett, K., Butler, C., Bone, A., Kovats, S., 2019. Coastal flooding and frontline health care services: challenges for flood risk resilience in the English health care system. J Health Serv Res Policy 24, 219\textendash{}228. \sphinxurl{https://doi.org/10.1177/1355819619840672} -Lee, B.S., Haran, M., Fuller, R., Pollard, D., Keller, K., 2020. A Fast Particle\sphinxhyphen{}Based Approach for Calibrating a 3\sphinxhyphen{}D Model of the Antarctic Ice Sheet. Ann. Appl. Stat. in the press. -Lee, B.S., Haran, M., Keller, K., 2017. Multidecadal Scale Detection Time for Potentially Increasing Atlantic Storm Surges in a Warming Climate. Geophys Res Lett 44, 10,617\sphinxhyphen{}10,623. \sphinxurl{https://doi.org/10.1002/2017GL074606} -Lempert, R.J., 2002. A new decision sciences for complex systems. Proc. Natl. Acad. Sci. 99, 7309\textendash{}7313. \sphinxurl{https://doi.org/10.1073/pnas.082081699} -Liu, J., West, M., 2001. Combined Parameter and State Estimation in Simulation\sphinxhyphen{}Based Filtering, in: Doucet, A., de Freitas, N., Gordon, N. (Eds.), Sequential Monte Carlo Methods in Practice. Springer New York, New York, NY, pp. 197\textendash{}223. \sphinxurl{https://doi.org/10.1007/978-1-4757-3437-9\_10} -Liu, X., Guillas, S., 2017. Dimension Reduction for Gaussian Process Emulation: An Application to the Influence of Bathymetry on Tsunami Heights. SIAMASA J Uncertain. Quantif. 5, 787\textendash{}812. \sphinxurl{https://doi.org/10.1137/16M1090648} -Lorenz, E.N., 1963. Deterministic Nonperiodic Flow. J Atmos Sci 20, 130\textendash{}141. \sphinxurl{https://doi.org/10.1175/1520-0469(1963})020\textless{}0130:DNF\textgreater{}2.0.CO;2 -Loucks, D.P., Beek, E. van, 2017. Water Resource Systems Planning and Management: An Introduction to Methods, Models, and Applications. Springer International Publishing. -Lukacs, P.M., Burnham, K.P., Anderson, D.R., 2010. Model selection bias and Freedman’s paradox. Ann Inst Stat Math 62, 117\textendash{}125. \sphinxurl{https://doi.org/10.1007/s10463-009-0234-4} -Makowski, D., Wallach, D., Tremblay, M., 2002. Using a Bayesian approach to parameter estimation; comparison of the GLUE and MCMC methods. Agronomie 22, 191\textendash{}203. -Maniyar, D.M., Cornford, D., Boukouvalas, A., 2007. Dimensionality Reduction in the Emulator Setting (No. NCRG/2007/005). Neural Computing Research Group. -Marchau, V.A.W.J., Walker, W.E., Bloemen, P.J.T.M., Popper, S.W. (Eds.), 2019. Decision Making under Deep Uncertainty: From Theory to Practice. Springer International Publishing. \sphinxurl{https://doi.org/10.1007/978-3-030-05252-2} -Martin, A.D., 2010. MCMCpack: Markov chain Monte Carlo (MCMC) Package. R package version 1.0\sphinxhyphen{}7. HttpCRAN R\sphinxhyphen{}Proj. Orgpackage MCMCpack. -Massmann, C., Wagener, T., Holzmann, H., 2014. A new approach to visualizing time\sphinxhyphen{}varying sensitivity indices for environmental model diagnostics across evaluation time\sphinxhyphen{}scales. Environ. Model. Softw. 51, 190\textendash{}194. \sphinxurl{https://doi.org/10.1016/j.envsoft.2013.09.033} -Matthews, P., 1993. A slowly mixing Markov chain with implications for Gibbs sampling. Stat Probab Lett 17, 231\textendash{}236. \sphinxurl{https://doi.org/10.1016/0167-7152(93)90172-F} -McInerney, D., Keller, K., 2008. Economically optimal risk reduction strategies in the face of uncertain climate thresholds. Clim Change 91, 29\textendash{}41. \sphinxurl{https://doi.org/10.1007/s10584-006-9137-z} -McKay, M.D., Beckman, R.J., Conover, W.J., 1979. A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code. Technometrics 21, 239\textendash{}245. \sphinxurl{https://doi.org/10.2307/1268522} -Medda, S., Bhar, K.K., 2019. Comparison of single\sphinxhyphen{}site and multi\sphinxhyphen{}site stochastic models for streamflow generation. Appl. Water Sci. 9, 67. -Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E., 1953. Equation of State Calculations by Fast Computing Machines. J Chem Phys 21, 1087\textendash{}1092. \sphinxurl{https://doi.org/10.1063/1.1699114} -Metropolis, N., Ulam, S., 1949. The Monte Carlo Method. J. Am. Stat. Assoc. 44, 335\textendash{}341. \sphinxurl{https://doi.org/10.1080/01621459.1949.10483310} -Milly, P.C.D., Betancourt, J., Falkenmark, M., Hirsch, R.M., Kundzewicz, Z.W., Lettenmaier, D.P., Stouffer, R.J., 2008. Stationarity is dead: Whither water management? Science 319, 573\textendash{}574. \sphinxurl{https://doi.org/10.1126/science.1151915} -Moallemi, E.A., Kwakkel, J., de Haan, F.J., Bryan, B.A., 2020a. Exploratory modeling for analyzing coupled human\sphinxhyphen{}natural systems under uncertainty. Glob. Environ. Change 65, 102186. \sphinxurl{https://doi.org/10.1016/j.gloenvcha.2020.102186} -Moallemi, E.A., Zare, F., Reed, P.M., Elsawah, S., Ryan, M.J., Bryan, B.A., 2020b. Structuring and evaluating decision support processes to enhance the robustness of complex human\textendash{}natural systems. Environ. Model. Softw. 123, 104551. \sphinxurl{https://doi.org/10.1016/j.envsoft.2019.104551} -Montgomery, D.C., 2017. Design and analysis of experiments. John Wiley \& Sons. -Morgan, M.G., Keith, D.W., 2008. Improving the way we think about projecting future energy use and emissions of carbon dioxide. Clim Change 90, 189\textendash{}215. \sphinxurl{https://doi.org/10.1007/s10584-008-9458-1} -Morris, M.D., 1991. Factorial Sampling Plans for Preliminary Computational Experiments. Technometrics 33, 161\textendash{}174. \sphinxurl{https://doi.org/10.1080/00401706.1991.10484804} -Morris, M.D., Mitchell, T.J., 1995. Exploratory designs for computational experiments. J. Stat. Plan. Inference 43, 381\textendash{}402. \sphinxurl{https://doi.org/10.1016/0378-3758(94)00035-T} -National Research Council, 2014. Convergence: facilitating transdisciplinary integration of life sciences, physical sciences, engineering, and beyond. National Academies Press, Washington, D.C., USA. -Naylor, T.H., Finger, J.M., 1967. Verification of computer simulation models. Manag. Sci. 14, B\sphinxhyphen{}92. -Neal, R.M., 2011. MCMC Using Hamiltonian Dynamics, in: Brooks, S., Gelman, A., Jones, G.L., Meng, X.\sphinxhyphen{}L. (Eds.), Handbook of Markov Chain Monte Carlo, Handbooks of Modern Statistical Methods. CRC Press, Boca Raton, FL, pp. 113\textendash{}162. -Nearing, G.S., Ruddell, B.L., Bennett, A.R., Prieto, C., Gupta, H.V., 2020. Does Information Theory Provide a New Paradigm for Earth Science? Hypothesis Testing. Water Resour. Res. 56, e2019WR024918. \sphinxurl{https://doi.org/10.1029/2019WR024918} -Nelder, J.A., Wedderburn, R.W.M., 1972. Generalized Linear Models. J R Stat Soc Ser A 135, 370\textendash{}384. \sphinxurl{https://doi.org/10.2307/2344614} -Norton, J., 2015. An introduction to sensitivity assessment of simulation models. Environ. Model. Softw. 69, 166\textendash{}174. \sphinxurl{https://doi.org/10.1016/j.envsoft.2015.03.020} -Oddo, P.C., Lee, B.S., Garner, G.G., Srikrishnan, V., Reed, P.M., Forest, C.E., Keller, K., 2017. Deep Uncertainties in Sea\sphinxhyphen{}Level Rise and Storm Surge Projections: Implications for Coastal Flood Risk Management. Risk Anal. \sphinxurl{https://doi.org/10.1111/risa.12888} -O’Neill, B.C., Crutzen, P., Grübler, A., Duong, M.H., Keller, K., Kolstad, C., Koomey, J., Lange, A., Obersteiner, M., Oppenheimer, M., Pepper, W., Sanderson, W., Schlesinger, M., Treich, N., Ulph, A., Webster, M., Wilson, C., 2006. Learning and climate change. Clim Policy 6, 585\textendash{}589. \sphinxurl{https://doi.org/10.1080/14693062.2006.9685623} -Oppenheimer, M., O’Neill, B.C., Webster, M., 2008. Negative learning. Clim Change 89, 155\textendash{}172. \sphinxurl{https://doi.org/10.1007/s10584-008-9405-1} -Oreskes, N., Shrader\sphinxhyphen{}Frechette, K., Belitz, K., 1994. Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences. Science 263, 641\textendash{}646. \sphinxurl{https://doi.org/10.1126/science.263.5147.641} -Park, J.\sphinxhyphen{}S., 1994. Optimal Latin\sphinxhyphen{}hypercube designs for computer experiments. J. Stat. Plan. Inference 39, 95\textendash{}111. \sphinxurl{https://doi.org/10.1016/0378-3758(94)90115-5} -Pianosi, F., Beven, K., Freer, J., Hall, J.W., Rougier, J., Stephenson, D.B., Wagener, T., 2016. Sensitivity analysis of environmental models: A systematic review with practical workflow. Environ. Model. Softw. 79, 214\textendash{}232. \sphinxurl{https://doi.org/10.1016/j.envsoft.2016.02.008} -Pollard, D., DeConto, R.M., 2012. Description of a hybrid ice sheet\sphinxhyphen{}shelf model, and application to Antarctica. Geosci. Model Dev. 5, 1273\textendash{}1295. \sphinxurl{https://doi.org/10.5194/gmd-5-1273-2012} -Pruett, W.A., Hester, R.L., 2016. The Creation of Surrogate Models for Fast Estimation of Complex Model Outcomes. PLOS ONE 11, e0156574. \sphinxurl{https://doi.org/10.1371/journal.pone.0156574} -Ragulina, G., Reitan, T., 2017. Generalized extreme value shape parameter and its nature for extreme precipitation using long time series and the Bayesian approach. Hydrol. Sci. J. 62, 863\textendash{}879. -Rakovec, O., Hill, M.C., Clark, M.P., Weerts, A.H., Teuling, A.J., Uijlenhoet, R., 2014. Distributed Evaluation of Local Sensitivity Analysis (DELSA), with application to hydrologic models. Water Resour. Res. 50, 409\textendash{}426. \sphinxurl{https://doi.org/10.1002/2013WR014063} -Rasmussen, D.J., Buchanan, M.K., Kopp, R.E., Oppenheimer, M., 2020. A flood damage allowance framework for coastal protection with deep uncertainty in sea‐level rise. Earths Future. \sphinxurl{https://doi.org/10.1029/2019EF001340} -Robert, C.P., 2007. The Bayesian choice: from decision\sphinxhyphen{}theoretic foundations to computational implementation, 2nd ed. Springer, New York. -Rohmer, J., Le Cozannet, G., Manceau, J.\sphinxhyphen{}C., 2019. Addressing ambiguity in probabilistic assessments of future coastal flooding using possibility distributions. Clim Change. \sphinxurl{https://doi.org/10.1007/s10584-019-02443-4} -Ruckert, K.L., Shaffer, G., Pollard, D., Guan, Y., Wong, T.E., Forest, C.E., Keller, K., 2017. Assessing the Impact of Retreat Mechanisms in a Simple Antarctic Ice Sheet Model Using Bayesian Calibration. PLoS One 12, e0170052. \sphinxurl{https://doi.org/10.1371/journal.pone.0170052} -Ruckert, K.L., Srikrishnan, V., Keller, K., 2019. Characterizing the deep uncertainties surrounding coastal flood hazard projections: A case study for Norfolk, VA. Sci Rep 9, 11373. \sphinxurl{https://doi.org/10.1038/s41598-019-47587-6} -Sacks, J., Welch, W.J., Mitchell, T.J., Wynn, H.P., 1989. Design and Analysis of Computer Experiments. Stat Sci 4, 409\textendash{}423. -Saltelli, A., 2002. Making best use of model evaluations to compute sensitivity indices. Comput. Phys. Commun. 145, 280\textendash{}297. -Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., Wu, Q., 2019. Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices. Environ. Model. Softw. 114, 29\textendash{}39. \sphinxurl{https://doi.org/10.1016/j.envsoft.2019.01.012} -Saltelli, A., Annoni, P., 2010. How to avoid a perfunctory sensitivity analysis. Environ. Model. Softw. 25, 1508\textendash{}1517. \sphinxurl{https://doi.org/10.1016/j.envsoft.2010.04.012} -Saltelli, A., Funtowicz, S., 2014. When all models are wrong. Issues Sci. Technol. 30, 79\textendash{}85. -Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S., 2008. Global Sensitivity Analysis: The Primer, 1 edition. ed. Wiley\sphinxhyphen{}Interscience, Chichester, England; Hoboken, NJ. -Saltelli, A., Stark, P.B., Becker, W., Stano, P., 2015. Climate models as economic guides scientific challenge or quixotic quest? Issues Sci. Technol. 31, 79\textendash{}84. -Saltelli, A., Tarantola, S., 2002. On the relative importance of input factors in mathematical models: safety assessment for nuclear waste disposal. J. Am. Stat. Assoc. 97, 702\textendash{}709. -Saltelli, A., Tarantola, S., Campolongo, F., Ratto, M., 2004. Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. John Wiley \& Sons. -Saltelli, A., Tarantola, S., Chan, K.P.\sphinxhyphen{}S., 1999. A Quantitative Model\sphinxhyphen{}Independent Method for Global Sensitivity Analysis of Model Output. Technometrics 41, 39\textendash{}56. \sphinxurl{https://doi.org/10.1080/00401706.1999.10485594} -Schlather, M., 2002. Models for stationary max\sphinxhyphen{}stable random fields. Extremes 5, 33\textendash{}44. -Schwarz, G., 1978. Estimating the Dimension of a Model. Ann Stat 6, 461\textendash{}464. \sphinxurl{https://doi.org/10.1214/aos/1176344136} -Shafii, M., Tolson, B., Matott, L.S., 2014. Uncertainty\sphinxhyphen{}based multi\sphinxhyphen{}criteria calibration of rainfall\sphinxhyphen{}runoff models: a comparative study. Stoch Env. Res Risk Assess 28, 1493\textendash{}1510. \sphinxurl{https://doi.org/10.1007/s00477-014-0855-x} -Shibata, R., 1989. Statistical Aspects of Model Selection, in: Willems, J.C. (Ed.), From Data to Model. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 215\textendash{}240. \sphinxurl{https://doi.org/10.1007/978-3-642-75007-6\_5} -Sibley Heather M., Vroman Noah D., Shewbridge Scott E., 2017. Quantitative Risk\sphinxhyphen{}Informed Design of Levees. Geo\sphinxhyphen{}Risk 2017 76\textendash{}90. \sphinxurl{https://doi.org/10.1061/9780784480717.008} -Sminchisescu, C., Welling, M., Hinton, G., 2003. A mode\sphinxhyphen{}hopping MCMC sampler. Technical Report CSRG\sphinxhyphen{}478, University of Toronto, submitted to Machine …. -Sobol, I.M., 2001. Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math. Comput. Simul. 55, 271\textendash{}280. \sphinxurl{https://doi.org/10.1016/S0378-4754(00)00270-6} -Sobol, I.M., 1976. Uniformly distributed sequences with an additional uniform property. USSR Comput. Math. Math. Phys. 16, 236\textendash{}242. -Sobol’, I.M., 1967. On the distribution of points in a cube and the approximate evaluation of integrals. USSR Comput. Math. Math. Phys. 7, 86\textendash{}112. \sphinxurl{https://doi.org/10.1016/0041-5553(67)90144-9} -Spear, R.C., Hornberger, G.M., 1980. Eutrophication in peel inlet—II. Identification of critical uncertainties via generalized sensitivity analysis. Water Res. 14, 43\textendash{}49. \sphinxurl{https://doi.org/10.1016/0043-1354(80)90040-8} -Spiegelhalter, D.J., Best, N.G., Carlin, B.P., Van Der Linde, A., 2002. Bayesian measures of model complexity and fit. J R Stat Soc Ser. B Stat Methodol 64, 583\textendash{}639. \sphinxurl{https://doi.org/10.1111/1467-9868.00353} -Srikrishnan, V, Alley, R.B., Keller, K., 2019. Investing in Science and Using the Results to Improve Climate Risk Management. EOS. -Srikrishnan, Vivek, Guan, Y., Tol, R.S.J., Keller, K., 2019. Fossil fuel resources, decarbonization, and economic growth drive the feasibility of Paris climate targets. ArXiv StatAP. -Sriver, R.L., Lempert, R.J., Wikman\sphinxhyphen{}Svahn, P., Keller, K., 2018. Characterizing uncertain sea\sphinxhyphen{}level rise projections to support investment decisions. PLoS One 13, e0190641. \sphinxurl{https://doi.org/10.1371/journal.pone.0190641} -Stedinger, J.R., Vogel, R.M., Lee, S.U., Batchelder, R., 2008. Appraisal of the generalized likelihood uncertainty estimation (GLUE) method: APPRAISAL OF THE GLUE METHOD. Water Resour Res 44, 191. \sphinxurl{https://doi.org/10.1029/2008WR006822} -Steinschneider, S., Wi, S., Brown, C., 2015. The integrated effects of climate and hydrologic uncertainty on future flood risk assessments. Hydrol. Process. 29, 2823\textendash{}2839. \sphinxurl{https://doi.org/10.1002/hyp.10409} -Stone, M., 1977. An Asymptotic Equivalence of Choice of Model by Cross\sphinxhyphen{}Validation and Akaike’s Criterion. J R Stat Soc Ser. B Stat Methodol 39, 44\textendash{}47. -Surowiec, I., Vikström, L., Hector, G., Johansson, E., Vikström, C., Trygg, J., 2017. Generalized Subset Designs in Analytical Chemistry. Anal. Chem. 89, 6491\textendash{}6497. \sphinxurl{https://doi.org/10.1021/acs.analchem.7b00506} -Tang, B., 1993. Orthogonal Array\sphinxhyphen{}Based Latin Hypercubes. J. Am. Stat. Assoc. 88, 1392\textendash{}1397. \sphinxurl{https://doi.org/10.1080/01621459.1993.10476423} -Tang, Y., Reed, P., Wagener, T., Werkhoven, K. van, 2007. Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation. Hydrol. Earth Syst. Sci. 11, 793\textendash{}817. \sphinxurl{https://doi.org/10.5194/hess-11-793-2007} -Toulmin, S., 1977. From form to function: philosophy and history of science in the 1950s and now. Daedalus 143\textendash{}162. -Urban, N.M., Keller, K., 2010. Probabilistic hindcasts and projections of the coupled climate, carbon cycle and Atlantic meridional overturning circulation system: A Bayesian fusion of century\sphinxhyphen{}scale observations with a simple model. Tellus A 62, 737\textendash{}750. -Van Oort, N., 2011. Service reliability and urban public transport design. -Van Schepdael, A., Carlier, A., Geris, L., 2016. Sensitivity Analysis by Design of Experiments, in: Geris, L., Gomez\sphinxhyphen{}Cabrero, D. (Eds.), Uncertainty in Biology: A Computational Modeling Approach, Studies in Mechanobiology, Tissue Engineering and Biomaterials. Springer International Publishing, Cham, pp. 327\textendash{}366. \sphinxurl{https://doi.org/10.1007/978-3-319-21296-8\_13} -van Vuuren, D.P., Edmonds, J., Kainuma, M., Riahi, K., Thomson, A., Hibbard, K., Hurtt, G.C., Kram, T., Krey, V., Lamarque, J.\sphinxhyphen{}F., Masui, T., Meinshausen, M., Nakicenovic, N., Smith, S.J., Rose, S.K., 2011. The representative concentration pathways: an overview. Clim Change 109, 5. \sphinxurl{https://doi.org/10.1007/s10584-011-0148-z} -Vega‐Westhoff, B., Sriver, R.L., Hartin, C.A., Wong, T.E., Keller, K., 2019. Impacts of Observational Constraints Related to Sea Level on Estimates of Climate Sensitivity. Earths Future. \sphinxurl{https://doi.org/10.1029/2018EF001082} -Vehtari, A., Gelman, A., Gabry, J., 2017. Practical Bayesian model evaluation using leave\sphinxhyphen{}one\sphinxhyphen{}out cross\sphinxhyphen{}validation and WAIC. Stat Comput 27, 1413\textendash{}1432. -Vihola, M., 2012. Robust adaptive Metropolis algorithm with coerced acceptance rate. Stat Comput 22, 997\textendash{}1008. \sphinxurl{https://doi.org/10.1007/s11222-011-9269-5} -Vogel, R.M., 2017. Stochastic watershed models for hydrologic risk management. Water Secur. 1, 28\textendash{}35. \sphinxurl{https://doi.org/10.1016/j.wasec.2017.06.001} -Vogel, R.M., Stedinger, J.R., 1988. The value of stochastic streamflow models in overyear reservoir design applications. Water Resour. Res. 24, 1483\textendash{}1490. \sphinxurl{https://doi.org/10.1029/WR024i009p01483} -Vousdoukas, M.I., Bouziotas, D., Giardino, A., Bouwer, L.M., Mentaschi, L., Voukouvalas, E., Feyen, L., 2018. Understanding epistemic uncertainty in large\sphinxhyphen{}scale coastal flood risk assessment for present and future climates. Nat Hazards Earth Syst Sci 18, 2127\textendash{}2142. \sphinxurl{https://doi.org/10.5194/nhess-18-2127-2018} -Vrugt, J.A., Beven, K.J., 2018. Embracing equifinality with efficiency: Limits of Acceptability sampling using the DREAM(LOA) algorithm. J Hydrol 559, 954\textendash{}971. \sphinxurl{https://doi.org/10.1016/j.jhydrol.2018.02.026} -Walker, W., Haasnoot, M., Kwakkel, J., Walker, W.E., Haasnoot, M., Kwakkel, J.H., 2013. Adapt or Perish: A Review of Planning Approaches for Adaptation under Deep Uncertainty. Sustainability 5, 955\textendash{}979. \sphinxurl{https://doi.org/10.3390/su5030955} -Walker, W.E., Harremoës, P., Rotmans, J., Sluijs, J.P. van der, Asselt, M.B.A. van, Janssen, P., Krauss, M.P.K. von, 2003. Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model\sphinxhyphen{}Based Decision Support. Integr. Assess. 4, 5\textendash{}17. \sphinxurl{https://doi.org/10.1076/iaij.4.1.5.16466} -Walker, W.E., Lempert, R.J., Kwakkel, J.H., 2013. Deep Uncertainty, in: Gass, S.I., Fu, M.C. (Eds.), Encyclopedia of Operations Research and Management Science. Springer US, pp. 395\textendash{}402. \sphinxurl{https://doi.org/10.1007/978-1-4419-1153-7\_1140} -Watanabe, S., 2010. Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory. J Mach Learn Res 11, 3571\textendash{}3594. -Weaver, C.P., Lempert, R.J., Brown, C., Hall, J.A., Revell, D., Sarewitz, D., 2013. Improving the contribution of climate model information to decision making: the value and demands of robust decision frameworks. Wiley Interdiscip. Rev. Clim. Change 4, 39\textendash{}60. \sphinxurl{https://doi.org/10.1002/wcc.202} -White, D.D., Wutich, A., Larson, K.L., Gober, P., Lant, T., Senneville, C., 2010. Credibility, salience, and legitimacy of boundary objects: water managers’ assessment of a simulation model in an immersive decision theater. Sci. Public Policy 37, 219\textendash{}232. \sphinxurl{https://doi.org/10.3152/030234210X497726} -Wilks, D.S., Wilby, R.L., 1999. The weather generation game: a review of stochastic weather models. Prog. Phys. Geogr. 23, 329\textendash{}357. -Wirtz, D., Nowak, W., 2017. The rocky road to extended simulation frameworks covering uncertainty, inversion, optimization and control. Environ. Model. Softw. 93, 180\textendash{}192. \sphinxurl{https://doi.org/10.1016/j.envsoft.2016.10.003} -Wolff, C., Nikoletopoulos, T., Hinkel, J., Vafeidis, A.T., 2020. Future urban development exacerbates coastal exposure in the Mediterranean. Sci Rep 10, 14420. \sphinxurl{https://doi.org/10.1038/s41598-020-70928-9} -Wong, T.E., Keller, K., 2017. Deep Uncertainty Surrounding Coastal Flood Risk Projections: A Case Study for New Orleans. Earths Future 5, 1015\textendash{}1026. \sphinxurl{https://doi.org/10.1002/2017EF000607} -Wong, T.E., Klufas, A., Srikrishnan, V., Keller, K., 2018. Neglecting model structural uncertainty underestimates upper tails of flood hazard. Env. Res Lett 13, 074019. \sphinxurl{https://doi.org/10.1088/1748-9326/aacb3d} -Xian, S., Lin, N., Kunreuther, H., 2017. Optimal house elevation for reducing flood\sphinxhyphen{}related losses. J Hydrol 548, 63\textendash{}74. \sphinxurl{https://doi.org/10.1016/j.jhydrol.2017.02.057} -Xiu, D., Karniadakis, G.E., 2002. The Wiener\textendash{}Askey Polynomial Chaos for Stochastic Differential Equations. SIAM J. Sci. Comput. 24, 619\textendash{}644. \sphinxurl{https://doi.org/10.1137/S1064827501387826} -Zarekarizi, M., Srikrishnan, V., Keller, K., 2020. Neglecting Uncertainties Biases House\sphinxhyphen{}Elevation Decisions to Manage Riverine Flood Risks. Nat Commun. \sphinxurl{https://doi.org/10.1038/s41467-020-19188-9} -Zaremba, S.K., 1968. The mathematical basis of Monte Carlo and quasi\sphinxhyphen{}Monte Carlo methods. SIAM Rev. 10, 303\textendash{}314. -Zellner, A., Tian, G.C., 1964. Bayesian analysis of the regression model with autocorrelated errors. J Am Stat Assoc 763\textendash{}778. -Zhang, X., Che, L., Shahidehpour, M., Alabdulwahab, A.S., Abusorrah, A., 2015. Reliability\sphinxhyphen{}based optimal planning of electricity and natural gas interconnections for multiple energy hubs. IEEE Trans Smart Grid 8, 1658\textendash{}1667. -Zscheischler, J., Martius, O., Westra, S., Bevacqua, E., Raymond, C., Horton, R.M., van den Hurk, B., AghaKouchak, A., Jézéquel, A., Mahecha, M.D., Maraun, D., Ramos, A.M., Ridder, N.N., Thiery, W., Vignotto, E., 2020. A typology of compound weather and climate events. Nat. Rev. Earth Environ. 1\textendash{}15. \sphinxurl{https://doi.org/10.1038/s43017-020-0060-z} +\chapter{Glossary} +\label{\detokenize{8_glossary:glossary}}\label{\detokenize{8_glossary::doc}} +\sphinxAtStartPar -\chapter{Glossary} -\label{\detokenize{9_glossary:glossary}}\label{\detokenize{9_glossary::doc}} \chapter{Indices and tables} \label{\detokenize{index:indices-and-tables}}\begin{itemize} @@ -766,6 +545,45 @@ \chapter{Indices and tables} \end{itemize} +\begin{sphinxthebibliography}{10} +\bibitem[1]{index:id11} +\sphinxAtStartPar +Roger Cooke and others. \sphinxstyleemphasis{Experts in uncertainty: opinion and subjective probability in science}. Oxford University Press on Demand, 1991. +\bibitem[2]{index:id2} +\sphinxAtStartPar +National Research Council and others. \sphinxstyleemphasis{Convergence: Facilitating transdisciplinary integration of life sciences, physical sciences, engineering, and beyond}. National Academies Press, 2014. +\bibitem[3]{index:id3} +\sphinxAtStartPar +Sondoss Elsawah, Tatiana Filatova, Anthony J Jakeman, Albert J Kettner, Moira L Zellner, Ioannis N Athanasiadis, Serena H Hamilton, Robert L Axtell, Daniel G Brown, Jonathan M Gilligan, and others. Eight grand challenges in socio\sphinxhyphen{}environmental systems modeling. \sphinxstyleemphasis{Socio\sphinxhyphen{}Environmental Systems Modelling}, 2:16226\textendash{}16226, 2020. +\bibitem[4]{index:id9} +\sphinxAtStartPar +Saul I Gass and Carl M Harris. Encyclopedia of operations research and management science. \sphinxstyleemphasis{Journal of the Operational Research Society}, 48(7):759\textendash{}760, 1997. +\bibitem[5]{index:id4} +\sphinxAtStartPar +Yacov Y Haimes. Risk modeling of interdependent complex systems of systems: theory and practice. \sphinxstyleemphasis{Risk analysis}, 38(1):84\textendash{}98, 2018. +\bibitem[6]{index:id5} +\sphinxAtStartPar +Dirk Helbing. Globally networked risks and how to respond. \sphinxstyleemphasis{Nature}, 497(7447):51\textendash{}59, 2013. +\bibitem[7]{index:id8} +\sphinxAtStartPar +Jan H Kwakkel, Warren E Walker, and Marjolijn Haasnoot. Coping with the wickedness of public policy problems: approaches for decision making under deep uncertainty. 2016. +\bibitem[8]{index:id12} +\sphinxAtStartPar +Enayat A Moallemi, Jan Kwakkel, Fjalar J de Haan, and Brett A Bryan. Exploratory modeling for analyzing coupled human\sphinxhyphen{}natural systems under uncertainty. \sphinxstyleemphasis{Global Environmental Change}, 65:102186, 2020. +\bibitem[9]{index:id6} +\sphinxAtStartPar +Andrea Saltelli, Ksenia Aleksankina, William Becker, Pamela Fennell, Federico Ferretti, Niels Holst, Sushan Li, and Qiongli Wu. Why so many published sensitivity analyses are false: a systematic review of sensitivity analysis practices. \sphinxstyleemphasis{Environmental modelling \& software}, 114:29\textendash{}39, 2019. +\bibitem[10]{index:id10} +\sphinxAtStartPar +Andrea Saltelli, Philip B Stark, William Becker, and Pawel Stano. Climate models as economic guides scientific challenge or quixotic quest? \sphinxstyleemphasis{Issues in Science and Technology}, 31(3):79\textendash{}84, 2015. +\bibitem[11]{index:id13} +\sphinxAtStartPar +Warren E Walker, Poul Harremoës, Jan Rotmans, Jeroen P Van Der Sluijs, Marjolein BA Van Asselt, Peter Janssen, and Martin P Krayer von Krauss. Defining uncertainty: a conceptual basis for uncertainty management in model\sphinxhyphen{}based decision support. \sphinxstyleemphasis{Integrated assessment}, 4(1):5\textendash{}17, 2003. +\bibitem[12]{index:id7} +\sphinxAtStartPar +Daniel Wirtz and Wolfgang Nowak. The rocky road to extended simulation frameworks covering uncertainty, inversion, optimization and control. \sphinxstyleemphasis{Environmental Modelling \& Software}, 93:180\textendash{}192, 2017. +\end{sphinxthebibliography} + \renewcommand{\indexname}{Index} diff --git a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.toc b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.toc index 217f8c1..f5ce432 100644 --- a/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.toc +++ b/docs/build/latex/addressinguncertaintyinmultisectordynamicsresearch.toc @@ -54,6 +54,6 @@ \contentsline {subsection}{\numberline {6.8.1}Markov Chain Monte Carlo with the True Model}{25}{subsection.6.8.1}% \contentsline {subsection}{\numberline {6.8.2}Markov Chain Monte Carlo with Surrogate Models}{25}{subsection.6.8.2}% \contentsline {chapter}{\numberline {7}Conclusion}{27}{chapter.7}% -\contentsline {chapter}{\numberline {8}References}{29}{chapter.8}% -\contentsline {chapter}{\numberline {9}Glossary}{37}{chapter.9}% -\contentsline {chapter}{\numberline {10}Indices and tables}{39}{chapter.10}% +\contentsline {chapter}{\numberline {8}Glossary}{29}{chapter.8}% +\contentsline {chapter}{\numberline {9}Indices and tables}{31}{chapter.9}% +\contentsline {chapter}{Bibliography}{33}{chapter*.3}% diff --git a/docs/build/latex/im3.png b/docs/build/latex/im3.png deleted file mode 100644 index de17482..0000000 Binary files a/docs/build/latex/im3.png and /dev/null differ diff --git a/docs/source/1_introduction.rst b/docs/source/1_introduction.rst index affc935..db535fa 100644 --- a/docs/source/1_introduction.rst +++ b/docs/source/1_introduction.rst @@ -12,9 +12,9 @@ This guidance text has been developed in support of the Integrated Multisector M *Understand the implications of uncertainty in data, observations, models, and model coupling approaches for projections of human-natural system dynamics.* -Addressing the objectives above poses a strong transdisciplinary challenge that heavily depends on a diversity of models and, more specifically, a consistent framing for making model-based science inferences. The term transdisciplinary science as used here formally implies a deep integration of disciplines to aid our hypothesis driven understanding of coupled human-natural systems--bridging differences in theory, hypothesis generation, modeling, and modes of inference (National Research Council, 2014). The IM3 MSD research focus and questions require a deep integration across disciplines, where new modes of analysis can emerge that rapidly synthesize and exploit advances for making decision relevant insights that at minimum acknowledge uncertainty and more ideally promote a rigorous quantitative mapping of its effects on the generality of claimed scientific insights. More broadly, diverse scientific disciplines engaged in the science of coupled human-natural systems, ranging from natural sciences to engineering and economics, employ a diversity of numerical computer models to study and understand their underlying systems of focus. The utility of these computer models hinges on their ability to represent the underlying real systems with sufficient fidelity and enable the inference of novel insights. This is particularly challenging in the case of coupled human-natural systems where there exists a multitude of interdependent human and natural processes taking place that could potentially be represented. These processes usually translate into modeled representations that are highly complex, non-linear, and exhibit strong interactions and threshold behaviors (Elsawah et al., 2020; Haimes, 2018; Helbing, 2013). Model complexity and detail have also been increasing as a result of our improving understanding of these processes, the availability of data, and the rapid growth in computing power (Saltelli et al., 2019). As model complexity grows, modelers need to specify a lot more information than before: additional model inputs and relationships as more processes are represented, higher resolution data as more observations are collected, new coupling relationships and interactions as models are put together to answer multisector questions (e.g., the land-water-energy nexus). Typically, not all of this information is well known, nor is the impact of these many uncertainties on model outputs well understood. It is further especially difficult to distinguish the effects of individual as well as interacting sources of uncertainty when modeling coupled systems with multisector and multiscale dynamics (Wirtz and Nowak, 2017). +Addressing the objectives above poses a strong transdisciplinary challenge that heavily depends on a diversity of models and, more specifically, a consistent framing for making model-based science inferences. The term transdisciplinary science as used here formally implies a deep integration of disciplines to aid our hypothesis driven understanding of coupled human-natural systems--bridging differences in theory, hypothesis generation, modeling, and modes of inference :cite:p:`national2014convergence`. The IM3 MSD research focus and questions require a deep integration across disciplines, where new modes of analysis can emerge that rapidly synthesize and exploit advances for making decision relevant insights that at minimum acknowledge uncertainty and more ideally promote a rigorous quantitative mapping of its effects on the generality of claimed scientific insights. More broadly, diverse scientific disciplines engaged in the science of coupled human-natural systems, ranging from natural sciences to engineering and economics, employ a diversity of numerical computer models to study and understand their underlying systems of focus. The utility of these computer models hinges on their ability to represent the underlying real systems with sufficient fidelity and enable the inference of novel insights. This is particularly challenging in the case of coupled human-natural systems where there exists a multitude of interdependent human and natural processes taking place that could potentially be represented. These processes usually translate into modeled representations that are highly complex, non-linear, and exhibit strong interactions and threshold behaviors :cite:p:`elsawah2020eight,haimes2018risk,helbing2013globally`. Model complexity and detail have also been increasing as a result of our improving understanding of these processes, the availability of data, and the rapid growth in computing power :cite:p:`saltelli2019so`. As model complexity grows, modelers need to specify a lot more information than before: additional model inputs and relationships as more processes are represented, higher resolution data as more observations are collected, new coupling relationships and interactions as models are put together to answer multisector questions (e.g., the land-water-energy nexus). Typically, not all of this information is well known, nor is the impact of these many uncertainties on model outputs well understood. It is further especially difficult to distinguish the effects of individual as well as interacting sources of uncertainty when modeling coupled systems with multisector and multiscale dynamics :cite:p:`wirtz2017rocky`. -Given the challenge and opportunity posed by the disciplinary diversity of IM3, we utilized a team-wide survey to allow the project’s membership to provide their views on how their areas typically address uncertainty, emphasizing key literature examples and domain-specific reviews. Our synthesis of this survey information in Figure 1 summaries the team’s perspectives, enabling a summary of the commonalities and differences for how different disciplinary areas are typically addressing uncertainty. Figure 1 highlights the non-trivial challenge posed by seeking to carefully consider uncertainty across an MSD focused transdisciplinary team. There are significant differences across the team’s contributing disciplines in terms of the methodological approaches and tools used in the treatment of uncertainty. The horizontal axis of the figure represents a conceptual continuum of methodological approaches, ranging from deterministic (no uncertainty) modeling to the theoretical case of fully engaging in modeling all sources of uncertainty. The vertical axis of plot maps the analysis tools that are used in the disciplines’ literature, spanning error-driven historical analyses to full uncertainty quantification. Given that Figure 1 is a conceptual illustration, the mapping of each discipline’s boundaries is not meant to imply exactness. They encompass the scope of feedback attained in the team-wide survey responses. The color circles designate specific sources of uncertainty that could be considered. Within the mapped disciplinary approaches, the color circles distinguish those sources of uncertainty that are addressed in the bodies of literature reported by respondents. Note the complete absence of grey circles designating that at present few if any studies report results for understanding how model coupling relationships shape uncertainty. We can briefly distinguish the key terms of uncertainty quantification (UQ) and uncertainty characterization (UC). UQ refers to the formal focus on the full specification of likelihoods as well as distributional forms necessary to infer the joint probabilistic response across all modeled factors of interest (Cooke, 1991). Alternatively, uncertainty characterization as defined here, refers to exploratory modeling of alternative hypotheses for the co-evolutionary dynamics of influences, stressors, as well as path dependent changes in the form and function of modelled systems (Moallemi et al., 2020a; Walker et al., 2003). Uncertain factors are any model component which is affected by uncertainty: inputs, resolution levels, coupling relationships, model relationships and parameters. When a model has been established as a sufficiently accurate representation of the system some of these factors may reflect elements of the real-world system that the model represents (for example, a population level parameter would reflect a sufficiently accurate representation of the population level in the system under study). As discussed in later sections, the choice of UQ or UC depends on the specific goals of studies, the availability of data, the types of uncertainties (e.g., well-characterized or deep), the complexity of underlying models as well as the computational limits. Deep uncertainty (as opposed to well-characterized) refers to situations where expert opinions consulted on a decision do not know or cannot agree on system boundaries, or the outcomes of interest and their relative importance, or the prior probability distribution for the various uncertain factors present (Kwakkel et al., 2016; W. E. Walker et al., 2013). +Given the challenge and opportunity posed by the disciplinary diversity of IM3, we utilized a team-wide survey to allow the project’s membership to provide their views on how their areas typically address uncertainty, emphasizing key literature examples and domain-specific reviews. Our synthesis of this survey information in Figure 1 summaries the team’s perspectives, enabling a summary of the commonalities and differences for how different disciplinary areas are typically addressing uncertainty. Figure 1 highlights the non-trivial challenge posed by seeking to carefully consider uncertainty across an MSD focused transdisciplinary team. There are significant differences across the team’s contributing disciplines in terms of the methodological approaches and tools used in the treatment of uncertainty. The horizontal axis of the figure represents a conceptual continuum of methodological approaches, ranging from deterministic (no uncertainty) modeling to the theoretical case of fully engaging in modeling all sources of uncertainty. The vertical axis of plot maps the analysis tools that are used in the disciplines’ literature, spanning error-driven historical analyses to full uncertainty quantification. Given that Figure 1 is a conceptual illustration, the mapping of each discipline’s boundaries is not meant to imply exactness. They encompass the scope of feedback attained in the team-wide survey responses. The color circles designate specific sources of uncertainty that could be considered. Within the mapped disciplinary approaches, the color circles distinguish those sources of uncertainty that are addressed in the bodies of literature reported by respondents. Note the complete absence of grey circles designating that at present few if any studies report results for understanding how model coupling relationships shape uncertainty. We can briefly distinguish the key terms of uncertainty quantification (UQ) and uncertainty characterization (UC). UQ refers to the formal focus on the full specification of likelihoods as well as distributional forms necessary to infer the joint probabilistic response across all modeled factors of interest :cite:p:`cooke1991experts`. Alternatively, uncertainty characterization as defined here, refers to exploratory modeling of alternative hypotheses for the co-evolutionary dynamics of influences, stressors, as well as path dependent changes in the form and function of modelled systems :cite:p:`moallemi2020exploratory,walker2003defining`. Uncertain factors are any model component which is affected by uncertainty: inputs, resolution levels, coupling relationships, model relationships and parameters. When a model has been established as a sufficiently accurate representation of the system some of these factors may reflect elements of the real-world system that the model represents (for example, a population level parameter would reflect a sufficiently accurate representation of the population level in the system under study). As discussed in later sections, the choice of UQ or UC depends on the specific goals of studies, the availability of data, the types of uncertainties (e.g., well-characterized or deep), the complexity of underlying models as well as the computational limits. Deep uncertainty (as opposed to well-characterized) refers to situations where expert opinions consulted on a decision do not know or cannot agree on system boundaries, or the outcomes of interest and their relative importance, or the prior probability distribution for the various uncertain factors present :cite:p:`kwakkel2016coping,gass1997encyclopedia`. .. figure:: _static/figure1_state_of_the_science.png :alt: Figure 1 @@ -23,4 +23,4 @@ Given the challenge and opportunity posed by the disciplinary diversity of IM3, State-of-the-art in different modeling communities, as reported in the survey distributed to IM3 teams. Deterministic Historical Evaluation: model evaluation under fully determined conditions defined using historical observations; Local Sensitivity Analysis: model evaluation performed by varying uncertain factors around specific reference values; Global Sensitivity Analysis: model evaluation performed by varying uncertain factors throughout their entire feasible value space; Uncertainty Characterization: model evaluation under alternative factor hypotheses to explore their implications for model output uncertainty; Uncertainty Quantification: representation of model output uncertainty using probability distributions; Traditional statistical inference: use of analysis results to describe deterministic or probabilistic outcomes resulting from the presence of uncertainty; Narrative scenarios: use of a limited decision-relevant number of scenarios to describe (sets of) changing system outcomes; Exploratory modeling for scenario discovery: use of large ensembles of uncertain conditions to discover decision-relevant combinations of uncertain factors -At present, there is no singular guide for confronting the computational and conceptual challenges of the multi-model, transdisciplinary workflows that characterize ambitious projects such as IM3 (Saltelli et al., 2015). The primary aim of this text is to begin to address this gap and provide guidance for facing these challenges. Chapter 2 provides an overview of diagnostic modeling and the different perspectives for how we should evaluate our models. Chapter 3 the basic methods and concepts for sensitivity analysis. Chapter 4 delves into more technical applications of sensitivity analysis to support diagnostic model evaluation and exploratory modeling. Chapter 5 transitions to an overview of the key concepts and tools for UQ. Chapter 6 transitions to the use of UQ to capture risks and extremes in MSD systems. Chapter 7 provides concluding remarks across the UC and UQ topics covered in this text. The appendices of this text include a glossary of the key concepts as well as example test cases and scripts to showcase various UC and UQ related tools. +At present, there is no singular guide for confronting the computational and conceptual challenges of the multi-model, transdisciplinary workflows that characterize ambitious projects such as IM3 :cite:p:`saltelli2015climate`. The primary aim of this text is to begin to address this gap and provide guidance for facing these challenges. Chapter 2 provides an overview of diagnostic modeling and the different perspectives for how we should evaluate our models. Chapter 3 the basic methods and concepts for sensitivity analysis. Chapter 4 delves into more technical applications of sensitivity analysis to support diagnostic model evaluation and exploratory modeling. Chapter 5 transitions to an overview of the key concepts and tools for UQ. Chapter 6 transitions to the use of UQ to capture risks and extremes in MSD systems. Chapter 7 provides concluding remarks across the UC and UQ topics covered in this text. The appendices of this text include a glossary of the key concepts as well as example test cases and scripts to showcase various UC and UQ related tools. diff --git a/docs/source/9_glossary.rst b/docs/source/8_glossary.rst similarity index 100% rename from docs/source/9_glossary.rst rename to docs/source/8_glossary.rst diff --git a/docs/source/8_references.rst b/docs/source/8_references.rst deleted file mode 100644 index 9a5f3f7..0000000 --- a/docs/source/8_references.rst +++ /dev/null @@ -1,223 +0,0 @@ -********** -References -********** - -Akaike, H., 1978. On the Likelihood of a Time Series Model. J. R. Stat. Soc. Ser. Stat. 27, 217–235. https://doi.org/10.2307/2988185 -Akaike, H., 1974. A new look at the statistical model identification. IEEE Trans. Autom. Control 19, 716–723. https://doi.org/10.1109/TAC.1974.1100705 -Anderson, B., Borgonovo, E., Galeotti, M., Roson, R., 2014. Uncertainty in Climate Change Modeling: Can Global Sensitivity Analysis Be of Help? Risk Anal. 34, 271–293. https://doi.org/10.1111/risa.12117 -Annan, J.D., Hargreaves, J.C., 2004. Efficient parameter estimation for a highly chaotic system. Tellus A 56, 520–526. https://doi.org/10.1111/j.1600-0870.2004.00073.x -Bakker, A.M.R., Wong, T.E., Ruckert, K.L., Keller, K., 2017. Sea-level projections representing the deeply uncertain contribution of the West Antarctic ice sheet. Sci Rep 7, 3880. https://doi.org/10.1038/s41598-017-04134-5 -Bankes, S., 1993. Exploratory Modeling for Policy Analysis. Oper. Res. 41, 435–449. https://doi.org/10.1287/opre.41.3.435 -Barlas, Y., 1996. Formal aspects of model validity and validation in system dynamics. Syst. Dyn. Rev. J. Syst. Dyn. Soc. 12, 183–210. -Barlas, Y., Carpenter, S., 1990. Philosophical roots of model validation: Two paradigms. Syst. Dyn. Rev. 6, 148–166. https://doi.org/10.1002/sdr.4260060203 -Bastos, L.S., O’Hagan, A., 2009. Diagnostics for Gaussian Process Emulators. Technometrics 51, 425–438. https://doi.org/10.1198/TECH.2009.08019 -Batouli, M., Mostafavi, A., 2018. Multiagent Simulation for Complex Adaptive Modeling of Road Infrastructure Resilience to Sea-Level Rise. Comput.-Aided Civ. Infrastruct. Eng. 33, 393–410. https://doi.org/10.1111/mice.12348 -Bayes, T., 1763. An Essay towards Solving a Problem in the Doctrine of Chance. Philos. Trans. R. Soc. Lond. 53, 370–418. -Betancourt, M., 2017. A Conceptual Introduction to Hamiltonian Monte Carlo. ArXiv StatME. -Beven, K., 2002. Towards a coherent philosophy for modelling the environment. Proc. R. Soc. Lond. Ser. Math. Phys. Eng. Sci. 458, 2465–2484. https://doi.org/10.1098/rspa.2002.0986 -Beven, K., 1993. Prophecy, reality and uncertainty in distributed hydrological modelling. Adv. Water Resour., Research Perspectives in Hydrology 16, 41–51. https://doi.org/10.1016/0309-1708(93)90028-E -Beven, K., Binley, A., 1992. The future of distributed models: Model calibration and uncertainty prediction. Hydrol. Process. 6, 279–298. https://doi.org/10.1002/hyp.3360060305 -Beven, K., Freer, J., 2001. Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology. J Hydrol. -Beven, K.J., 2018. On hypothesis testing in hydrology: Why falsification of models is still a really good idea. WIREs Water 5, e1278. https://doi.org/10.1002/wat2.1278 -Bobée, B., 1975. The Log Pearson type 3 distribution and its application in hydrology. Water Resour Res 11, 681–689. https://doi.org/10.1029/WR011i005p00681 -Bobée, B., Rasmussen, P.F., 1995. Recent advances in flood frequency analysis. Rev Geophys 33, 1111–1116. https://doi.org/10.1029/95RG00287 -Borgomeo, E., Farmer, C.L., Hall, J.W., 2015. Numerical rivers: A synthetic streamflow generator for water resources vulnerability assessments. Water Resour. Res. 51, 5382–5405. https://doi.org/10.1002/2014WR016827 -Borgonovo, E., 2010. Sensitivity analysis with finite changes: An application to modified EOQ models. Eur. J. Oper. Res. 200, 127–138. -Borgonovo, E., Plischke, E., 2016. Sensitivity analysis: A review of recent advances. Eur. J. Oper. Res. 248, 869–887. https://doi.org/10.1016/j.ejor.2015.06.032 -Boukouvalas, A., Sykes, P., Cornford, D., Maruri-Aguilar, H., 2014. Bayesian Precalibration of a Large Stochastic Microsimulation Model. IEEE Trans Intell Transp Syst 15, 1337–1347. https://doi.org/10.1109/tits.2014.2304394 -Box, G.E., Hunter, J.S., 1961. The 2 k—p fractional factorial designs. Technometrics 3, 311–351. -Bryant, B.P., Lempert, R.J., 2010. Thinking inside the box: A participatory, computer-assisted approach to scenario discovery. Technol. Forecast. Soc. Change 77, 34–49. https://doi.org/10.1016/j.techfore.2009.08.002 -Budescu, D.V., Broomell, S.B., Lempert, R.J., Keller, K., 2014. Aided and unaided decisions with imprecise probabilities in the domain of losses. EURO J. Decis. Process. 2, 31–62. https://doi.org/10.1007/s40070-013-0023-4 -Bürkner, P.-C., Gabry, J., Vehtari, A., 2019. Approximate leave-future-out cross-validation for Bayesian time series models. ArXiv StatME. -Burnham, K.P., Anderson, D.R., 2004. Multimodel Inference: Understanding AIC and BIC in Model Selection. Sociol Methods Res 33, 261–304. https://doi.org/10.1177/0049124104268644 -Burnham, K.P., Anderson, D.R., 2001. Kullback-Leibler information as a basis for strong inference in ecological studies. Wildl Res 28, 111–119. https://doi.org/10.1071/wr99107 -Cao, Y., Li, B., 2019. Assessing models for estimation and methods for uncertainty quantification for spatial return levels. Environmetrics 30, e2508. -Cash, D.W., Clark, W.C., Alcock, F., Dickson, N.M., Eckley, N., Guston, D.H., Jäger, J., Mitchell, R.B., 2003. Knowledge systems for sustainable development. Proc. Natl. Acad. Sci. 100, 8086–8091. -Ciccazzo, A., Pillo, G.D., Latorre, V., 2014. Support vector machines for surrogate modeling of electronic circuits. Neural Comput. Appl. 24, 69–76. https://doi.org/10.1007/s00521-013-1509-5 -Coles, S., Bawa, J., Trenner, L., Dorazio, P., 2001. An introduction to statistical modeling of extreme values. Springer. -Cooke, R.M., 1991. Experts in uncertainty: Opinion and subjective probability in science, Experts in uncertainty: Opinion and subjective probability in science. Oxford University Press, New York, NY, US. -Coronese, M., Lamperti, F., Keller, K., Chiaromonte, F., Roventini, A., 2019. Evidence for sharp increase in the economic damages of extreme natural disasters. Proc Natl Acad Sci U A 201907826. https://doi.org/10.1073/pnas.1907826116 -Cukier, R.I., Fortuin, C.M., Shuler, K.E., Petschek, A.G., Schaibly, J.H., 1973. Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory. J. Chem. Phys. 59, 3873–3878. https://doi.org/10.1063/1.1680571 -Currin, C., Mitchell, T., Morris, M., Ylvisaker, D., 1991. Bayesian Prediction of Deterministic Functions, with Applications to the Design and Analysis of Computer Experiments. J Am Stat Assoc 86, 953–963. https://doi.org/10.1080/01621459.1991.10475138 -Dalal, I.L., Stefan, D., Harwayne-Gidansky, J., 2008. Low discrepancy sequences for Monte Carlo simulations on reconfigurable platforms, in: 2008 International Conference on Application-Specific Systems, Architectures and Processors. Presented at the 2008 International Conference on Application-Specific Systems, Architectures and Processors, pp. 108–113. https://doi.org/10.1109/ASAP.2008.4580163 -DeConto, R.M., Pollard, D., 2016. Contribution of Antarctica to past and future sea-level rise. Nature 531, 591–597. https://doi.org/10.1038/nature17145 -Del Moral, P., Doucet, A., Jasra, A., 2006. Sequential Monte Carlo samplers. J R Stat Soc Ser. B Stat Methodol 68, 411–436. https://doi.org/10.1111/j.1467-9868.2006.00553.x -Demarta, S., McNeil, A.J., 2005. The t copula and related copulas. Int. Stat. Rev. 73, 111–129. -Doucet, A., Freitas, N., Gordon, N., 2001. An Introduction to Sequential Monte Carlo Methods. Seq. Monte Carlo Methods Pract. 3–14. https://doi.org/10.1007/978-1-4757-3437-9_1 -Doucet, A., Godsill, S., Andrieu, C., 2000. On sequential Monte Carlo sampling methods for Bayesian filtering. Stat Comput 10, 197–208. https://doi.org/10.1023/A:1008935410038 -Eason, J., Cremaschi, S., 2014. Adaptive sequential sampling for surrogate model generation with artificial neural networks. Comput. Chem. Eng. 68, 220–232. https://doi.org/10.1016/j.compchemeng.2014.05.021 -Edwards, N.R., Cameron, D., Rougier, J., 2011. Precalibrating an intermediate complexity climate model. Clim Dyn 37, 1469–1482. https://doi.org/10.1007/s00382-010-0921-0 -Eker, S., Rovenskaya, E., Obersteiner, M., Langan, S., 2018. Practice and perspectives in the validation of resource management models. Nat. Commun. 9, 1–10. https://doi.org/10.1038/s41467-018-07811-9 -El Adlouni, S., Bobée, B., Ouarda, T.B.M.J., 2008. On the tails of extreme event distributions in hydrology. J Hydrol 355, 16–33. https://doi.org/10.1016/j.jhydrol.2008.02.011 -Ellsberg, D., 1961. Risk, Ambiguity, and the Savage Axioms. Q J Econ 75, 643–669. https://doi.org/10.2307/1884324 -Elsawah, S., Filatova, T., Jakeman, A.J., Kettner, A.J., Zellner, M.L., Athanasiadis, I.N., Hamilton, S.H., Axtell, R.L., Brown, D.G., Gilligan, J.M., Janssen, M.A., Robinson, D.T., Rozenberg, J., Ullah, I.I.T., Lade, S.J., 2020. Eight grand challenges in socio-environmental systems modeling. Socio-Environ. Syst. Model. 2, 16226–16226. https://doi.org/10.18174/sesmo.2020a16226 -Evensen, G., 1994. Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res. Oceans 99, 10143–10162. https://doi.org/10.1029/94JC00572 -Ferdous, M.R., Wesselink, A., Brandimarte, L., Di Baldassarre, G., Rahman, M.M., 2019. The levee effect along the Jamuna River in Bangladesh. Water Int 44, 496–519. https://doi.org/10.1080/02508060.2019.1619048 -Field, C.B., Barros, V.R., Dokken, D.J., Mach, K.J., Mastrandrea, M.D., Bilir, T.E., Chatterjee, M., Ebi, K.L., Estrada, Y.O., Genova, R.C., Girma, B., Kissel, E.S., Levy, A.N., MacCracken, S., Mastrandrea, P.R., White, L.L., 2014. Summary for Policymakers, in: Climate Change 2014: Impacts, Adaptation, and Vulnerability. Part A: Global and Sectoral Aspects. Contribution of Working Group II to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge, United Kingdom, and New York, NY, USA, pp. 1–32. -Fisher, R.A., 1960. The design of experiments. Oliver and Boyd, Edinburgh, Scotland, UK. -Flegal, J.M., Haran, M., Jones, G.L., 2008. Markov Chain Monte Carlo: Can We Trust the Third Significant Figure? Stat Sci 23, 250–260. https://doi.org/10.1214/08-STS257 -Frankignoul, C., Hasselmann, K., 1977. Stochastic climate models, Part II Application to sea-surface temperature anomalies and thermocline variability. Tell’Us 29, 289–305. https://doi.org/10.3402/tellusa.v29i4.11362 -Fuller, R.W., Wong, T.E., Keller, K., 2017. Probabilistic inversion of expert assessments to inform projections about Antarctic ice sheet responses. PLoS One 12, e0190115. https://doi.org/10.1371/journal.pone.0190115 -Gao, L., Bryan, B.A., Nolan, M., Connor, J.D., Song, X., Zhao, G., 2016. Robust global sensitivity analysis under deep uncertainty via scenario analysis. Environ. Model. Softw. 76, 154–166. https://doi.org/10.1016/j.envsoft.2015.11.001 -Gelfand, A.E., Hills, S.E., Racine-Poon, A., Adrian F. M. Smith, 1990. Illustration of Bayesian Inference in Normal Data Models Using Gibbs Sampling. J Am Stat Assoc 85, 972–985. https://doi.org/10.2307/2289594 -Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B., 2013. Bayesian Data Analysis, Third. ed. CRC Press. -Gelman, A., Hwang, J., Vehtari, A., 2014. Understanding predictive information criteria for Bayesian models. Stat Comput 24, 997–1016. https://doi.org/10.1007/s11222-013-9416-2 -Gelman, A., Meng, X.-L., Stern, H., 1996. Posterior predictive assessment of model fitness via realized discrepancies. Stat Sin 6, 733–760. -Gelman, A., Rubin, D.B., 1992. Inference from Iterative Simulation Using Multiple Simulations. Stat Sci 7, 457–511. https://doi.org/10.1214/ss/1177011136 -Gelman, A., Shalizi, C.R., 2013. Philosophy and the practice of Bayesian statistics. Br J Math Stat Psychol 66, 8–38. https://doi.org/10.1111/j.2044-8317.2011.02037.x -Gelman, A., Simpson, D., Betancourt, M., 2017. The Prior Can Often Only Be Understood in the Context of the Likelihood. Entropy 19, 555. https://doi.org/10.3390/e19100555 -Geman, S., Geman, D., 1984. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans Pattern Anal Mach Intell 6, 721–741. https://doi.org/10.1109/tpami.1984.4767596 -Geyer, C.J., 2011. Introduction to Markov Chain Monte Carlo, in: Brooks, S., Gelman, A., Jones, G.L., Meng, X.-L. (Eds.), Handbook of Markov Chain Monte Carlo. Chapman & Hall/CRC, Boca Raton, FL, pp. 3–48. -Geyer, C.J., Johnson, L.T., 2013. mcmc: Markov Chain Monte Carlo. R Package Version 0 9-2 URL HttpCRAN R-Proj. Orgpackage Mcmc. -Ghanem, R.G., Spanos, P.D., 1991. Spectral stochastic finite-element formulation for reliability analysis. J Eng Mech 117, 2351–2372. -Goodman, J., Weare, J., 2010. Ensemble samplers with affine invariance. Commun. Appl. Math. Comput. Sci. 5, 65–80. https://doi.org/10.2140/camcos.2010.5.65 -Gordon, N.J., Salmond, D.J., Smith, A.F.M., 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEE Proc. F Radar Signal Process. 140, 107–113. https://doi.org/10.1049/ip-f-2.1993.0015 -Gorissen, D., De Tommasi, L., Crombecq, K., Dhaene, T., 2009. Sequential modeling of a low noise amplifier with neural networks and active learning. Neural Comput. Appl. 18, 485–494. https://doi.org/10.1007/s00521-008-0223-1 -Gupta, H.V., Clark, M.P., Vrugt, J.A., Abramowitz, G., Ye, M., 2012. Towards a comprehensive assessment of model structural adequacy. Water Resour. Res. 48. https://doi.org/10.1029/2011WR011044 -Gupta, H.V., Sorooshian, S., Yapo, P.O., 1998. Toward improved calibration of hydrologic models: Multiple and noncommensurable measures of information. Water Resour. Res. 34, 751–763. https://doi.org/10.1029/97WR03495 -Gupta, H.V., Wagener, T., Liu, Y., 2008. Reconciling theory with observations: elements of a diagnostic approach to model evaluation. Hydrol. Process. Int. J. 22, 3802–3813. -Hadjimichael, A., Quinn, J., Reed, P., 2020. Advancing Diagnostic Model Evaluation to Better Understand Water Shortage Mechanisms in Institutionally Complex River Basins. Water Resour. Res. 56, e2020WR028079. https://doi.org/10.1029/2020WR028079 -Haer, T., Husby, T.G., Botzen, W.J.W., Aerts, J.C.J.H., 2020. The safe development paradox: An agent-based model for flood risk under climate change in the European Union. Glob Env. Change 60, 102009. https://doi.org/10.1016/j.gloenvcha.2019.102009 -Haimes, Y.Y., 2018. Risk Modeling of Interdependent Complex Systems of Systems: Theory and Practice. Risk Anal. 38, 84–98. https://doi.org/10.1111/risa.12804 -Hall, J.W., Lempert, R.J., Keller, K., Hackbarth, A., Mijere, C., McInerney, D.J., 2012. Robust climate policies under uncertainty: A comparison of robust decision making and info-gap methods. Risk Anal. 32, 1657–1672. https://doi.org/10.1111/j.1539-6924.2012.01802.x -Hamm, N.A.S., Hall, J.W., Anderson, M.G., 2006. Variance-based sensitivity analysis of the probability of hydrologically induced slope instability. Comput. Geosci. 32, 803–817. https://doi.org/10.1016/j.cageo.2005.10.007 -Han, Y., Ash, K., Mao, L., Peng, Z.-R., 2020. An agent-based model for community flood adaptation under uncertain sea-level rise. Clim Change. https://doi.org/10.1007/s10584-020-02802-6 -Hastings, W.K., 1970. Monte Carlo Sampling Methods Using Markov Chains and Their Applications. Biometrika 57, 97–109. https://doi.org/10.2307/2334940 -Helbing, D., 2013. Globally networked risks and how to respond. Nature 497, 51–59. https://doi.org/10.1038/nature12047 -Helton, J.C., Johnson, J.D., Sallaberry, C.J., Storlie, C.B., 2006. Survey of sampling-based methods for uncertainty and sensitivity analysis. Reliab. Eng. Syst. Saf., The Fourth International Conference on Sensitivity Analysis of Model Output (SAMO 2004) 91, 1175–1209. https://doi.org/10.1016/j.ress.2005.11.017 -Herman, J.D., Reed, P.M., Wagener, T., 2013. Time-varying sensitivity analysis clarifies the effects of watershed model formulation on model behavior. Water Resour. Res. 49, 1400–1414. https://doi.org/10.1002/wrcr.20124 -Herman, J.D., Zeff, H.B., Lamontagne, J.R., Reed, P.M., Characklis, G.W., 2016. Synthetic Drought Scenario Generation to Support Bottom-Up Water Supply Vulnerability Assessments. J. Water Resour. Plan. Manag. 142, 04016050. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000701 -Herrera, M., Natarajan, S., Coley, D.A., Kershaw, T., Ramallo-González, A.P., Eames, M., Fosas, D., Wood, M., 2017. A review of current and future weather data for building simulation. Build. Serv. Eng. Res. Technol. 38, 602–627. https://doi.org/10.1177/0143624417705937 -Hoffman, M.D., Gelman, A., 2014. The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo. J Mach Learn Res 15, 1593–1623. -Houtekamer, P.L., Mitchell, H.L., 1998. Data Assimilation Using an Ensemble Kalman Filter Technique. Mon. Weather Rev. 126, 796–811. https://doi.org/10.1175/1520-0493(1998)126<0796:DAUAEK>2.0.CO;2 -Hu, L., Keller, C.A., Long, M.S., Sherwen, T., Auer, B., Da Silva, A., Nielsen, J.E., Pawson, S., Thompson, M.A., Trayanov, A.L., Travis, K.R., Grange, S.K., Evans, M.J., Jacob, D.J., 2018. Global simulation of tropospheric chemistry at 12.5 km resolution: performance and evaluation of the GEOS-Chem chemical module (v10-1) within the NASA GEOS Earth system model (GEOS-5 ESM). Geosci Model Dev 11, 4603–4620. https://doi.org/10.5194/gmd-11-4603-2018 -Hu, Y.-M., Hendry, M., Heng, I.S., 2014. Efficient Exploration of Multi-Modal Posterior Distributions. ArXiv Astro-PhIM. -Hurrell, J.W., Holland, M.M., Gent, P.R., Ghan, S., Kay, J.E., Kushner, P.J., Lamarque, J.-F., Large, W.G., Lawrence, D., Lindsay, K., Lipscomb, W.H., Long, M.C., Mahowald, N., Marsh, D.R., Neale, R.B., Rasch, P., Vavrus, S., Vertenstein, M., Bader, D., Collins, W.D., Hack, J.J., Kiehl, J., Marshall, S., 2013. The Community Earth System Model: A Framework for Collaborative Research. Bull Am Meteorol Soc 94, 1339–1360. https://doi.org/10.1175/BAMS-D-12-00121.1 -Hurvich, C.M., Tsai, C.-L., 1989. Regression and time series model selection in small samples. Biometrika 76, 297–307. -Iooss, B., Boussouf, L., Feuillard, V., Marrel, A., 2010. Numerical studies of the metamodel fitting and validation processes. Int. J. Adv. Syst. Meas. 3, 11–21. -Jaynes, E.T., 2003. Probability Theory: The Logic of Science. Cambridge University Press. -Jin, R., Chen, W., Sudjianto, A., 2008. An Efficient Algorithm for Constructing Optimal Design of Computer Experiments. Presented at the ASME 2003 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers Digital Collection, pp. 545–554. https://doi.org/10.1115/DETC2003/DAC-48760 -Johnson, D.R., Fischbach, J.R., Ortiz, D.S., 2013. Estimating Surge-Based Flood Risk with the Coastal Louisiana Risk Assessment Model. J Coast Res 109–126. https://doi.org/10.2112/si_67_8 -Kandlikar, M., Risbey, J., Dessai, S., 2005. Representing and communicating deep uncertainty in climate-change assessments. C R Geosci 337, 443–455. -Kasmalkar, I.G., Serafin, K.A., Miao, Y., Bick, I.A., Ortolano, L., Ouyang, D., Suckale, J., 2020. When floods hit the road: Resilience to flood-related traffic disruption in the San Francisco Bay Area and beyond. Sci Adv 6, eaba2423. https://doi.org/10.1126/sciadv.aba2423 -Kass, R.E., Raftery, A.E., 1995. Bayes Factors. J Am Stat Assoc 90, 773–795. https://doi.org/10.1080/01621459.1995.10476572 -Keller, K., McInerney, D., 2008. The dynamics of learning about a climate threshold. Clim Dyn 30, 321–332. https://doi.org/10.1007/s00382-007-0290-5 -Kennedy, M.C., O’Hagan, A., 2001. Bayesian calibration of computer models. J R Stat Soc Ser. B Stat Methodol 63, 425–464. https://doi.org/10.1111/1467-9868.00294 -Kirsch, B.R., Characklis, G.W., Zeff, H.B., 2013. Evaluating the Impact of Alternative Hydro-Climate Scenarios on Transfer Agreements: Practical Improvement for Generating Synthetic Streamflows. J. Water Resour. Plan. Manag. 139, 396–406. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000287 -Kleindorfer, G.B., O’Neill, L., Ganeshan, R., 1998. Validation in simulation: Various positions in the philosophy of science. Manag. Sci. 44, 1087–1099. -Koutsoyiannis, D., 2004. Statistics of extremes and estimation of extreme rainfall: I. Theoretical investigation/Statistiques de valeurs extrêmes et estimation de précipitations extrêmes: I. Recherche théorique. Hydrol. Sci. J. 49. -Kraan, B.C., Cooke, R.M., 2000. Uncertainty in compartmental models for hazardous materials - a case study. J Hazard Mater 71, 253–268. https://doi.org/10.1016/S0304-3894(99)00082-5 -Kucherenko, S., Albrecht, D., Saltelli, A., 2015. Exploring multi-dimensional spaces: a Comparison of Latin Hypercube and Quasi Monte Carlo Sampling Techniques. ArXiv150502350 Stat. -Kullback, S., Leibler, R.A., 1951. On Information and Sufficiency. Ann Math Stat 22, 79–86. https://doi.org/10.1214/aoms/1177729694 -Kumar, P., 2011. Typology of hydrologic predictability. Water Resour. Res. 47. https://doi.org/10.1029/2010WR009769 -Kwakkel, J.H., Walker, W.E., Haasnoot, M., 2016. Coping with the Wickedness of Public Policy Problems: Approaches for Decision Making under Deep Uncertainty. J. Water Resour. Plan. Manag. 142, 01816001. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000626 -Lamontagne, J.R., Reed, P.M., Link, R., Calvin, K.V., Clarke, L.E., Edmonds, J.A., 2018. Large Ensemble Analytic Framework for Consequence-Driven Discovery of Climate Change Scenarios. Earths Future 6, 488–504. https://doi.org/10.1002/2017EF000701 -Lamontagne, J.R., Stedinger, J.R., 2018. Generating Synthetic Streamflow Forecasts with Specified Precision. J. Water Resour. Plan. Manag. 144, 04018007. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000915 -Landeg, O., Whitman, G., Walker-Springett, K., Butler, C., Bone, A., Kovats, S., 2019. Coastal flooding and frontline health care services: challenges for flood risk resilience in the English health care system. J Health Serv Res Policy 24, 219–228. https://doi.org/10.1177/1355819619840672 -Lee, B.S., Haran, M., Fuller, R., Pollard, D., Keller, K., 2020. A Fast Particle-Based Approach for Calibrating a 3-D Model of the Antarctic Ice Sheet. Ann. Appl. Stat. in the press. -Lee, B.S., Haran, M., Keller, K., 2017. Multidecadal Scale Detection Time for Potentially Increasing Atlantic Storm Surges in a Warming Climate. Geophys Res Lett 44, 10,617-10,623. https://doi.org/10.1002/2017GL074606 -Lempert, R.J., 2002. A new decision sciences for complex systems. Proc. Natl. Acad. Sci. 99, 7309–7313. https://doi.org/10.1073/pnas.082081699 -Liu, J., West, M., 2001. Combined Parameter and State Estimation in Simulation-Based Filtering, in: Doucet, A., de Freitas, N., Gordon, N. (Eds.), Sequential Monte Carlo Methods in Practice. Springer New York, New York, NY, pp. 197–223. https://doi.org/10.1007/978-1-4757-3437-9_10 -Liu, X., Guillas, S., 2017. Dimension Reduction for Gaussian Process Emulation: An Application to the Influence of Bathymetry on Tsunami Heights. SIAMASA J Uncertain. Quantif. 5, 787–812. https://doi.org/10.1137/16M1090648 -Lorenz, E.N., 1963. Deterministic Nonperiodic Flow. J Atmos Sci 20, 130–141. https://doi.org/10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2 -Loucks, D.P., Beek, E. van, 2017. Water Resource Systems Planning and Management: An Introduction to Methods, Models, and Applications. Springer International Publishing. -Lukacs, P.M., Burnham, K.P., Anderson, D.R., 2010. Model selection bias and Freedman’s paradox. Ann Inst Stat Math 62, 117–125. https://doi.org/10.1007/s10463-009-0234-4 -Makowski, D., Wallach, D., Tremblay, M., 2002. Using a Bayesian approach to parameter estimation; comparison of the GLUE and MCMC methods. Agronomie 22, 191–203. -Maniyar, D.M., Cornford, D., Boukouvalas, A., 2007. Dimensionality Reduction in the Emulator Setting (No. NCRG/2007/005). Neural Computing Research Group. -Marchau, V.A.W.J., Walker, W.E., Bloemen, P.J.T.M., Popper, S.W. (Eds.), 2019. Decision Making under Deep Uncertainty: From Theory to Practice. Springer International Publishing. https://doi.org/10.1007/978-3-030-05252-2 -Martin, A.D., 2010. MCMCpack: Markov chain Monte Carlo (MCMC) Package. R package version 1.0-7. HttpCRAN R-Proj. Orgpackage MCMCpack. -Massmann, C., Wagener, T., Holzmann, H., 2014. A new approach to visualizing time-varying sensitivity indices for environmental model diagnostics across evaluation time-scales. Environ. Model. Softw. 51, 190–194. https://doi.org/10.1016/j.envsoft.2013.09.033 -Matthews, P., 1993. A slowly mixing Markov chain with implications for Gibbs sampling. Stat Probab Lett 17, 231–236. https://doi.org/10.1016/0167-7152(93)90172-F -McInerney, D., Keller, K., 2008. Economically optimal risk reduction strategies in the face of uncertain climate thresholds. Clim Change 91, 29–41. https://doi.org/10.1007/s10584-006-9137-z -McKay, M.D., Beckman, R.J., Conover, W.J., 1979. A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code. Technometrics 21, 239–245. https://doi.org/10.2307/1268522 -Medda, S., Bhar, K.K., 2019. Comparison of single-site and multi-site stochastic models for streamflow generation. Appl. Water Sci. 9, 67. -Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E., 1953. Equation of State Calculations by Fast Computing Machines. J Chem Phys 21, 1087–1092. https://doi.org/10.1063/1.1699114 -Metropolis, N., Ulam, S., 1949. The Monte Carlo Method. J. Am. Stat. Assoc. 44, 335–341. https://doi.org/10.1080/01621459.1949.10483310 -Milly, P.C.D., Betancourt, J., Falkenmark, M., Hirsch, R.M., Kundzewicz, Z.W., Lettenmaier, D.P., Stouffer, R.J., 2008. Stationarity is dead: Whither water management? Science 319, 573–574. https://doi.org/10.1126/science.1151915 -Moallemi, E.A., Kwakkel, J., de Haan, F.J., Bryan, B.A., 2020a. Exploratory modeling for analyzing coupled human-natural systems under uncertainty. Glob. Environ. Change 65, 102186. https://doi.org/10.1016/j.gloenvcha.2020.102186 -Moallemi, E.A., Zare, F., Reed, P.M., Elsawah, S., Ryan, M.J., Bryan, B.A., 2020b. Structuring and evaluating decision support processes to enhance the robustness of complex human–natural systems. Environ. Model. Softw. 123, 104551. https://doi.org/10.1016/j.envsoft.2019.104551 -Montgomery, D.C., 2017. Design and analysis of experiments. John Wiley & Sons. -Morgan, M.G., Keith, D.W., 2008. Improving the way we think about projecting future energy use and emissions of carbon dioxide. Clim Change 90, 189–215. https://doi.org/10.1007/s10584-008-9458-1 -Morris, M.D., 1991. Factorial Sampling Plans for Preliminary Computational Experiments. Technometrics 33, 161–174. https://doi.org/10.1080/00401706.1991.10484804 -Morris, M.D., Mitchell, T.J., 1995. Exploratory designs for computational experiments. J. Stat. Plan. Inference 43, 381–402. https://doi.org/10.1016/0378-3758(94)00035-T -National Research Council, 2014. Convergence: facilitating transdisciplinary integration of life sciences, physical sciences, engineering, and beyond. National Academies Press, Washington, D.C., USA. -Naylor, T.H., Finger, J.M., 1967. Verification of computer simulation models. Manag. Sci. 14, B-92. -Neal, R.M., 2011. MCMC Using Hamiltonian Dynamics, in: Brooks, S., Gelman, A., Jones, G.L., Meng, X.-L. (Eds.), Handbook of Markov Chain Monte Carlo, Handbooks of Modern Statistical Methods. CRC Press, Boca Raton, FL, pp. 113–162. -Nearing, G.S., Ruddell, B.L., Bennett, A.R., Prieto, C., Gupta, H.V., 2020. Does Information Theory Provide a New Paradigm for Earth Science? Hypothesis Testing. Water Resour. Res. 56, e2019WR024918. https://doi.org/10.1029/2019WR024918 -Nelder, J.A., Wedderburn, R.W.M., 1972. Generalized Linear Models. J R Stat Soc Ser A 135, 370–384. https://doi.org/10.2307/2344614 -Norton, J., 2015. An introduction to sensitivity assessment of simulation models. Environ. Model. Softw. 69, 166–174. https://doi.org/10.1016/j.envsoft.2015.03.020 -Oddo, P.C., Lee, B.S., Garner, G.G., Srikrishnan, V., Reed, P.M., Forest, C.E., Keller, K., 2017. Deep Uncertainties in Sea-Level Rise and Storm Surge Projections: Implications for Coastal Flood Risk Management. Risk Anal. https://doi.org/10.1111/risa.12888 -O’Neill, B.C., Crutzen, P., Grübler, A., Duong, M.H., Keller, K., Kolstad, C., Koomey, J., Lange, A., Obersteiner, M., Oppenheimer, M., Pepper, W., Sanderson, W., Schlesinger, M., Treich, N., Ulph, A., Webster, M., Wilson, C., 2006. Learning and climate change. Clim Policy 6, 585–589. https://doi.org/10.1080/14693062.2006.9685623 -Oppenheimer, M., O’Neill, B.C., Webster, M., 2008. Negative learning. Clim Change 89, 155–172. https://doi.org/10.1007/s10584-008-9405-1 -Oreskes, N., Shrader-Frechette, K., Belitz, K., 1994. Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences. Science 263, 641–646. https://doi.org/10.1126/science.263.5147.641 -Park, J.-S., 1994. Optimal Latin-hypercube designs for computer experiments. J. Stat. Plan. Inference 39, 95–111. https://doi.org/10.1016/0378-3758(94)90115-5 -Pianosi, F., Beven, K., Freer, J., Hall, J.W., Rougier, J., Stephenson, D.B., Wagener, T., 2016. Sensitivity analysis of environmental models: A systematic review with practical workflow. Environ. Model. Softw. 79, 214–232. https://doi.org/10.1016/j.envsoft.2016.02.008 -Pollard, D., DeConto, R.M., 2012. Description of a hybrid ice sheet-shelf model, and application to Antarctica. Geosci. Model Dev. 5, 1273–1295. https://doi.org/10.5194/gmd-5-1273-2012 -Pruett, W.A., Hester, R.L., 2016. The Creation of Surrogate Models for Fast Estimation of Complex Model Outcomes. PLOS ONE 11, e0156574. https://doi.org/10.1371/journal.pone.0156574 -Ragulina, G., Reitan, T., 2017. Generalized extreme value shape parameter and its nature for extreme precipitation using long time series and the Bayesian approach. Hydrol. Sci. J. 62, 863–879. -Rakovec, O., Hill, M.C., Clark, M.P., Weerts, A.H., Teuling, A.J., Uijlenhoet, R., 2014. Distributed Evaluation of Local Sensitivity Analysis (DELSA), with application to hydrologic models. Water Resour. Res. 50, 409–426. https://doi.org/10.1002/2013WR014063 -Rasmussen, D.J., Buchanan, M.K., Kopp, R.E., Oppenheimer, M., 2020. A flood damage allowance framework for coastal protection with deep uncertainty in sea‐level rise. Earths Future. https://doi.org/10.1029/2019EF001340 -Robert, C.P., 2007. The Bayesian choice: from decision-theoretic foundations to computational implementation, 2nd ed. Springer, New York. -Rohmer, J., Le Cozannet, G., Manceau, J.-C., 2019. Addressing ambiguity in probabilistic assessments of future coastal flooding using possibility distributions. Clim Change. https://doi.org/10.1007/s10584-019-02443-4 -Ruckert, K.L., Shaffer, G., Pollard, D., Guan, Y., Wong, T.E., Forest, C.E., Keller, K., 2017. Assessing the Impact of Retreat Mechanisms in a Simple Antarctic Ice Sheet Model Using Bayesian Calibration. PLoS One 12, e0170052. https://doi.org/10.1371/journal.pone.0170052 -Ruckert, K.L., Srikrishnan, V., Keller, K., 2019. Characterizing the deep uncertainties surrounding coastal flood hazard projections: A case study for Norfolk, VA. Sci Rep 9, 11373. https://doi.org/10.1038/s41598-019-47587-6 -Sacks, J., Welch, W.J., Mitchell, T.J., Wynn, H.P., 1989. Design and Analysis of Computer Experiments. Stat Sci 4, 409–423. -Saltelli, A., 2002. Making best use of model evaluations to compute sensitivity indices. Comput. Phys. Commun. 145, 280–297. -Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., Wu, Q., 2019. Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices. Environ. Model. Softw. 114, 29–39. https://doi.org/10.1016/j.envsoft.2019.01.012 -Saltelli, A., Annoni, P., 2010. How to avoid a perfunctory sensitivity analysis. Environ. Model. Softw. 25, 1508–1517. https://doi.org/10.1016/j.envsoft.2010.04.012 -Saltelli, A., Funtowicz, S., 2014. When all models are wrong. Issues Sci. Technol. 30, 79–85. -Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S., 2008. Global Sensitivity Analysis: The Primer, 1 edition. ed. Wiley-Interscience, Chichester, England; Hoboken, NJ. -Saltelli, A., Stark, P.B., Becker, W., Stano, P., 2015. Climate models as economic guides scientific challenge or quixotic quest? Issues Sci. Technol. 31, 79–84. -Saltelli, A., Tarantola, S., 2002. On the relative importance of input factors in mathematical models: safety assessment for nuclear waste disposal. J. Am. Stat. Assoc. 97, 702–709. -Saltelli, A., Tarantola, S., Campolongo, F., Ratto, M., 2004. Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. John Wiley & Sons. -Saltelli, A., Tarantola, S., Chan, K.P.-S., 1999. A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output. Technometrics 41, 39–56. https://doi.org/10.1080/00401706.1999.10485594 -Schlather, M., 2002. Models for stationary max-stable random fields. Extremes 5, 33–44. -Schwarz, G., 1978. Estimating the Dimension of a Model. Ann Stat 6, 461–464. https://doi.org/10.1214/aos/1176344136 -Shafii, M., Tolson, B., Matott, L.S., 2014. Uncertainty-based multi-criteria calibration of rainfall-runoff models: a comparative study. Stoch Env. Res Risk Assess 28, 1493–1510. https://doi.org/10.1007/s00477-014-0855-x -Shibata, R., 1989. Statistical Aspects of Model Selection, in: Willems, J.C. (Ed.), From Data to Model. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 215–240. https://doi.org/10.1007/978-3-642-75007-6_5 -Sibley Heather M., Vroman Noah D., Shewbridge Scott E., 2017. Quantitative Risk-Informed Design of Levees. Geo-Risk 2017 76–90. https://doi.org/10.1061/9780784480717.008 -Sminchisescu, C., Welling, M., Hinton, G., 2003. A mode-hopping MCMC sampler. Technical Report CSRG-478, University of Toronto, submitted to Machine …. -Sobol, I.M., 2001. Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math. Comput. Simul. 55, 271–280. https://doi.org/10.1016/S0378-4754(00)00270-6 -Sobol, I.M., 1976. Uniformly distributed sequences with an additional uniform property. USSR Comput. Math. Math. Phys. 16, 236–242. -Sobol’, I.M., 1967. On the distribution of points in a cube and the approximate evaluation of integrals. USSR Comput. Math. Math. Phys. 7, 86–112. https://doi.org/10.1016/0041-5553(67)90144-9 -Spear, R.C., Hornberger, G.M., 1980. Eutrophication in peel inlet—II. Identification of critical uncertainties via generalized sensitivity analysis. Water Res. 14, 43–49. https://doi.org/10.1016/0043-1354(80)90040-8 -Spiegelhalter, D.J., Best, N.G., Carlin, B.P., Van Der Linde, A., 2002. Bayesian measures of model complexity and fit. J R Stat Soc Ser. B Stat Methodol 64, 583–639. https://doi.org/10.1111/1467-9868.00353 -Srikrishnan, V, Alley, R.B., Keller, K., 2019. Investing in Science and Using the Results to Improve Climate Risk Management. EOS. -Srikrishnan, Vivek, Guan, Y., Tol, R.S.J., Keller, K., 2019. Fossil fuel resources, decarbonization, and economic growth drive the feasibility of Paris climate targets. ArXiv StatAP. -Sriver, R.L., Lempert, R.J., Wikman-Svahn, P., Keller, K., 2018. Characterizing uncertain sea-level rise projections to support investment decisions. PLoS One 13, e0190641. https://doi.org/10.1371/journal.pone.0190641 -Stedinger, J.R., Vogel, R.M., Lee, S.U., Batchelder, R., 2008. Appraisal of the generalized likelihood uncertainty estimation (GLUE) method: APPRAISAL OF THE GLUE METHOD. Water Resour Res 44, 191. https://doi.org/10.1029/2008WR006822 -Steinschneider, S., Wi, S., Brown, C., 2015. The integrated effects of climate and hydrologic uncertainty on future flood risk assessments. Hydrol. Process. 29, 2823–2839. https://doi.org/10.1002/hyp.10409 -Stone, M., 1977. An Asymptotic Equivalence of Choice of Model by Cross-Validation and Akaike’s Criterion. J R Stat Soc Ser. B Stat Methodol 39, 44–47. -Surowiec, I., Vikström, L., Hector, G., Johansson, E., Vikström, C., Trygg, J., 2017. Generalized Subset Designs in Analytical Chemistry. Anal. Chem. 89, 6491–6497. https://doi.org/10.1021/acs.analchem.7b00506 -Tang, B., 1993. Orthogonal Array-Based Latin Hypercubes. J. Am. Stat. Assoc. 88, 1392–1397. https://doi.org/10.1080/01621459.1993.10476423 -Tang, Y., Reed, P., Wagener, T., Werkhoven, K. van, 2007. Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation. Hydrol. Earth Syst. Sci. 11, 793–817. https://doi.org/10.5194/hess-11-793-2007 -Toulmin, S., 1977. From form to function: philosophy and history of science in the 1950s and now. Daedalus 143–162. -Urban, N.M., Keller, K., 2010. Probabilistic hindcasts and projections of the coupled climate, carbon cycle and Atlantic meridional overturning circulation system: A Bayesian fusion of century-scale observations with a simple model. Tellus A 62, 737–750. -Van Oort, N., 2011. Service reliability and urban public transport design. -Van Schepdael, A., Carlier, A., Geris, L., 2016. Sensitivity Analysis by Design of Experiments, in: Geris, L., Gomez-Cabrero, D. (Eds.), Uncertainty in Biology: A Computational Modeling Approach, Studies in Mechanobiology, Tissue Engineering and Biomaterials. Springer International Publishing, Cham, pp. 327–366. https://doi.org/10.1007/978-3-319-21296-8_13 -van Vuuren, D.P., Edmonds, J., Kainuma, M., Riahi, K., Thomson, A., Hibbard, K., Hurtt, G.C., Kram, T., Krey, V., Lamarque, J.-F., Masui, T., Meinshausen, M., Nakicenovic, N., Smith, S.J., Rose, S.K., 2011. The representative concentration pathways: an overview. Clim Change 109, 5. https://doi.org/10.1007/s10584-011-0148-z -Vega‐Westhoff, B., Sriver, R.L., Hartin, C.A., Wong, T.E., Keller, K., 2019. Impacts of Observational Constraints Related to Sea Level on Estimates of Climate Sensitivity. Earths Future. https://doi.org/10.1029/2018EF001082 -Vehtari, A., Gelman, A., Gabry, J., 2017. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Stat Comput 27, 1413–1432. -Vihola, M., 2012. Robust adaptive Metropolis algorithm with coerced acceptance rate. Stat Comput 22, 997–1008. https://doi.org/10.1007/s11222-011-9269-5 -Vogel, R.M., 2017. Stochastic watershed models for hydrologic risk management. Water Secur. 1, 28–35. https://doi.org/10.1016/j.wasec.2017.06.001 -Vogel, R.M., Stedinger, J.R., 1988. The value of stochastic streamflow models in overyear reservoir design applications. Water Resour. Res. 24, 1483–1490. https://doi.org/10.1029/WR024i009p01483 -Vousdoukas, M.I., Bouziotas, D., Giardino, A., Bouwer, L.M., Mentaschi, L., Voukouvalas, E., Feyen, L., 2018. Understanding epistemic uncertainty in large-scale coastal flood risk assessment for present and future climates. Nat Hazards Earth Syst Sci 18, 2127–2142. https://doi.org/10.5194/nhess-18-2127-2018 -Vrugt, J.A., Beven, K.J., 2018. Embracing equifinality with efficiency: Limits of Acceptability sampling using the DREAM(LOA) algorithm. J Hydrol 559, 954–971. https://doi.org/10.1016/j.jhydrol.2018.02.026 -Walker, W., Haasnoot, M., Kwakkel, J., Walker, W.E., Haasnoot, M., Kwakkel, J.H., 2013. Adapt or Perish: A Review of Planning Approaches for Adaptation under Deep Uncertainty. Sustainability 5, 955–979. https://doi.org/10.3390/su5030955 -Walker, W.E., Harremoës, P., Rotmans, J., Sluijs, J.P. van der, Asselt, M.B.A. van, Janssen, P., Krauss, M.P.K. von, 2003. Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support. Integr. Assess. 4, 5–17. https://doi.org/10.1076/iaij.4.1.5.16466 -Walker, W.E., Lempert, R.J., Kwakkel, J.H., 2013. Deep Uncertainty, in: Gass, S.I., Fu, M.C. (Eds.), Encyclopedia of Operations Research and Management Science. Springer US, pp. 395–402. https://doi.org/10.1007/978-1-4419-1153-7_1140 -Watanabe, S., 2010. Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory. J Mach Learn Res 11, 3571–3594. -Weaver, C.P., Lempert, R.J., Brown, C., Hall, J.A., Revell, D., Sarewitz, D., 2013. Improving the contribution of climate model information to decision making: the value and demands of robust decision frameworks. Wiley Interdiscip. Rev. Clim. Change 4, 39–60. https://doi.org/10.1002/wcc.202 -White, D.D., Wutich, A., Larson, K.L., Gober, P., Lant, T., Senneville, C., 2010. Credibility, salience, and legitimacy of boundary objects: water managers’ assessment of a simulation model in an immersive decision theater. Sci. Public Policy 37, 219–232. https://doi.org/10.3152/030234210X497726 -Wilks, D.S., Wilby, R.L., 1999. The weather generation game: a review of stochastic weather models. Prog. Phys. Geogr. 23, 329–357. -Wirtz, D., Nowak, W., 2017. The rocky road to extended simulation frameworks covering uncertainty, inversion, optimization and control. Environ. Model. Softw. 93, 180–192. https://doi.org/10.1016/j.envsoft.2016.10.003 -Wolff, C., Nikoletopoulos, T., Hinkel, J., Vafeidis, A.T., 2020. Future urban development exacerbates coastal exposure in the Mediterranean. Sci Rep 10, 14420. https://doi.org/10.1038/s41598-020-70928-9 -Wong, T.E., Keller, K., 2017. Deep Uncertainty Surrounding Coastal Flood Risk Projections: A Case Study for New Orleans. Earths Future 5, 1015–1026. https://doi.org/10.1002/2017EF000607 -Wong, T.E., Klufas, A., Srikrishnan, V., Keller, K., 2018. Neglecting model structural uncertainty underestimates upper tails of flood hazard. Env. Res Lett 13, 074019. https://doi.org/10.1088/1748-9326/aacb3d -Xian, S., Lin, N., Kunreuther, H., 2017. Optimal house elevation for reducing flood-related losses. J Hydrol 548, 63–74. https://doi.org/10.1016/j.jhydrol.2017.02.057 -Xiu, D., Karniadakis, G.E., 2002. The Wiener--Askey Polynomial Chaos for Stochastic Differential Equations. SIAM J. Sci. Comput. 24, 619–644. https://doi.org/10.1137/S1064827501387826 -Zarekarizi, M., Srikrishnan, V., Keller, K., 2020. Neglecting Uncertainties Biases House-Elevation Decisions to Manage Riverine Flood Risks. Nat Commun. https://doi.org/10.1038/s41467-020-19188-9 -Zaremba, S.K., 1968. The mathematical basis of Monte Carlo and quasi-Monte Carlo methods. SIAM Rev. 10, 303–314. -Zellner, A., Tian, G.C., 1964. Bayesian analysis of the regression model with autocorrelated errors. J Am Stat Assoc 763–778. -Zhang, X., Che, L., Shahidehpour, M., Alabdulwahab, A.S., Abusorrah, A., 2015. Reliability-based optimal planning of electricity and natural gas interconnections for multiple energy hubs. IEEE Trans Smart Grid 8, 1658–1667. -Zscheischler, J., Martius, O., Westra, S., Bevacqua, E., Raymond, C., Horton, R.M., van den Hurk, B., AghaKouchak, A., Jézéquel, A., Mahecha, M.D., Maraun, D., Ramos, A.M., Ridder, N.N., Thiery, W., Vignotto, E., 2020. A typology of compound weather and climate events. Nat. Rev. Earth Environ. 1–15. https://doi.org/10.1038/s43017-020-0060-z diff --git a/docs/source/conf.py b/docs/source/conf.py index e990a26..e88e752 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -36,11 +36,15 @@ 'sphinx.ext.coverage', 'sphinx.ext.mathjax', 'sphinx.ext.ifconfig', - 'sphinx.ext.githubpages'] + 'sphinx.ext.githubpages', + 'sphinxcontrib.bibtex'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] +# bibliography files +bibtex_bibfiles = ['refs.bib'] + # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. diff --git a/docs/source/index.rst b/docs/source/index.rst index 689e624..137c196 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -18,8 +18,10 @@ Addressing Uncertainty in MultiSector Dynamics Research 5_uncertainty_quantification_the_basics 6_uncertainty_quantification_a_tool_for_capturing_risks_and_extremes 7_conclusion - 8_references - 9_glossary + 8_glossary + +.. bibliography:: + :style: plain Indices and tables diff --git a/docs/source/refs.bib b/docs/source/refs.bib new file mode 100644 index 0000000..0f4f82e --- /dev/null +++ b/docs/source/refs.bib @@ -0,0 +1,114 @@ +@book{national2014convergence, + title={Convergence: Facilitating transdisciplinary integration of life sciences, physical sciences, engineering, and beyond}, + author={National Research Council and others}, + year={2014}, + publisher={National Academies Press} +} + +@article{elsawah2020eight, + title={Eight grand challenges in socio-environmental systems modeling}, + author={Elsawah, Sondoss and Filatova, Tatiana and Jakeman, Anthony J and Kettner, Albert J and Zellner, Moira L and Athanasiadis, Ioannis N and Hamilton, Serena H and Axtell, Robert L and Brown, Daniel G and Gilligan, Jonathan M and others}, + journal={Socio-Environmental Systems Modelling}, + volume={2}, + pages={16226--16226}, + year={2020} +} + +@article{haimes2018risk, + title={Risk modeling of interdependent complex systems of systems: Theory and practice}, + author={Haimes, Yacov Y}, + journal={Risk analysis}, + volume={38}, + number={1}, + pages={84--98}, + year={2018}, + publisher={Wiley Online Library} +} + +@article{helbing2013globally, + title={Globally networked risks and how to respond}, + author={Helbing, Dirk}, + journal={Nature}, + volume={497}, + number={7447}, + pages={51--59}, + year={2013}, + publisher={Nature Publishing Group} +} + +@article{saltelli2019so, + title={Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices}, + author={Saltelli, Andrea and Aleksankina, Ksenia and Becker, William and Fennell, Pamela and Ferretti, Federico and Holst, Niels and Li, Sushan and Wu, Qiongli}, + journal={Environmental modelling \& software}, + volume={114}, + pages={29--39}, + year={2019}, + publisher={Elsevier} +} + +@article{wirtz2017rocky, + title={The rocky road to extended simulation frameworks covering uncertainty, inversion, optimization and control}, + author={Wirtz, Daniel and Nowak, Wolfgang}, + journal={Environmental Modelling \& Software}, + volume={93}, + pages={180--192}, + year={2017}, + publisher={Elsevier} +} + +@misc{kwakkel2016coping, + title={Coping with the wickedness of public policy problems: approaches for decision making under deep uncertainty}, + author={Kwakkel, Jan H and Walker, Warren E and Haasnoot, Marjolijn}, + year={2016}, + publisher={American Society of Civil Engineers} +} + +@article{gass1997encyclopedia, + title={Encyclopedia of operations research and management science}, + author={Gass, Saul I and Harris, Carl M}, + journal={Journal of the Operational Research Society}, + volume={48}, + number={7}, + pages={759--760}, + year={1997}, + publisher={Taylor \& Francis} +} + +@article{saltelli2015climate, + title={Climate models as economic guides scientific challenge or quixotic quest?}, + author={Saltelli, Andrea and Stark, Philip B and Becker, William and Stano, Pawel}, + journal={Issues in Science and Technology}, + volume={31}, + number={3}, + pages={79--84}, + year={2015}, + publisher={JSTOR} +} + +@book{cooke1991experts, + title={Experts in uncertainty: opinion and subjective probability in science}, + author={Cooke, Roger and others}, + year={1991}, + publisher={Oxford University Press on Demand} +} + +@article{moallemi2020exploratory, + title={Exploratory modeling for analyzing coupled human-natural systems under uncertainty}, + author={Moallemi, Enayat A and Kwakkel, Jan and de Haan, Fjalar J and Bryan, Brett A}, + journal={Global Environmental Change}, + volume={65}, + pages={102186}, + year={2020}, + publisher={Elsevier} +} + +@article{walker2003defining, + title={Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support}, + author={Walker, Warren E and Harremo{\"e}s, Poul and Rotmans, Jan and Van Der Sluijs, Jeroen P and Van Asselt, Marjolein BA and Janssen, Peter and Krayer von Krauss, Martin P}, + journal={Integrated assessment}, + volume={4}, + number={1}, + pages={5--17}, + year={2003}, + publisher={Taylor \& Francis} +}