From 26d7a22e1284028683f6e38f1b15277fe96c412a Mon Sep 17 00:00:00 2001 From: wpbonelli Date: Fri, 6 Sep 2024 15:53:55 -0400 Subject: [PATCH 1/2] update sdd --- docs/dev/sdd.md | 205 ++++++++++++++---------------------------------- flopy4/attrs.py | 7 +- 2 files changed, 65 insertions(+), 147 deletions(-) diff --git a/docs/dev/sdd.md b/docs/dev/sdd.md index d11cbc4..289e454 100644 --- a/docs/dev/sdd.md +++ b/docs/dev/sdd.md @@ -5,8 +5,6 @@ - [Principles](#principles) - [Overview](#overview) - - [Runtime](#runtime) - - [Plugins](#plugins) - [Objects](#objects) - [Counting the ways](#counting-the-ways) - [Context resolution](#context-resolution) @@ -53,118 +51,50 @@ the object model in rare instances. ## Overview -FloPy can provide a generic framework for hydrologic -models. +FloPy can provide a basic set of building blocks with +which MODFLOW 6 input parameters can be defined and +configured: **parameters** and **contexts** (groups +of parameters and possibly nested contexts). -FloPy can consist of plugins, each defining a wrapper -for a given hydrologic program. Programs are expected -to provide an unambiguous input specification. +Ideally, the input framework might be agnostic to the +program it represents. This could make it possible to +support other programs with a consistent/unified API, +though that is not an immediate goal. -FloPy can provide a basic set of building blocks with -which a program's input parameters can be defined and -configured. This will consist of parameters in nested -contexts. (This is made more rigorous below.) - -Provided an input specification for a program, FloPy -can generate an object-oriented Python interface for -it. This will consist of an **object model** (input -data model) and **IO module** (data access layer). - -Once these exist, they *are* the specification — -specification documents should be derivable from them -in reverse. - -FloPy will provide a **plugin runtime** which accepts -a program selection and an input configuration. - -The runtime will validate the latter, run the program, -report its progress, and make its results available. - -### Runtime - -FloPy will provide a plugin runtime whose purpose is to -wrap/run arbitrary hydrologic programs. **Simulation** -is the fundamental abstraction: we could consider the -simulation a *plan for how to execute a program*. - -This at odds with the standard terminology in MODFLOW 6, -where a simulation means the runtime itself. FloPy, as -an interface to programs, could reasonably call the thing -that becomes the simulation the simulation; seems benign -(and maybe even appropriate) effacement of a meaningful -distinction for reasons of precedent and familiarity. - -A distinct abstraction could represent the "task" that -runs the program. A third could represent its output. -The latter should be derivable from the simulation, if -results are available in a given workspace, so results -can still be retrieved easily in a subsequent session, -or by someone else provided the workspace contents. - -Runs could have an autogenerated GUID and an optional -name. Anonymous runs' names could default to the GUID. - -Scheduling seems like it may benefit from asynchrony. -While programs should ideally make maximal use of the -resources provided to them, one might want to run more -than one single-threaded program at once, without the -need for a separate Python interpreter for each one. - -An awaitable (coroutine-based) API, returning futures -instead of blocking, could allow an arbitrary number -of concurrent runs. - -If this is pursued, a synchronous alternative should -be provided which runs programs directly as done now. - -### Plugins - -A developer will first implement a plugin supporting some -hydrologic modeling program. This *must* include an input -specification and *may* involve overriding hooks provided -by the FloPy framework to customize various behavior. If -the program requires input in a bespoke language, or its -specification is written in such a language, the plugin -*must* include an explicit language specification, which -is used to generate a parser for the language. By default -FloPy will support standard data interchange formats such -such as JSON, TOML, and YAML. - -Provided a valid input specification, FloPy will generate -a Python interface for the given program. At minimum this -consists of an object model and a data access layer. The -former is user-facing, and will invoke the latter behind -the scenes to read and write plugin input configurations. - -A model developer will interact with the plugin, using the -interface layer to construct a model, and with the runtime -framework, to run the model. The interface layer will also -provide access to model results. +Provided an **input specification** for MF6, FloPy +can generate an object-oriented Python interface, +consisting of an **object model** (components of +the simulation and their input/output contexts) +and an **IO module** (data access layer). -### Objects +Once these exist, an input specification should be +derivable from them in reverse. -We want: +A **runtime** can validate the configuration, run the +program, report its progress, and expose its results. -- an intuitive, consistent, & expressive interface -to a broad range of programs +FloPy will support the MODFLOW 6 input language as well as +standard data interchange formats such such as JSON, TOML, +and YAML. -- a small core codebase and a largely autogenerated -user-facing input data model +### Objects -- an unsurprising and uncomplicated core framework -accessible to new contributors +Ideally, an object model might have -- more consistent (and fewer) points of entry +- a small core codebase and a largely autogenerated +user-facing API -- easy access to a program's input specification +- consistent (and few) points of entry -- easy access to a simulation's input configuration +- easy access to components' input specification -- hierarchical namespacing and context resolution +- easy access to components' current input data -- automatic enforcement of program invariants +- hierarchical namespacing and context-awareness -...and more. +- validation capabilities, and a way to detect when a +simulation is "dirty": needing validation again after +parameters have changed. #### Counting the ways @@ -243,76 +173,59 @@ specification of arbitrary structure and content to a program-agnostic data mode A **parameter** is a program input variable. -A parameter is a leaf in the **context tree**. The +A parameter is a leaf in the **context**. A context +is a map of parameters and/or nested contexts. The simulation is the root. -A parameter is a primitive value or a **composite** -of such. +A parameter is a primitive value or a **composite**. Primitive parameters are **scalar** (int, float, bool, -string, path) or **array-like**. +string, path), **array-like**, or **tabular**. > [!NOTE] > Ideally a data model would be dependency-agnostic, -but we view NumPy as de facto standard library and -accept its array primitives — especially as -they have made recent advancements in type hinting. +but NumPy's array primitives are nearly universal. If there is ever need to define array abstractions of our own, we could take inspiration from [astropy](https://github.com/astropy/astropy). Composite parameters are **record** and **union** (product and sum, respectively) types, as well as -**lists** of primitives or records. A record is a -named and ordered tuple of primitives. +**lists** of records. -> [!NOTE] -> Records are shown as `Dict` for demonstration, -but need implementing as an `attrs`-based class -so the parameter specification is discoverable -upon import. +A record is a tuple of scalar parameters. -A list may constrain its elements to parameters of -a single scalar or record type, or may hold unions -of such. - -> [!NOTE] -> On this view an MF6 keystring is a `typing.Union` -of records and a period block is a list of `Union`s -of records. Most packages' `packagedata` block, on -the other hand, have a regular shape, and can thus -be considered tabular. - -A context is a map of parameters. So is a record; -the operative difference is that composites cannot -contain nested parameters. A context is a non-leaf -node in the tree which can contain both parameters -and other contexts. - -We envision a nested hierarchy of `attrs`-based -classes, all acting like dictionaries, making up -the context tree. These will include composites: -strongly typed records and unions will be more -convenient to work with. - -So, FloPy can define a parameter as: +The data model can be specified roughly as: ```python from typing import Dict, List from numpy.typing import ArrayLike from pandas import DataFrame -Scalar = bool | int | float | str | Path -Record = Dict[str, Scalar | List[Scalar]] -List = List[Scalar | Record] + +Scalar = Union[bool, int, float, str, Path] +Record = Tuple[Scalar, ...] Array = ArrayLike Table = DataFrame -Param = Scalar | Record | List | Array | Table +Param = Union[Scalar, Array, Table, Record] ``` -This is proposed as a general foundation onto which -it should be possible to map input specifications -for a wide range of programs, not only MODFLOW 6. +An MF6 keystring can be represented as a union of +records. Period blocks are lists of record unions. +Most packages' `packagedata` block, on the other +hand, are tabular (regularly shaped), and can be +represented with a `DataFrame`. + +TODO: how to specify the table schema/dtypes? + +A nested hierarchy of `attrs`-based classes can +form the context tree and composite parameters, +though records are simply tuples (and contexts +dictionaries) from a serializer's perspective. + +It should be possible to map input specifications +for a wide range of programs onto this foundation, +not only MODFLOW 6. #### Arrays diff --git a/flopy4/attrs.py b/flopy4/attrs.py index 1dffb1b..87f89ac 100644 --- a/flopy4/attrs.py +++ b/flopy4/attrs.py @@ -2,6 +2,7 @@ from typing import ( Any, Optional, + Tuple, TypeVar, Union, ) @@ -28,7 +29,11 @@ """A table input parameter.""" -Param = Union[Scalar, Array, Table] +Record = Tuple[Scalar, ...] +"""A record input parameter.""" + + +Param = Union[Scalar, Array, Table, Record] """An input parameter.""" From 907e851cd4a813e8ccfd59f6d129b937055bb252 Mon Sep 17 00:00:00 2001 From: wpbonelli Date: Fri, 6 Sep 2024 15:54:51 -0400 Subject: [PATCH 2/2] doctoc srs --- docs/dev/srs.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/dev/srs.md b/docs/dev/srs.md index d25fc2a..cede183 100644 --- a/docs/dev/srs.md +++ b/docs/dev/srs.md @@ -4,9 +4,14 @@ - [Introduction](#introduction) - - [Intended audience](#intended-audience) - - [Value proposition](#value-proposition) - - [Project scope](#project-scope) + - [Product scope](#product-scope) + - [Product value](#product-value) + - [Intended audience:](#intended-audience) + - [Intended use:](#intended-use) + - [Use cases](#use-cases) +- [System requirements and functional requirements](#system-requirements-and-functional-requirements) +- [External interface requirements](#external-interface-requirements) +- [Non-functional requirements (NRFs)](#non-functional-requirements-nrfs) - [Motivation](#motivation) - [Consistency](#consistency) - [Maintenance](#maintenance)