Scope, architecture, and dependency structure

[rafmudaf]

Acknowledging that I lack nearly all context for the background of windIO, I propose here an architecture and scope of use for this software repository along with a structure for managing dependencies.

Scope

As the definition of an ontology, a software repository is useful for various infrastructure and examples, but it is not required. However, the developers of windIO have created nice tools to make adoption of the ontology simpler. In that regard, I propose that windIO adopt the scope listed below. These are all already in place, but this intends to answer the question "What is windIO as a software repository?":

1. Definition of the ontology
2. Definition of the file format for providing the definition in 1
3. Software library for loading and validating the file defined in 2
4. Host for documentation (including examples) for using the windIO ontology and software library

Architecture

The proposed architecture below closely follows the scope stated above. Note that the top-level windIO object in the diagram is intended to be a namespace that provides access to the objects beneath it.

classDiagram
    windIO *-- loader
    windIO *-- Plant
    windIO *-- Turbine    
    class loader{
      +dict load()
      +validate()
    }
    class Plant{
      +common
      +energy_resource
      +site
      +turbine
      +wind_energy_system
      +wind_farm
    }
    class Turbine{
      +ontology
    }

Loading

The ontology definition and it's YAML representation are defined in the Plant and Turbine modules via YAML schemas (the windIO.plant and windIO.turbine modules in the software library). The module for loading and validating the input files is available through windIO.loader, and it contains functionality similar to the existing Loader, load_yaml and validate_yaml functions in yml_utils.

Dependencies

Currently, windIO has dependencies as depicted below. Using windIO requires the user to install both windIO and the simulation software they'll use. Also, there will be missing dependencies in windIO unless all software that implement this ontology are installed since data structures from the calling-codes are used in the implementations. Lastly, this structure requires any software implementing windIO to make a change to the windIO library itself.

flowchart TD
    BB[[PyWake]] --> B(pywake_utils)
    B(pywake_utils) --> A[windIO]
    CC[[TopFarm]] --> C(topfarm_utils)
    C(topfarm_utils) --> A[windIO]
    DD[[FOXES]] -.-> D(foxes_utils)
    D(foxes_utils) -.-> A[windIO]
    EE[[FLORIS]] -.-> E(floris_utils)
    E(floris_utils) -.-> A[windIO]
    A --> z{{"analysis script"}}
    EE -.-> z

Loading

I propose to require that any software implementing windIO include it as a dependency, and that they include their windIO interface in their software projects.

flowchart TD
    A[windIO] --> BB[[PyWake]] --> B(pywake_windIO)
    A[windIO] --> CC[[TopFarm]] --> C(topfarm_windIO)
    A[windIO] --> DD[[FOXES]] --> D(foxes_windIO)
    A[windIO] --> EE[[FLORIS]] --> E(floris_windIO)
    DD --> z{{"analysis script"}}
    D --> z{{"analysis script"}}

Loading

There are currently components included in windIO that fall outside of this scope; specifically, the software-specific implementations of the windIO ontology (pywake and topfarm). I propose that all implementations of windIO be included in their host software repositories.

[rafmudaf]

@kilojoules @ptrbortolotti @bayc @madsmpedersen @SchmJo
I'm happy to hear any feedback and suggested changes to the proposals here

[kenloen]

Sorry I did not see this discussion post before.

We have also discussed the structure of the main windIO at our Task 55 meetings. Currently, the main repo is a Python module that also includes schemas. That somewhat makes it unclear what windIO is - is it a Python module or the schema definition? I therefore think it would make sense to split the main repo into a main windIO repo that contains the schemas and their documentation and then another repo (maybe pywindio ?) that contains the current Python module. That would make it clearer for people to understand what windIO is as I see it.

Regardless of splitting the schema and Python module, I fully agree with the proposed change. windIO should be the dependency of the other modules and not the other way around.

[kenloen]

cc @kilojoules @ptrbortolotti @fzahle @bayc @gbarter

[rafmudaf]

This is a great point @kenloen. I think windIO can (and should) be both a schema definition and a Python module, but what's missing is a clear statement of this. This could be addressed by changing the first line in the README to something like:

windIO is a definition of input and output data for the computational modeling of wind energy systems.
It includes the following components:
- Schema defining windIO components
- Python library for interacting with files that conform to the schema
- windIO input files for reference wind turbine and wind farm models

A similar statement could be added at the top of the homepage of the docs. What do you think?

As for splitting the repositories, my opinion is that this would add more complexity and would nevertheless require the same clear statements as above. For example, if they were split, it should be clearly stated that "windIO" is a schema definition and there's a software library for interacting with it in code called pywindio. While I don't have a strong opinion, for simplicity, I lean toward keeping one repository but improving the documentation and also improving the architecture (i.e. #75).

[kenloen]

Maintaining two repos, which to many of us would have a significant overlap, could become tedious.

But we must differentiate the two. Notably, the version of the Python module and the schema could become an issue, as if we make a bugfix in the Python module, which will bump the version, but the schema did not change. It, therefore, could become unclear to the user what changed between versions - if anything. The opposite would also be an issue, with only the schema changing and nothing changed on the Python side.

From the Python side, we would also need to be able to fetch any of the versions of the windIO schema, and I don't know how we would do that if the Python module only contains a specific version.

[ptrbortolotti]

I am very much against switching to two repos. We are already struggling to maintain one repo. @rafmudaf please feel free to submit a PR with the proposed changes to the readmes. I will also setup a meeting with you to finally get things going

[kenloen]

I do understand that having two repos would mean more to maintain - but as mentioned in my post above, I do think we are running the risk of maintaining two different things (a schema definition and a Python module) in the same repo. I think that will be an issue, and it will be hard for users to understand the changes between versions. Splitting the two at a later stage, I think, will break many more projects that will depend on the Python module.

Another thing to this is the ability to fetch the different schema versions for the same Python instance, which becomes difficult if they are strongly linked with the Python version.