Toward self service data commons provisioning

[erikerlandson]

To maximize community scalability for the data commons, it is desirable to have automation around the following:

• provisioning a new db schema (a collection of databases)
• provisioning a new os-climate gh team (which in our plan is isomorphic to DB user groups)
• exposing individual tables to the public (or to some more restricted user group)

[erikerlandson]

Here is one concept:

A community member fills out a github issue to create a new db schema. This triggers (via a github action) the creation of a series of github teams. These teams are, in turn coupled to the creation of trino user-groups. These trino user group may well not need to be created explicitly, if we get the github-group-provider working:
https://github.com/os-climate/trino-github-group-provider

The particular teams (and therefore groups) for a new schema might be:

• schema (or schema-project, schema-workstream, etc)
• schema-pipeline-admin
• schema-pipeline-developers
• schema-data-clearance-low
• schema-data-clearance-medium
• schema-data-clearance-high

The schema-data-clearance-xxx levels would be to support concepts like the ITR (black/gray/crystal) visibilities.
A fourth implied level of visibility is "public" - any authenticated user can read such data. Ideally most data sets have public visibility.

By default, only people in schema-pipeline-admin and schema-pipeline-developers can see any new tables created in schema, and only these groups can create, write, update, etc such tables. If any member of schema-pipeline-admin wishes to expose a new table as readable to the public, they do this by filing another kind of issue, which triggers another github action. This action creates a PR against operate-first/apps which adds particular rules to rules.json that specifically exposes that table. Similar workflow for schema-data-clearance-xxx - in which case one must be a member of schema-data-clearance-xxx github team, etc.

There can be variations on the above to support exposing certain individual table columns to various visibilities.

[caldeirav]

This seems doable if we follow specific conventions / structures for the data schemas / ingestion pipelines and groups. Updates on rules.json should however go through a proper release process (DEV / TEST / PROD) so we also have to start thinking about managing multiple environments and the process to promote changes across.

Q: How would the Git-based model compare to using an actual workflow automation tool that could also help with scheduling pipelines, such as Airflow?

[erikerlandson]

For these particular tasks, github actions are a good fit (IMO) because it is easy for a community member to file an issue, where they fill out some values, and it is easy to set up github actions that trigger on that. Also, the "outputs" are all on git: new github teams, new pull-req on operate-first/apps, etc.

[MichaelTiemannOSC]

Very timely!