README.md

Architecture

Summary

• Metadata API: stac-fastapi implementation of STAC API
  • stac-browser lightweight web-frontend
• Data API: pygeoapi implementation of OGCAPIFeatures and OGCAPICoverages
• Data storage: zarr and (geo)parquet
  • These analysis-ready cloud-optimised (ARCO) formats provide a method of direct data access
• Metadata storage: Frictionless
• Data pipeline: Argo workflows
• Deployment: AWS usingTerraform, Kubernetes and Flux

Components

• Infrastructure as Code (IaC)
  • External services + Cloud infrastructure + Kubernetes bootstrap: rimrep-terraform
  • GitOps: rimrep-flux
• Observability
  • New Relic
  • Weave GitOps (front end for Flux)
• Metadata system
  • Metadata entry tool: rimrep-metcalf
  • Metadata API backend: rimrep-stac-fastapi
    • Storage: AWS RDS PostgreSQL
  • Metadata API frontend: rimrep-stac-browser
  • Metadata catalog point-of-truth: rimrep-catalog
• Data system
  • Data storage: AWS S3
  • Data API backend (and simple frontend): rimrep-pygeoapi
  • Data API auth: KrakenD API Gateway
  • Direct access: AWS S3 or KrakenD API Gateway
• Data pipeline
  • Argo Workflows (rimrep-flux)
  • Python modules: rimrep-data-pipeline
• Auth
  • Okta for internal services
  • Keycloak for external services
  • Authentication in front of services: oauth2proxy
  • Authorization in front of data: KrakenD API Gateway
• Admin dashboard
  • Simple front-end: rimrep-dashboard
  • Allows users copy their JWT identity token to clipboard.

High level architecture

%%{
  init: {
    'theme': 'forest',
    'themeVariables': {
      'edgeLabelBackground': '#ffffff',
      'tertiaryTextColor': '#0f00aa',
      'clusterBkg': '#fafaff',
      'clusterBorder': '#0f00aa', 
      'fontSize': 24
    }
  }
}%%

flowchart TB
  classDef green fill:#97d8ad,stroke:#3b3b3b;
  classDef blue fill:#5fa5a3, stroke:#315554 ;

  subgraph "DMS"
    data_pipeline(<strong>Data Pipeline</strong>\n Argo Workflows):::green
    data_api(<strong>Data API</strong>\n pyGeoAPI):::green
    metadata_api(<strong>Metadata</strong> \n Browser/API/RDS):::green
    data_storage[(<strong>Data Storage AWS S3</strong>\n Zarr/parquet \n datapackage.json \nPublic/Private DS)]:::green
    catalog[("<strong>Catalog</strong> \n datapackage.json \n (table/grid)schema.json\ncollection.jsonnet")]:::green
    OAuth("<strong>OAuth</strong> \n Keycloak with AAF \n krakenD API Gateway"):::blue
    end

  subgraph "Data Providers" 
    data_providers(("Data"))
    external_metadata(("Metadata"))
  end


  github([DS issue ticket \n GitHub])
  external_users((External Users))

  data_providers --> |Ingested by| data_pipeline
  external_metadata --> |Automatically harvested \n Manually created| github
  github --> catalog
  data_pipeline -->|Configures| data_api
  data_pipeline -->|Configures| metadata_api
  catalog -->|metadata to Data Workflow| data_pipeline
  data_pipeline -->|Publish data to| data_storage
  data_storage -->|Access by| data_api

  metadata_api--> |Points to| data_api
  metadata_api-->|Points to| data_storage

  external_users ==> OAuth
  external_users .->|only public DS| data_storage
  OAuth ==> data_api
  OAuth ==> metadata_api

Loading