Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added the stub for the UI gateway server #1

Merged
merged 8 commits into from
Mar 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Check http://editorconfig.org for more information
# This is the main config file for this project:
root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
indent_style = space
indent_size = 2
trim_trailing_whitespace = true

[*.{py, pyi}]
indent_style = space
indent_size = 4

[*.md]
trim_trailing_whitespace = false

[*.{diff,patch}]
trim_trailing_whitespace = false
50 changes: 50 additions & 0 deletions .github/workflows/client.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Test and build client

on: [push, pull_request, workflow_dispatch]

jobs:
test_and_build_client:
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./client
strategy:
matrix:
python-version: ["3.12"]
steps:
- uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install poetry
run: pip install poetry

- name: Install dependencies
run: |
poetry config virtualenvs.in-project true
poetry install --no-root

- name: Check code
run: |
poetry run mypy .
poetry run flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics

- name: Run tests
run: poetry run pytest

# The version in PyPI must follow the PEP 440 standard.
# Semantic versioning is not supported by PyPI.
# Therefore, only the release version is published.
- name: Build and publish to pypi
run: |
if [[ ${{ github.ref }} =~ ^refs/tags/[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "This is a version tag. Build and publish a package."
poetry config pypi-token.pypi ${{ secrets.PYPI_TOKEN }}
poetry build
poetry publish
else
echo "This is not a version tag."
fi
138 changes: 138 additions & 0 deletions .github/workflows/server.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
name: Test and build server

on:
push:
branches:
- '**'
tags:
- '**'
paths-ignore:
- 'releases/**'
pull_request:
branches:
- '**'
workflow_dispatch:

env:
REGISTRY: ghcr.io
IMAGE_NAME: glaciation-heu/ui_gateway

jobs:
test_server:
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./server
strategy:
matrix:
python-version: ["3.12"]
steps:
- uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install poetry
run: pip install poetry

- name: Install dependencies
run: |
poetry config virtualenvs.in-project true
poetry install --no-root --with dev,test

- name: Run style checks
run: |
poetry run mypy .
poetry run isort . --check --diff
poetry run flake8 .
poetry run black . --check --diff

# Temporarily disabled, because there are no tests.
# - name: Run tests
# run: poetry run pytest

# Temporarily disabled, because for now it is a stub.
# - name: Verify OpenAPI spec has been updated
# run: |
# cd "${{ github.workspace }}"
# poetry run --directory server python ./tools/extract_openapi.py app.main:app --app-dir ./server --out ./api/openapi_generated.yaml --app_version_file ./VERSION
# git diff --exit-code ./api/openapi.yaml ./api/openapi_generated.yaml

# - name: Verify client has been updated
# run: |
# cd "${{ github.workspace }}"
# poetry --directory server run python ./tools/client_generator/generate.py --file ./api/openapi.yaml
# git diff --exit-code -- ./client

build_server:
needs: [test_server]
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./server
permissions:
contents: write
packages: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Install Helm
uses: azure/setup-helm@v3

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Make versions
run: |
cd "${{ github.workspace }}"
chmod +x ./tools/version.sh
./tools/version.sh "${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}"
echo "VERSION_APP=$(cat "./VERSION_APP")" >> $GITHUB_ENV
echo "DOCKER_IMAGES=$(cat "./DOCKER_IMAGES")" >> $GITHUB_ENV
echo "DOCKER_TAGS=$(cat "./VERSION_DOCKER")" >> $GITHUB_ENV

- name: Log into registry ${{ env.REGISTRY }}
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ github.token }}

- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: ${{ env.DOCKER_TAGS }}

- name: Build and push docker image
uses: docker/build-push-action@v5
with:
push: true
tags: ${{ env.DOCKER_IMAGES }}
labels: ${{ steps.meta.outputs.labels }}
context: "${{ github.workspace }}/server"

- name: Configure Git
run: |
git config user.name "$GITHUB_ACTOR"
git config user.email "$GITHUB_ACTOR@users.noreply.github.com"

- name: Create a github release
run: gh release create "${{ env.VERSION_APP }}"
env:
GH_TOKEN: ${{ github.token }}

- name: Publish Helm charts
uses: stefanprodan/helm-gh-pages@master
with:
token: ${{ github.token }}
charts_dir: "./server/charts/"
target_dir: "./helm-charts/"
78 changes: 78 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
default_language_version:
python: python3.12

default_stages: [commit, push]

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.4.0
hooks:
- id: trailing-whitespace
exclude: client/.*|.*\.md
- id: end-of-file-fixer
exclude: client/.*|.*\.md
- id: check-yaml
exclude: charts/
- id: check-added-large-files

- repo: https://github.com/koalaman/shellcheck-precommit
rev: v0.9.0
hooks:
- id: shellcheck

- repo: local
hooks:
- id: mypy
name: mypy
files: ^server/
entry: poetry --directory server run mypy
language: system
types: [python]

- repo: local
hooks:
- id: isort
name: isort
files: ^server/
entry: poetry --directory server run isort
language: system
types: [python]

- repo: local
hooks:
- id: flake8
name: flake8
files: ^server/
entry: poetry --directory server run flake8 --config ./server/.flake8
language: system
types: [python]

- repo: local
hooks:
- id: black
name: black
files: ^server/
entry: poetry --directory server run black
language: system
types: [python]

# Temporarily disabled, because for now it is a stub.
# - repo: local
# hooks:
# - id: openapi
# name: openapi
# entry: poetry --directory server run python ./tools/extract_openapi.py app.main:app --app-dir ./server --out ./api/openapi.yaml --app_version_file ./VERSION
# language: system
# types: [python]
# pass_filenames: false
# always_run: true

# - repo: local
# hooks:
# - id: generate_client
# name: generate client
# entry: poetry --directory server run python ./tools/client_generator/generate.py --file ./api/openapi.yaml
# language: system
# types: [python]
# pass_filenames: false
# always_run: true
48 changes: 42 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,45 @@
# ui_gateway
UI gateway service
# UI gateway service
An intermediary component between Glaciation Frontend and Metadata Service. The service adapts user interface queries into a series of SPARQL requests to MetadataService and telemetry requests from Storage Service. The service also checking user permissions via oauth server and filters the KGs that a user is not allowed to access.

## Development
Work on the server and client is conducted in their respective directories: server and client, as the server-side and client-side parts have different dependencies, configurations, etc.

## Build redoc
### Requirements
Python 3.10+

### Installation pre-commit hooks
```bash
npm i -g @redocly/cli@latest
redocly build-docs api/openapi.yaml --output=api/redoc.html
```
pip install pre-commit
pre-commit install
```

### Working on a server
Go to the `/server` folder to install dependencies and work on the server application.
Documentation on setting up the virtual environment, installing dependencies, and working with the server can be found [here](./server/README.md).

### Working on a client
Go to the `/client` folder to install dependencies and work on the client application.
Documentation on setting up the virtual environment, installing dependencies, and working with the client can be found [here](./client/README.md).

### Release
The application version is specified in the VERSION file. The version should follow the format a.a.a, where 'a' is a number.
To create a release, update the version in the VERSION file and add a tag in GIT.
The release version for branches, pull requests, and tags will be generated based on the base version in the VERSION file.

### GitHub Actions
GitHub Actions triggers testing, builds, and application publishing for each release.
https://docs.github.com/en/actions

During the build and publish process, a Docker image is built, a Helm chart is created, an openapi.yaml is generated.

**Initial setup**
1. Create the branch gh-pages and use it as a GitHub page https://pages.github.com/.
2. Set up secrets at `https://github.com/glaciation-heu/ui_gateway/settings/secrets/actions`:
- PYPI_TOKEN - The secret token for PyPI. https://pypi.org/help/#apitoken

**After execution**
1. The index.yaml file containing the list of Helm charts will be available at `https://glaciation-heu.github.io/ui_gateway/helm-charts/index.yaml`. You can this URL on https://artifacthub.io/.
2. A package of the client will be available at pypi.org.

## Collaboration guidelines
HIRO uses and requires from its partners [GitFlow with Forks](https://hirodevops.notion.site/GitFlow-with-Forks-3b737784e4fc40eaa007f04aed49bb2e?pvs=4)
1 change: 1 addition & 0 deletions VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.1.0
5 changes: 5 additions & 0 deletions api/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
## Build redoc file
```bash
npm i -g @redocly/cli@latest
redocly build-docs api/openapi.yaml --output=api/redoc.html
```
Loading
Loading