🧪 Run CI against actual server (#10)

.firecrest-demo-config.json

@@ -0,0 +1,8 @@
+{
+    "url": "http://localhost:8000/",
+    "token_uri": "http://localhost:8080/auth/realms/kcrealm/protocol/openid-connect/token",
+    "client_id": "firecrest-sample",
+    "client_secret": "b391e177-fa50-4987-beaf-e6d33ca93571",
+    "machine": "cluster",
+    "scratch_path": "/home/service-account-firecrest-sample"
+}

.github/workflows/server-tests.yml

@@ -0,0 +1,92 @@
+# Run pytest against an actual FirecREST server,
+# rather than just a mock server.
+
+name: Server
+
+on:
+  push:
+    branches: [main]
+    tags:
+      - 'v*'
+  pull_request:  # TODO delete
+
+jobs:
+
+  tests:
+
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: ["ubuntu-latest"]
+        python-version: ["3.9"]
+        firecrest-version: ["v1.13.0"]
+
+    services:
+     rabbitmq:
+        image: rabbitmq:3.8.14-management
+        ports:
+          - 5672:5672
+          - 15672:15672
+
+    steps:
+    - uses: actions/checkout@v3
+
+    - name: checkout the firecrest repository
+      uses: actions/checkout@v3
+      with:
+        repository: eth-cscs/firecrest
+        ref: ${{ matrix.firecrest-version }}
+        path: .demo-server
+
+    - name: Cache Docker images
+      uses: jpribyl/[email protected]
+      continue-on-error: true
+      with:
+        key: ${{ runner.os }}-docker-${{ matrix.firecrest-version }}
+
+    # note, for some reason, the certificator image fails to build
+    # if you build them in order, so here we build everything except that first
+    # and then it seems to work
+    - name: Build the FirecREST images
+      run: |
+        docker-compose build f7t-base
+        docker-compose build compute
+        docker-compose build status
+        docker-compose build storage
+        docker-compose build tasks
+        docker-compose build utilities
+        docker-compose build reservations
+        docker-compose build client
+        docker-compose build cluster
+        docker-compose build keycloak
+        docker-compose build kong
+        docker-compose build minio
+        docker-compose build taskpersistence
+        docker-compose build opa
+        docker-compose build openapi
+        docker-compose build jaeger
+
+        docker-compose build certificator
+      working-directory: .demo-server/deploy/demo
+
+    - name: Ensure permissions of SSH Keys
+      run: |
+        chmod 400 .demo-server/deploy/test-build/environment/keys/ca-key
+        chmod 400 .demo-server/deploy/test-build/environment/keys/user-key
+
+    - name: Start the FirecREST server
+      run: docker-compose up --detach
+      working-directory: .demo-server/deploy/demo
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.python-version }}
+        cache: 'pip'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e .[dev]
+    - name: Test with pytest
+      run: pytest -vv --cov=aiida_firecrest --firecrest-config .firecrest-demo-config.json

.github/workflows/tests.yml

@@ -43,6 +43,7 @@ jobs:
       uses: actions/setup-python@v4
       with:
         python-version: ${{ matrix.python-version }}
+        cache: 'pip'
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip

.gitignore

@@ -132,4 +132,3 @@ dmypy.json
 .vscode/
 .demo-server/
 _archive/
-.firecrest-config.json

README.md

@@ -1,6 +1,6 @@
 # aiida-firecrest [IN-DEVELOPMENT]
 
-AiiDA Transport/Scheduler plugins for interfacing with [FirecREST](https://products.cscs.ch/firecrest/) (currently based on [v1.12.0](https://github.com/eth-cscs/firecrest/releases/tag/v1.12.0)),
+AiiDA Transport/Scheduler plugins for interfacing with [FirecREST](https://products.cscs.ch/firecrest/) (currently based on [v1.13.0](https://github.com/eth-cscs/firecrest/releases/tag/v1.13.0)),
 via [pyfirecrest](https://github.com/eth-cscs/pyfirecrest).
 
 ## Usage
@@ -125,14 +125,18 @@ It is recommended to run the tests via [tox](https://tox.readthedocs.io/en/lates
 tox
 ```
 
-By default, the tests are run using a mock FirecREST server (in a temporary folder).
+By default, the tests are run using a mock FirecREST server, in a temporary folder
+(see [aiida_fircrest.utils_test.FirecrestConfig](aiida_firecrest/utils_test.py)).
+This allows for quick testing and debugging of the plugin, without needing to connect to a real server,
+but is obviously not guaranteed to be fully representative of the real behaviour.
+
 You can also provide connections details to a real FirecREST server:
 
 ```bash
-tox -- --firecrest-config=".firecrest-config.json"
+tox -- --firecrest-config=".firecrest-demo-config.json"
 ```
 
-The format of the `.firecrest-config.json` file is:
+The format of the `.firecrest-demo-config.json` file is:
 
 ```json
 {
@@ -148,8 +152,8 @@ The format of the `.firecrest-config.json` file is:
 In this mode, if you want to inspect the generated files, after a failure, you can use:
 
 ```bash
-tox -- --firecrest-config=".firecrest-config.json" --firecrest-no-clean
+tox -- --firecrest-config=".firecrest-demo-config.json" --firecrest-no-clean
 ```
 
 See [firecrest_demo.py](firecrest_demo.py) for how to start up a demo server.
-(note the issue with OSX abd turning off the AirPlay port)
+(note the issue with OSX and turning off the AirPlay port)

firecrest_demo.py

@@ -66,7 +66,6 @@ def main(args: list[str] | None = None):
 
     # ensure permissions of SSH keys (chmod 400)
     print("Ensuring permissions of SSH keys")
-    # tester/deploy/test-build/environment/keys/ca-key
     folder.joinpath("deploy", "test-build", "environment", "keys", "ca-key").chmod(
         0o400
     )