feat: backend tests for project endpoints (#900)

* feat: Implement comprehensive test coverage for the Create Project API and related functions

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat: added used dataset for test cases in test_data

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: removed the larger geojson file with compressed zip in test_data

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: pydantic & sqlalchemy deprecation warnings

* fix: pytest sqlalchemy logs, additional deprecation warns

* test: fix user + org creation prior to project creation

* test: move test_data dir under tests

* test: use relative test_data_path over abs paths

* ci: pytest workflow expose central-proxy & loopback

* ci: use docker-compose stack for pytest workflow

* ci: pyest workflow set env prior to .env copy

* ci: pytest workflow only set essential vars

* build(ci): compose mount pyproject.toml in ci container

* ci: add DEBUG and LOG_LEVEL to pytest workflow

* build(ci): ci add tag override for migrations too

* refactor(backend): default to LOG_LEVEL=INFO in backend config

* ci: add FRONTEND_MAIN_URL to pytest workflow env

* build: set backend ci img entrypoint to sleep infinity

* ci: run pytest via docker compose run (capture stdout)

* build(dockerfile): add wait-for-it to backend runtime stage

* ci: add wait-for-it for db prior to pytest run

* test: fix project_route tests (remaining appuser files)

* test: create odkcentral project with fmtm project

* fix: handle qrcodes entirely in memory (no png files)

* test: fix generate_appuser_files, pass xlsform path

* ci(pytest): wait for odk central to start before run

* ci(pytest): missed end of command for chaining

* build: ci image from debug-with-odk (bundle cert)

* ci(pytest): timeout 30s for central wait-for-it

* ci(pytest): further increase central startup timeout

* ci(pytest): add -strict to fail if wait-for-it false

* ci(pytest): --use-aliases flag for docker compose run

* build: add central as depends_on svc for api

* ci(pytest): use docker network names over dc svc names

* docs: for act gh workflow testing

* ci(pytest): remove long timeout for cental_api wait-for-it

* ci(pytest): debugging for central_api

* ci(pytest): add sleep infinity entrypoint to api

* ci(pytest): further debug, sleep 20 prior to cental logs

* ci(pytest): update default CENTRAL_DB_HOST=central-db

* ci(pytest): debug docker-compose content

* ci(pytest): further debug docker-compose config

* ci(pytest): finalise pytest workflow

* ci(pytest): revert sleep infinity entrypoint

* ci(pytest): check .env vars

* ci(pytest): access secreys directly from env

* ci: pytest

comment out image cache for now

* ci(pytest): secrets to env first, before dotenv

* ci(pytest): specify secrets: inherit to pytest

* ci(pytest): cache container images on pytest

* ci(pytest): reorder image caching in workflow

* ci(pytest): run cache steps as distinct jobs

* ci(pytest): rename odk cache jobs

* ci(pytest): set needs for image caches

* ci: pytest

update img cache to not use slashes

* ci(pytest): cache directly in the workflow (no reusable)

* ci(docs): fix doxygen openapi file caching

* ci(pytest): simultaneous image caching for workflow

* ci: use reusable image_cache workflow

* ci(pytest): fix dependent job naming

---------

Co-authored-by: sujanadh <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: spwoodcock <[email protected]>

.github/workflows/docs.yml

@@ -11,42 +11,32 @@ on:
   workflow_dispatch:
 
 jobs:
-  get_cache_key:
-    runs-on: ubuntu-latest
-    outputs:
-      cache_key: ${{ steps.set_cache_key.outputs.cache_key }}
-    steps:
-      - name: Set cache key
-        id: set_cache_key
-        run: echo "cache_key=docs-build-$(date --utc +'%Y-%m-%d_%H:%M:%S')" >> $GITHUB_OUTPUT
-
   build_doxygen:
     uses: hotosm/gh-workflows/.github/workflows/doxygen_build.yml@main
-    needs: [get_cache_key]
     with:
       cache_paths: |
         docs/apidocs
-      cache_key: ${{ needs.get_cache_key.outputs.cache_key }}
+        docs/openapi.json
+      cache_key: docs-build
 
   build_openapi_json:
     uses: hotosm/gh-workflows/.github/workflows/openapi_build.yml@main
-    needs: [get_cache_key]
     with:
       image: ghcr.io/hotosm/fmtm/backend:ci-${{ github.ref_name }}
       example_env_file_path: ".env.example"
       cache_paths: |
+        docs/apidocs
         docs/openapi.json
-      cache_key: ${{ needs.get_cache_key.outputs.cache_key }}
+      cache_key: docs-build
 
   publish_docs:
     uses: hotosm/gh-workflows/.github/workflows/mkdocs_build.yml@main
     needs:
-      - get_cache_key
       - build_doxygen
       - build_openapi_json
     with:
       image: ghcr.io/hotosm/fmtm/backend:ci-${{ github.ref_name }}
       cache_paths: |
         docs/apidocs
         docs/openapi.json
-      cache_key: ${{ needs.get_cache_key.outputs.cache_key }}
+      cache_key: docs-build

.github/workflows/pr_test.yml

@@ -17,5 +17,7 @@ jobs:
     uses: ./.github/workflows/r-pytest.yml
     with:
       image_tag: ci-${{ github.base_ref }}
+    secrets: inherit
+
   frontend-tests:
     uses: ./.github/workflows/r-frontend_tests.yml

.github/workflows/r-pytest.yml

@@ -14,68 +14,84 @@ permissions:
   contents: read
 
 jobs:
-  test:
+  cache-img-postgis:
+    uses: hotosm/gh-workflows/.github/workflows/image_cache.yml@main
+    with:
+      image_name: postgis/postgis:14-3.3-alpine
+      cache_key: img-postgis
+
+  cache-img-odk:
+    uses: hotosm/gh-workflows/.github/workflows/image_cache.yml@main
+    with:
+      image_name: ghcr.io/hotosm/fmtm/odkcentral:v2023.2.1
+      cache_key: img-odk
+
+  cache-img-odk-proxy:
+    uses: hotosm/gh-workflows/.github/workflows/image_cache.yml@main
+    with:
+      image_name: ghcr.io/hotosm/fmtm/odkcentral-proxy:latest
+      cache_key: img-odk-proxy
+
+  run-pytest:
     runs-on: ubuntu-latest
     environment:
       name: ${{ inputs.environment || 'test' }}
-    env:
-      ODK_CENTRAL_PASSWD: ${{ secrets.ODK_CENTRAL_PASSWD }}
-      OSM_CLIENT_ID: ${{ secrets.OSM_CLIENT_ID }}
-      OSM_CLIENT_SECRET: ${{ secrets.OSM_CLIENT_SECRET }}
-      OSM_SECRET_KEY: ${{ secrets.OSM_SECRET_KEY }}
+    needs:
+      - cache-img-postgis
+      - cache-img-odk
+      - cache-img-odk-proxy
 
-    container:
-      image: ghcr.io/hotosm/fmtm/backend:${{ inputs.image_tag }}
-      env:
-        ODK_CENTRAL_URL: ${{ vars.ODK_CENTRAL_URL }}
-        ODK_CENTRAL_USER: ${{ vars.ODK_CENTRAL_USER }}
-        ODK_CENTRAL_PASSWD: ${{ env.ODK_CENTRAL_PASSWD }}
-        OSM_CLIENT_ID: ${{ env.OSM_CLIENT_ID }}
-        OSM_CLIENT_SECRET: ${{ env.OSM_CLIENT_SECRET }}
-        OSM_SECRET_KEY: ${{ env.OSM_SECRET_KEY }}
-        FRONTEND_MAIN_URL: ${{ vars.FRONTEND_MAIN_URL }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
 
-    services:
-      # Start backend database
-      fmtm-db:
-        image: "postgis/postgis:14-3.3-alpine"
-        env:
-          POSTGRES_PASSWORD: fmtm
-          POSTGRES_DB: fmtm
-          POSTGRES_USER: fmtm
-        options: >-
-          --health-cmd pg_isready
-          --health-interval 10s
-          --health-timeout 5s
-          --health-retries 5
+      - name: Restore Img Caches
+        id: restore-imgs
+        uses: actions/cache@v3
+        with:
+          path: |
+            ${{ needs.cache-img-postgis.outputs.cache_path }}
+            ${{ needs.cache-img-odk.outputs.cache_path }}
+            ${{ needs.cache-img-odk-proxy-cache.outputs.cache_path }}
+          key: |
+            ${{ needs.cache-img-postgis.outputs.cache_key }}
+            ${{ needs.cache-img-odk.outputs.cache_key }}
+            ${{ needs.cache-img-odk-proxy-cache.outputs.cache_key }}
 
-      # Start ODK Central database
-      central-db:
-        image: "postgis/postgis:14-3.3-alpine"
-        env:
-          POSTGRES_PASSWORD: odk
-          POSTGRES_DB: odk
-          POSTGRES_USER: odk
-        options: >-
-          --health-cmd pg_isready
-          --health-interval 10s
-          --health-timeout 5s
-          --health-retries 5
+      - name: Load Cached Imgs
+        if: steps.restore-imgs.outputs.cache-hit == 'true'
+        run: |
+          docker image load --input ${{ needs.cache-img-postgis.outputs.cache_path }} || true
+          docker image load --input ${{ needs.cache-img-odk.outputs.cache_path }} || true
+          docker image load --input ${{ needs.cache-img-odk-proxy-cache.outputs.cache_path }} || true
 
-      # Start ODK Central
-      central:
-        image: "ghcr.io/hotosm/fmtm/odkcentral:v2023.2.1"
+      - name: Environment to .env
         env:
-          SYSADMIN_EMAIL: ${{ vars.ODK_CENTRAL_USER }}
-          SYSADMIN_PASSWD: ${{ secrets.ODK_CENTRAL_PASSWD }}
-          DB_HOST: central-db
+          DEBUG: True
+          LOG_LEVEL: DEBUG
+          API_TAG_OVERRIDE: "${{ inputs.image_tag }}"
+          FRONTEND_MAIN_URL: "${{ vars.FRONTEND_MAIN_URL }}"
+          ODK_CENTRAL_URL: "${{ vars.ODK_CENTRAL_URL }}"
+          ODK_CENTRAL_USER: "${{ vars.ODK_CENTRAL_USER }}"
+          ODK_CENTRAL_PASSWD: "${{ secrets.ODK_CENTRAL_PASSWD }}"
+          OSM_CLIENT_ID: "${{ secrets.OSM_CLIENT_ID }}"
+          OSM_CLIENT_SECRET: "${{ secrets.OSM_CLIENT_SECRET }}"
+          OSM_SECRET_KEY: "${{ secrets.OSM_SECRET_KEY }}"
+        run: |
+          echo "DEBUG=${DEBUG}" >> .env
+          echo "LOG_LEVEL=${LOG_LEVEL}" >> .env
+          echo "API_TAG_OVERRIDE=${API_TAG_OVERRIDE}" >> .env
+          echo "FRONTEND_MAIN_URL=${FRONTEND_MAIN_URL}" >> .env
+          echo "ODK_CENTRAL_URL=${ODK_CENTRAL_URL}" >> .env
+          echo "ODK_CENTRAL_USER=${ODK_CENTRAL_USER}" >> .env
+          echo "ODK_CENTRAL_PASSWD=${ODK_CENTRAL_PASSWD}" >> .env
+          echo "OSM_CLIENT_ID=${OSM_CLIENT_ID}" >> .env
+          echo "OSM_CLIENT_SECRET=${OSM_CLIENT_SECRET}" >> .env
+          echo "OSM_SECRET_KEY=${OSM_SECRET_KEY}" >> .env
 
-      # Start proxy to access ODK Central
-      central-proxy:
-        image: "ghcr.io/hotosm/fmtm/odkcentral-proxy:latest"
-
-    steps:
-      - uses: actions/checkout@v4
-      - name: Run pytest
-        working-directory: src/backend
-        run: pytest
+      - name: Run PyTest
+        run: |
+          docker compose run api \
+            wait-for-it fmtm-db:5432 --strict \
+            -- wait-for-it central:8383 --strict \
+            -- pytest

.github/workflows/tests/test_ci.sh

@@ -12,19 +12,31 @@ set -e
 # Feed to act using -s flag: -s GITHUB_TOKEN=input_personal_access_token
 
 # PR
-act pull_request -W .github/workflows/pr.yml -e .github/workflows/tests/pr_payload.json
+act pull_request -W .github/workflows/pr_test.yml \
+    -e .github/workflows/tests/pr_payload.json \
+    --var-file=.env --secret-file=.env
 
 # Build and deploy
-act push -W .github/workflows/build_and_deploy.yml -e .github/workflows/tests/push_payload.json
+act push -W .github/workflows/build_and_deploy.yml \
+    -e .github/workflows/tests/push_payload.json \
+    --var-file=.env --secret-file=.env
 
 # CI Img Build
-act push -W .github/workflows/build_ci_img.yml -e .github/workflows/tests/push_payload.json
+act push -W .github/workflows/build_ci_img.yml \
+    -e .github/workflows/tests/push_payload.json \
+    --var-file=.env --secret-file=.env
 
 # ODK Img Build
-act push -W .github/workflows/build_odk_imgs.yml -e .github/workflows/tests/push_payload.json
+act push -W .github/workflows/build_odk_imgs.yml \
+    -e .github/workflows/tests/push_payload.json \
+    --var-file=.env --secret-file=.env
 
 # Docs
-act push -W .github/workflows/docs.yml -e .github/workflows/tests/push_payload.json
+act push -W .github/workflows/docs.yml \
+    -e .github/workflows/tests/push_payload.json \
+    --var-file=.env --secret-file=.env
 
 # Wiki
-act push -W .github/workflows/wiki.yml -e .github/workflows/tests/push_payload.json
+act push -W .github/workflows/wiki.yml \
+    -e .github/workflows/tests/push_payload.json \
+    --var-file=.env --secret-file=.env

docker-compose.yml

@@ -45,12 +45,12 @@ services:
     restart: "unless-stopped"
 
   api:
-    image: "ghcr.io/hotosm/fmtm/backend:debug"
+    image: "ghcr.io/hotosm/fmtm/backend:${API_TAG_OVERRIDE:-debug}"
     build:
       context: src/backend
       target: debug-with-odk
       args:
-        APP_VERSION: debug
+        APP_VERSION: "debug"
     container_name: fmtm_api
     # Uncomment these to debug with a terminal debugger like pdb
     # Then `docker attach fmtm_api` to debug
@@ -60,7 +60,9 @@ services:
       - fmtm_logs:/opt/logs
       - fmtm_images:/opt/app/images
       - fmtm_tiles:/opt/tiles
+      - ./src/backend/pyproject.toml:/opt/pyproject.toml
       - ./src/backend/app:/opt/app
+      - ./src/backend/tests:/opt/tests
       # - ../osm-fieldwork/osm_fieldwork:/home/appuser/.local/lib/python3.10/site-packages/osm_fieldwork
     depends_on:
       - fmtm-db
@@ -76,7 +78,7 @@ services:
     restart: "unless-stopped"
 
   migrations:
-    image: "ghcr.io/hotosm/fmtm/backend:debug"
+    image: "ghcr.io/hotosm/fmtm/backend:${API_TAG_OVERRIDE:-debug}"
     container_name: fmtm_migrations
     depends_on:
       - fmtm-db
@@ -140,7 +142,7 @@ services:
       - SYSADMIN_EMAIL=${ODK_CENTRAL_USER}
       - SYSADMIN_PASSWD=${ODK_CENTRAL_PASSWD}
       - HTTPS_PORT=${HTTPS_PORT:-443}
-      - DB_HOST=${CENTRAL_DB_HOST:-postgres14}
+      - DB_HOST=${CENTRAL_DB_HOST:-central-db}
       - DB_USER=${CENTRAL_DB_USER:-odk}
       - DB_PASSWORD=${CENTRAL_DB_PASSWORD:-odk}
       - DB_NAME=${CENTRAL_DB_NAME:-odk}

src/backend/Dockerfile

@@ -90,6 +90,7 @@ RUN set -ex \
     -y --no-install-recommends \
         "nano" \
         "curl" \
+        "wait-for-it" \
         "libpcre3" \
         "mime-support" \
         "postgresql-client" \
@@ -137,21 +138,22 @@ CMD ["python", "-m", "debugpy", "--listen", "0.0.0.0:5678", \
 
 
 FROM debug-no-odk as debug-with-odk
-# Add the SSL cert for debug odkcentral
 USER root
+# Add the SSL cert for debug odkcentral
 COPY --from=ghcr.io/hotosm/fmtm/odkcentral-proxy:latest \
     /etc/nginx/central-fullchain.crt /usr/local/share/ca-certificates/
 RUN update-ca-certificates
 USER appuser
 
 
 
-FROM runtime as ci
+FROM debug-with-odk as ci
 # Run all ci as root
 USER root
 ARG PYTHON_IMG_TAG
 COPY --from=extract-deps \
     /opt/python/requirements-ci.txt /opt/python/
+# Copy packages from user to root dirs (run ci as root)
 RUN mv /home/appuser/.local/bin/* /usr/local/bin/ \
     && mv /home/appuser/.local/lib/python${PYTHON_IMG_TAG}/site-packages/* \
     /usr/local/lib/python${PYTHON_IMG_TAG}/site-packages/ \

src/backend/app/central/central_crud.py

@@ -23,7 +23,6 @@
 
 # import osm_fieldwork
 # Qr code imports
-import segno
 from fastapi import HTTPException
 from fastapi.responses import JSONResponse
 from loguru import logger as log
@@ -162,6 +161,7 @@ def create_appuser(
     project_id: int, name: str, odk_credentials: project_schemas.ODKCentral = None
 ):
     """Create an app-user on a remote ODK Server.
+
     If odk credentials of the project are provided, use them to create an app user.
     """
     if odk_credentials:
@@ -177,8 +177,13 @@ def create_appuser(
         pw = settings.ODK_CENTRAL_PASSWD
 
     app_user = OdkAppUser(url, user, pw)
+
+    log.debug(
+        "ODKCentral: attempting user creation: name: " f"{name} | project: {project_id}"
+    )
     result = app_user.create(project_id, name)
-    log.info(f"Created app user: {result.json()}")
+
+    log.debug(f"ODKCentral response: {result.json()}")
     return result
 
 
@@ -422,6 +427,8 @@ def generate_updated_xform(
     """Update the version in an XForm so it's unique."""
     name = os.path.basename(xform).replace(".xml", "")
     outfile = xform
+
+    log.debug(f"Reading xlsform: {xlsform}")
     if form_type != "xml":
         try:
             xls2xform_convert(xlsform_path=xlsform, xform_path=outfile, validate=False)
@@ -553,10 +560,6 @@ def create_qrcode(project_id: int, token: str, name: str, odk_central_url: str =
     qr_data = base64.b64encode(
         zlib.compress(json.dumps(qr_code_setting).encode("utf-8"))
     )
-
-    # Generate qr code using segno
-    qrcode = segno.make(qr_data, micro=False)
-    qrcode.save(f"/tmp/{name}_qr.png", scale=5)
     return qr_data
 
 
@@ -689,7 +692,7 @@ def generate_updated_xform_for_janakpur(
     """Update the version in an XForm so it's unique."""
     name = os.path.basename(xform).replace(".xml", "")
 
-    print("Name in form = ", name)
+    log.debug(f"Name in form = {name}")
 
     outfile = xform
     if form_type != "xml":