feat: update API to use events and task states + fixes to backend ref…

…actor (#1838)

* build: add migrations to update task_history --> task_events

* refactor(backend): wip update to use TaskEvent / State enums

* refactor(frontend): wip update to use TaskEvent / State enums

* build: add trigger to set task_event.state automatically from event

* refactor: continued wip to use task events

* refactor: fixes to task state/event distinction

* fix: additional fixes to backend usage action --> event

* build: fix migration conflict if task_history does not exist

* build: add conflate to enums in case for future

* refactor: further wip updates for task events

* refactor(frontend): update enums

* feat: set db default for UUID --> gen_random_uuid()

* build: fix during development building postgis db images

* build: fix migrations

* fix: working event creation endpoint on tasks

* fix(frontend): update to use new /events endpoint

* fix: basemap generation via frontend (use POST)

* build: add validation and comment to automatic state trigger func

* test: fix task event + comment routes

* fix: usage of task events endpoints

* fix: activities panel status --> state

* build: move api healthcheck to docker-compose file

* build: fix up-mapper working with playwright

* build: update playwright on mapper frontend to match react frontend

* Fix/event driven tasks continuation (#1839)

* build: add migrations to update task_history --> task_events

* refactor(backend): wip update to use TaskEvent / State enums

* refactor(frontend): wip update to use TaskEvent / State enums

* build: add trigger to set task_event.state automatically from event

* refactor: continued wip to use task events

* refactor: fixes to task state/event distinction

* fix: additional fixes to backend usage action --> event

* build: fix migration conflict if task_history does not exist

* build: add conflate to enums in case for future

* refactor: further wip updates for task events

* fix(taskSelectionPopup): project task index display on task popup

* fix(submissionsInfographics): handle undefined task_history

* fix(dialogTaskActions): remove taskBoundary from args, pass geojsonStyles to as args

* fix(taskEvent): use response state to track and update current task state color

* fix(featureSelectionPopup): page crash on taskStateEnum undefined fix

* fix(getTaskStatusStyle): update feature style key based on entiy_state enums

* fix(projectInfo): change card title accessor key

* fix(taskSubmissionsMap): fix taskGeojson schema

* fix(submissionDetails): update get comments api

* fix(updateReviewStatusModal): update reviewList id, update post submission comments api

---------

Co-authored-by: spwoodcock <[email protected]>

* fix(backend): fix logic returning all tasks with project

* refactor: update logging on task event

* build: add NOT NULL constraints to required task_event fields

* fix(frontend): marking task as mapped when not all buildings done

* fix(frontend): getting task history comments=false, comments=true

* fix: enable tasks/activity endpoint until merged with dashboard

* build: add Just command to load prod data into current db

* fix: use timezone aware timezones from db, fix dashboard api

* build: ensure task_events.created_at has timezone

* test: fix e2e test expecting 'mapped' when 'unlocked_to_validate'

* fix(frontend): bug where task state was not updated in task dialog

* fix: remove duplicate use of state selector

* test: small fixes to playwright test strings to validate

* build: disable ui-mapper in playwright tests for now

* build: update cloudflare/cloudflared version 2024.5.0 --> 2024.10.1

* build: simplify justfile command for tunnelling

* fix(frontend): check for empty taskId when new geopoint submission

* fix: updating review state for a submission

---------

Co-authored-by: Nishit Suwal <[email protected]>

.env.example

@@ -18,7 +18,7 @@ API_PREFIX=${API_PREFIX}
 OSM_CLIENT_ID=${OSM_CLIENT_ID}
 OSM_CLIENT_SECRET=${OSM_CLIENT_SECRET}
 OSM_URL=${OSM_URL:-"https://www.openstreetmap.org"}
-OSM_SCOPE=${OSM_SCOPE:-["read_prefs", "send_messages"]}
+OSM_SCOPE=${OSM_SCOPE:-'["read_prefs", "send_messages"]'}
 OSM_LOGIN_REDIRECT_URI="http${FMTM_DOMAIN:+s}://${FMTM_DOMAIN:-127.0.0.1:7051}/osmauth/"
 OSM_SECRET_KEY=${OSM_SECRET_KEY}
 

Justfile

@@ -16,6 +16,8 @@
 #     along with FMTM.  If not, see <https:#www.gnu.org/licenses/>.
 #
 
+set dotenv-load
+
 mod start 'contrib/just/start/Justfile'
 mod stop 'contrib/just/stop/Justfile'
 mod build 'contrib/just/build/Justfile'

contrib/just/start/Justfile

@@ -78,7 +78,7 @@ tunnel:
   docker compose \
     -f docker-compose.yml \
     -f contrib/tunnel/docker-compose.yml \
-    up -d --wait
+    up --wait
 
   # Workaround to until PR merged:
   # https://github.com/cloudflare/cloudflared/pull/1135
@@ -93,18 +93,14 @@ tunnel:
   just --unstable dotenv update "EXTRA_CORS_ORIGINS" "${fmtm_url}"
   just --unstable dotenv update "S3_ENDPOINT" "${s3_url}"
 
-  # Restart the API and UI with environment variables set
-  API_URL="${api_url}" docker compose \
-    -f docker-compose.yml \
-    -f contrib/tunnel/docker-compose.yml \
-    up -d api ui
-
-  # Restart ODK Central with domain override (for form download urls)
+  # Restart the containers with env vars set
+  # (API url for frontend, domain for Central form download urls)
   CENTRAL_DOMAIN_OVERRIDE="$(echo "${odk_url}" | sed 's|^https://||')" \
+  API_URL="$(echo "${api_url}" | sed 's|^https://||')" \
   docker compose \
     -f docker-compose.yml \
     -f contrib/tunnel/docker-compose.yml \
-    up -d central
+    up -d api ui central
 
   just --unstable start _print-tunnel-urls "$fmtm_url" "$api_url" "$odk_url" "$s3_url"
 

contrib/just/test/Justfile

@@ -55,3 +55,37 @@ frontend-interactive:
 coverage:
   docker compose run --rm --entrypoint='sh -c' api \
     'coverage run -m pytest && coverage report -m'
+
+# Load prod data into current database (WARNING: deletes local db data)
+[no-cd]
+load-prod-data:
+  #!/usr/bin/env sh
+  cd {{justfile_directory()}}
+
+  docker compose up --wait
+  # We cannot have electric using a logical replication slot though
+  docker compose down electric
+
+  # Get latest db dump filename
+  docker compose exec --no-TTY s3 mc alias set prod https://s3.fmtm.hotosm.org "" ""
+  latest_file=$(docker compose exec --no-TTY s3 mc ls prod/fmtm-db-backups/fmtm \
+    | awk '{print $NF}' | sort | tail -n 1)
+  echo "Latest backup file: $latest_file"
+
+  # Copy file to current machine
+  docker compose exec --no-TTY s3 \
+    mc cp prod/fmtm-db-backups/fmtm/"$latest_file" /tmp/"$latest_file"
+  docker compose cp s3:/tmp/"$latest_file" /tmp/"$latest_file"
+
+  echo "Dropping existing database ${FMTM_DB_NAME} as user ${FMTM_DB_USER}"
+  docker compose exec --no-TTY -e PGPASSWORD=${FMTM_DB_PASSWORD} ${FMTM_DB_HOST} \
+    dropdb --echo --if-exists --force -U ${FMTM_DB_USER}  ${FMTM_DB_NAME}
+
+  echo "Creating new database ${FMTM_DB_NAME} as user ${FMTM_DB_USER}"
+  docker compose exec --no-TTY -e PGPASSWORD=${FMTM_DB_PASSWORD} ${FMTM_DB_HOST} \
+    createdb --echo -U ${FMTM_DB_USER} -O ${FMTM_DB_USER} ${FMTM_DB_NAME}
+
+  echo "Loading data into database ${FMTM_DB_NAME} as user ${FMTM_DB_USER}"
+  gunzip -c /tmp/"$latest_file" | \
+  docker compose exec --no-TTY -e PGPASSWORD=${FMTM_DB_PASSWORD} ${FMTM_DB_HOST} \
+    pg_restore --verbose -U ${FMTM_DB_USER} -d ${FMTM_DB_NAME}

contrib/playwright/docker-compose.yml

@@ -30,12 +30,6 @@ services:
         "critical",
         "--no-access-log",
       ]
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8000/__lbheartbeat__"]
-      start_period: 60s
-      interval: 10s
-      timeout: 5s
-      retries: 10
 
   ui:
     # This hostname is used for Playwright test internal networking
@@ -50,6 +44,19 @@ services:
       start_period: 5s
       timeout: 5s
 
+  # ui-mapper:
+  #   # This hostname is used for Playwright test internal networking
+  #   hostname: fmtm.dev.test
+  #   environment:
+  #     VITE_API_URL: "http://fmtm.dev.test:8000"
+  #   command: --port 80
+  #   healthcheck:
+  #     test: timeout 5s bash -c ':> /dev/tcp/127.0.0.1/80' || exit 1
+  #     interval: 5s
+  #     retries: 3
+  #     start_period: 5s
+  #     timeout: 5s
+
   ui-test:
     image: "mcr.microsoft.com/playwright:${PLAYWRIGHT_TAG:-v1.48.1}"
     depends_on:

contrib/tunnel/docker-compose.yml

@@ -21,7 +21,7 @@ networks:
 
 services:
   frontend-tunnel:
-    image: "docker.io/cloudflare/cloudflared:2024.5.0"
+    image: "docker.io/cloudflare/cloudflared:2024.10.1"
     depends_on:
       proxy:
         condition: service_healthy
@@ -31,7 +31,7 @@ services:
     command: tunnel --url http://proxy:80
 
   api-tunnel:
-    image: "docker.io/cloudflare/cloudflared:2024.5.0"
+    image: "docker.io/cloudflare/cloudflared:2024.10.1"
     depends_on:
       api:
         condition: service_healthy
@@ -41,7 +41,7 @@ services:
     command: tunnel --url http://api:8000
 
   central-tunnel:
-    image: "docker.io/cloudflare/cloudflared:2024.5.0"
+    image: "docker.io/cloudflare/cloudflared:2024.10.1"
     depends_on:
       central:
         condition: service_healthy
@@ -51,7 +51,7 @@ services:
     command: tunnel --url http://central:8383
 
   s3-tunnel:
-    image: "docker.io/cloudflare/cloudflared:2024.5.0"
+    image: "docker.io/cloudflare/cloudflared:2024.10.1"
     depends_on:
       s3:
         condition: service_healthy

docker-compose.yml

@@ -43,6 +43,8 @@ services:
         condition: service_started
       ui:
         condition: service_started
+      ui-mapper:
+        condition: service_started
       central:
         condition: service_started
         required: false
@@ -97,6 +99,12 @@ services:
     networks:
       - fmtm-net
     restart: "unless-stopped"
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/__lbheartbeat__"]
+      start_period: 60s
+      interval: 10s
+      timeout: 5s
+      retries: 10
     deploy:
       replicas: ${API_REPLICAS:-1}
       resources:
@@ -237,7 +245,7 @@ services:
     image: "postgis/postgis:${POSTGIS_TAG:-14-3.5-alpine}"
     # Temp workaround until https://github.com/postgis/docker-postgis/issues/216
     build:
-      context: https://github.com/postgis/docker-postgis.git#master:14-3.4/alpine
+      context: https://github.com/postgis/docker-postgis.git#master:14-3.5/alpine
     command: -c 'max_connections=300' -c 'wal_level=logical'
     volumes:
       - fmtm_db_data:/var/lib/postgresql/data/
@@ -274,7 +282,7 @@ services:
       # AUTH_JWT_KEY: ${ENCRYPTION_KEY}
       # AUTH_JWT_AUD: ${FMTM_DOMAIN}
     ports:
-      - "7055:3000"
+      - "7055:7055"
     networks:
       - fmtm-net
     restart: "unless-stopped"

nginx/templates/dev/fmtm.conf.template

@@ -27,7 +27,7 @@ upstream mapper {
     # Enable sticky sessions based on an incoming client IP address
     ip_hash;
 
-    server ui-mapper:3000;
+    server ui-mapper:7055;
 }
 
 server {

src/backend/app/central/central_crud.py

@@ -33,7 +33,7 @@
 
 from app.central import central_deps, central_schemas
 from app.config import settings
-from app.db.enums import EntityStatus, HTTPStatus
+from app.db.enums import EntityState, HTTPStatus
 from app.db.models import DbXLSForm
 from app.db.postgis_utils import (
     geojson_to_javarosa_geom,
@@ -511,7 +511,7 @@ async def feature_geojson_to_entity_dict(
     properties = {
         str(key): str(value) for key, value in feature.get("properties", {}).items()
     }
-    # Set to TaskStatus enum READY value (0)
+    # Set to MappingState enum READY value (0)
     properties["status"] = "0"
 
     task_id = properties.get("task_id")
@@ -769,7 +769,7 @@ async def update_entity_mapping_status(
     odk_id: int,
     entity_uuid: str,
     label: str,
-    status: EntityStatus,
+    status: EntityState,
     dataset_name: str = "features",
 ) -> dict:
     """Update the Entity mapping status.
@@ -781,7 +781,7 @@ async def update_entity_mapping_status(
         odk_id (str): The project ID in ODK Central.
         entity_uuid (str): The unique entity UUID for ODK Central.
         label (str): New label, with emoji prepended for status.
-        status (EntityStatus): New EntityStatus to assign, in string form.
+        status (EntityState): New EntityState to assign, in string form.
         dataset_name (str): Override the default dataset / Entity list name 'features'.
 
     Returns:

src/backend/app/central/central_schemas.py

@@ -26,7 +26,7 @@
 from pydantic.functional_validators import field_validator, model_validator
 
 from app.config import HttpUrlStr, decrypt_value, encrypt_value
-from app.db.enums import EntityStatus
+from app.db.enums import EntityState
 
 
 class ODKCentral(BaseModel):
@@ -225,7 +225,7 @@ class EntityMappingStatus(EntityOsmID, EntityTaskID):
     """The status for mapping an Entity/feature."""
 
     updatedAt: Optional[str] = Field(exclude=True)  # noqa: N815
-    status: Optional[EntityStatus] = None
+    status: Optional[EntityState] = None
 
     @computed_field
     @property
@@ -238,18 +238,18 @@ class EntityMappingStatusIn(BaseModel):
     """Update the mapping status for an Entity."""
 
     entity_id: str
-    status: EntityStatus
+    status: EntityState
     label: str
 
     @field_validator("label", mode="before")
     @classmethod
     def append_status_emoji(cls, value: str, info: ValidationInfo) -> str:
         """Add 🔒 (locked), ✅ (complete) or ❌ (invalid) emojis."""
-        status = info.data.get("status", EntityStatus.UNLOCKED.value)
+        status = info.data.get("status", EntityState.READY.value)
         emojis = {
-            str(EntityStatus.LOCKED.value): "🔒",
-            str(EntityStatus.MAPPED.value): "✅",
-            str(EntityStatus.BAD.value): "❌",
+            str(EntityState.OPENED_IN_ODK.value): "🔒",
+            str(EntityState.SURVEY_SUBMITTED.value): "✅",
+            str(EntityState.MARKED_BAD.value): "❌",
         }
 
         # Remove any existing emoji at the start of the label
@@ -265,6 +265,6 @@ def append_status_emoji(cls, value: str, info: ValidationInfo) -> str:
 
     @field_validator("status", mode="after")
     @classmethod
-    def integer_status_to_string(cls, value: EntityStatus) -> str:
+    def integer_status_to_string(cls, value: EntityState) -> str:
         """Convert integer status to string for ODK Entity data."""
         return str(value.value)