Merge Release/2024 q2 into master

.github/workflows/ci.yml

@@ -6,7 +6,7 @@ jobs:
   build_api:
     runs-on: ubuntu-latest
     container:
-      image: python:3.9
+      image: python:3.12.2
       options: --user root
     steps:
       - name: Checkout ${{ github.event.repository.name }}
@@ -128,9 +128,9 @@ jobs:
           sonar.projectKey=usdot-jpo-ode_jpo-cvmanager
           sonar.projectName=jpo-cvmanager
           sonar.python.coverage.reportPaths=$GITHUB_WORKSPACE/services/cov.xml
-          sonar.python.version=3.12
+          sonar.python.version=3.12.2
           api.sonar.projectBaseDir=$GITHUB_WORKSPACE/services
-          api.sonar.sources=addons/images/bsm_query,addons/images/count_metric,addons/images/firmware_manager,addons/images/iss_health_check,addons/images/rsu_ping,api/src,common/pgquery.py
+          api.sonar.sources=addons/images/geo_msg_query,addons/images/count_metric,addons/images/firmware_manager,addons/images/iss_health_check,addons/images/rsu_status_check,api/src,common/pgquery.py
           api.sonar.tests=addons/tests,api/tests,common/tests
           webapp.sonar.projectBaseDir=$GITHUB_WORKSPACE/webapp
           webapp.sonar.sources=src

.gitignore

@@ -6,4 +6,5 @@ cov.xml
 .venv
 cov_html
 htmlcov
-.pytest_cache
+.pytest_cache
+local_blob_storage

.vscode/settings.json

@@ -17,7 +17,7 @@
   },
   "python.envFile": "${workspaceFolder}/.env",
   "terminal.integrated.env.windows": {
-    "PYTHONPATH": "${workspaceFolder}/services;${workspaceFolder}/services/addons/images/bsm_query;${workspaceFolder}/services/addons/images/count_metric;${workspaceFolder}/services/addons/images/firmware_manager;${workspaceFolder}/services/addons/images/iss_health_check;${workspaceFolder}/services/addons/images/rsu_ping_fetch;${workspaceFolder}/services/api/src;${workspaceFolder}/services/common"
+    "PYTHONPATH": "${workspaceFolder}/services;${workspaceFolder}/services/addons/images/geo_msg_query;${workspaceFolder}/services/addons/images/count_metric;${workspaceFolder}/services/addons/images/firmware_manager;${workspaceFolder}/services/addons/images/iss_health_check;${workspaceFolder}/services/addons/images/rsu_status_check;${workspaceFolder}/services/api/src;${workspaceFolder}/services/common"
   },
   "[python]": {
     "editor.defaultFormatter": "ms-python.black-formatter",

README.md

@@ -6,7 +6,7 @@ The JPO Connected Vehicle Manager is a web-based application that helps an organ
 
 <b>GUI:</b> ReactJS with Redux Toolkit and Mapbox GL
 
-<b>API:</b> Python
+<b>API:</b> Python 3.12.2
 
 <b>Features:</b>
 
@@ -88,7 +88,46 @@ The following steps are intended to help get a new user up and running the JPO C
 
 ### Debugging
 
-Note that it is recommended to work with the Python API from a [virtual environment](https://docs.python.org/3/library/venv.html). See [Visual Studio Code](https://code.visualstudio.com/docs/python/environments) documentation for more information on how to set up a virtual environment in VS Code.
+Note that it is recommended to work with the Python API from a [virtual environment](https://docs.python.org/3/library/venv.html).
+
+#### Setting up a virtual environment from the command line
+
+1. Verify that you have Python 3.12.2 installed on your machine by running the following command:
+   ```bash
+   python3.12 --version
+   ```
+   ```cmd
+   python --version
+   ```
+   If you have a different version installed, download and install Python 3.12.2 from the [Python website](https://www.python.org/downloads/).
+2. Open a terminal and navigate to the root of the project.
+3. Run the following command to create a virtual environment in the project root:
+   ```bash
+   python3.12 -m venv .venv
+   ```
+   ```cmd
+   python -m venv .venv
+   ```
+4. Activate the virtual environment:
+   ```bash
+   source .venv/bin/activate
+   ```
+   ```cmd
+   .venv\Scripts\activate
+   ```
+5. Install the required packages:
+   ```bash
+   pip3.12 install -r services/requirements.txt
+   ```
+   ```cmd
+   pip install -r services/requirements.txt
+   ```
+
+#### Setting up a virtual environment with VSCode
+
+See [Visual Studio Code](https://code.visualstudio.com/docs/python/environments) documentation for information on how to set up a virtual environment with VS Code.
+
+#### Debugging Profile
 
 A debugging profile has been set up for use with VSCode to allow ease of debugging with this application. To use this profile, simply open the project in VSCode and select the "Debug" tab on the left side of the screen. Then, select the "Debug Solution" profile and click the green play button. This will spin up a postgresql instance as well as the keycloak auth solution within docker containers. Once running, this will also start the debugger and attach it to the running API container. You can then set breakpoints and step through the code as needed.
 
@@ -137,17 +176,16 @@ For the "Debug Solution" to run properly on Windows 10/11 using WSL, the followi
 - WEBAPP_DOMAIN: The domain that the webapp will run on. This is required for Keycloak CORS authentication.
 - API_URI: The endpoint for the CV manager API, must be on a Keycloak Authorized domain.
 - COUNT_MESSAGE_TYPES: List of CV message types to query for counts.
+- VIEWER_MSG_TYPES: List of CV message types to query geospatially.
 - DOT_NAME: The name of the DOT using the CV Manager.
 - MAPBOX_INIT_LATITUDE: Initial latitude value to use for MapBox view state.
 - MAPBOX_INIT_LONGITUDE: Initial longitude value to use for MapBox view state.
 - MAPBOX_INIT_ZOOM: Initial zoom value to use for MapBox view state.
 
 <b>API Variables</b>
 
-- COUNTS_DB_TYPE: Set to either "MongoDB" or "BigQuery" depending on where the message counts are stored.
 - COUNTS_MSG_TYPES: Set to a list of message types to include in counts query. Sample format is described in the sample.env.
-- COUNTS_DB_NAME: The BigQuery table or MongoDB collection name where the RSU message counts are located.
-- BSM_DB_NAME: The database name for BSM visualization data.
+- GEO_DB_NAME: The database name for geospatial message visualization data. This is currently only supported for BSM and PSM message types.
 - SSM_DB_NAME: The database name for SSM visualization data.
 - SRM_DB_NAME: The database name for SRM visualization data.
 - FIRMWARE_MANAGER_ENDPOINT: Endpoint for the firmware manager deployment's API.
@@ -158,7 +196,7 @@ For the "Debug Solution" to run properly on Windows 10/11 using WSL, the followi
 - CSM_TARGET_SMTP_SERVER_ADDRESS: Destination SMTP server address.
 - CSM_TARGET_SMTP_SERVER_PORT: Destination SMTP server port.
 - API_LOGGING_LEVEL: The level of which the CV Manager API will log. (DEBUG, INFO, WARNING, ERROR)
-- WZDX_ENDPOINT: WZDX datafeed enpoint.
+- WZDX_ENDPOINT: WZDX datafeed endpoint.
 - WZDX_API_KEY: API key for the WZDX datafeed.
 - TIMEZONE: Timezone to be used for the API.
 - GOOGLE_APPLICATION_CREDENTIALS: Path to the GCP service account credentials file. Attached as a volume to the CV manager API service.
@@ -172,7 +210,7 @@ For the "Debug Solution" to run properly on Windows 10/11 using WSL, the followi
 
 <b>MongoDB Variables</b>
 
-- MONGO_DB_URI: URI for the MongoDB connection.
+- MONGO_DB_URI: URI for the MongoDB connections.
 - MONGO_DB_NAME: Database name for RSU counts.
 
 <b>Keycloak Variables</b>
@@ -189,6 +227,8 @@ For the "Debug Solution" to run properly on Windows 10/11 using WSL, the followi
 - GOOGLE_CLIENT_ID: GCP OAuth2.0 client ID for SSO Authentication within keycloak.
 - GOOGLE_CLIENT_SECRET: GCP OAuth2.0 client secret for SSO Authentication within keycloak.
 
+Environment variables from addon services can also be set in the main `.env` file. These variables are defined in their own `README` files in the `services/addons/images` location of this repository.
+
 ## License Information
 
 Licensed under the Apache License, Version 2.0 (the "License"); you may not use this

docker-compose-addons.yml

@@ -4,16 +4,20 @@ include:
   - docker-compose.yml
 
 services:
-  # ADDONS:
-  jpo_bsm_query:
+  jpo_geo_msg_query:
     build:
       context: ./services
-      dockerfile: Dockerfile.bsm_query
-    image: bsm_query:latest
-    restart: always
+      dockerfile: Dockerfile.geo_msg_query
+    image: geo_msg_query:latest
+    restart: on-failure:3
+    environment:
+      MONGO_DB_URI: ${MONGO_DB_URI}
+      MONGO_DB_NAME: ${MONGO_DB_NAME}
+      MONGO_INPUT_COLLECTIONS: ${GEO_INPUT_COLLECTIONS}
+      MONGO_GEO_OUTPUT_COLLECTION: ${GEO_DB_NAME}
+      MONGO_TTL: ${GEO_TTL_DURATION}
 
-    env_file:
-      - ./services/addons/images/bsm_query/.env
+      LOGGING_LEVEL: ${GEO_LOGGING_LEVEL}
     logging:
       options:
         max-size: '10m'
@@ -24,26 +28,64 @@ services:
       context: ./services
       dockerfile: Dockerfile.count_metric
     image: count_metric:latest
-    restart: always
+    restart: on-failure:3
+    environment:
+      ENABLE_EMAILER: ${ENABLE_EMAILER}
+      DEPLOYMENT_TITLE: ${DEPLOYMENT_TITLE}
+
+      SMTP_SERVER_IP: ${SMTP_SERVER_IP}
+      SMTP_USERNAME: ${SMTP_USERNAME}
+      SMTP_PASSWORD: ${SMTP_PASSWORD}
+      SMTP_EMAIL: ${SMTP_EMAIL}
+      SMTP_EMAIL_RECIPIENTS: ${SMTP_EMAIL_RECIPIENTS}
+
+      MESSAGE_TYPES: ${COUNT_MESSAGE_TYPES}
+      PROJECT_ID: ${GCP_PROJECT_ID}
+      ODE_KAFKA_BROKERS: ${ODE_KAFKA_BROKERS}
+
+      PG_DB_HOST: ${PG_DB_HOST}
+      PG_DB_NAME: ${PG_DB_NAME}
+      PG_DB_USER: ${PG_DB_USER}
+      PG_DB_PASS: ${PG_DB_PASS}
 
-    env_file:
-      - ./services/addons/images/count_metric/.env
+      DESTINATION_DB: ${COUNT_DESTINATION_DB}
+
+      MONGO_DB_URI: ${MONGO_DB_URI}
+      MONGO_DB_NAME: ${MONGO_DB_NAME}
+      INPUT_COUNTS_MONGO_COLLECTION_NAME: ${INPUT_COUNTS_MONGO_COLLECTION_NAME}
+      OUTPUT_COUNTS_MONGO_COLLECTION_NAME: ${OUTPUT_COUNTS_MONGO_COLLECTION_NAME}
+
+      KAFKA_BIGQUERY_TABLENAME: ${KAFKA_BIGQUERY_TABLENAME}
+
+      LOGGING_LEVEL: ${COUNTS_LOGGING_LEVEL}
     logging:
       options:
         max-size: '10m'
         max-file: '5'
 
-  jpo_rsu_ping_fetch:
+  rsu_status_check:
     build:
       context: ./services
-      dockerfile: Dockerfile.rsu_ping_fetch
-    image: rsu_ping_fetch:latest
-    restart: always
+      dockerfile: Dockerfile.rsu_status_check
+    image: rsu_status_check:latest
+    restart: on-failure:3
+    environment:
+      RSU_PING: ${RSU_PING}
+      ZABBIX: ${ZABBIX}
+      RSU_SNMP_FETCH: ${RSU_SNMP_FETCH}
 
-    depends_on:
-      - cvmanager_postgres
-    env_file:
-      - ./services/addons/images/rsu_ping/.env
+      ZABBIX_ENDPOINT: ${ZABBIX_ENDPOINT}
+      ZABBIX_USER: ${ZABBIX_USER}
+      ZABBIX_PASSWORD: ${ZABBIX_PASSWORD}
+
+      STALE_PERIOD: ${STALE_PERIOD}
+
+      PG_DB_HOST: ${PG_DB_HOST}
+      PG_DB_NAME: ${PG_DB_NAME}
+      PG_DB_USER: ${PG_DB_USER}
+      PG_DB_PASS: ${PG_DB_PASS}
+
+      LOGGING_LEVEL: ${RSU_STATUS_LOGGING_LEVEL}
     logging:
       options:
         max-size: '10m'
@@ -54,12 +96,27 @@ services:
       context: ./services
       dockerfile: Dockerfile.iss_health_check
     image: iss_health_check:latest
-    restart: always
-
+    restart: on-failure:3
     depends_on:
       - cvmanager_postgres
-    env_file:
-      - ./services/addons/images/iss_health_check/.env
+    environment:
+      ISS_API_KEY: ${ISS_API_KEY}
+      ISS_API_KEY_NAME: ${ISS_API_KEY_NAME}
+      ISS_PROJECT_ID: ${ISS_PROJECT_ID}
+      ISS_SCMS_TOKEN_REST_ENDPOINT: ${ISS_SCMS_TOKEN_REST_ENDPOINT}
+      ISS_SCMS_VEHICLE_REST_ENDPOINT: ${ISS_SCMS_VEHICLE_REST_ENDPOINT}
+
+      PG_DB_HOST: ${PG_DB_HOST}
+      PG_DB_NAME: ${PG_DB_NAME}
+      PG_DB_USER: ${PG_DB_USER}
+      PG_DB_PASS: ${PG_DB_PASS}
+
+      PROJECT_ID: ${GCP_PROJECT_ID}
+      GOOGLE_APPLICATION_CREDENTIALS: ${GOOGLE_APPLICATION_CREDENTIALS}
+
+      LOGGING_LEVEL: ${ISS_LOGGING_LEVEL}
+    volumes:
+      - ${GOOGLE_APPLICATION_CREDENTIALS}:/google/gcp_credentials.json
     logging:
       options:
         max-size: '10m'
@@ -70,7 +127,7 @@ services:
       context: services
       dockerfile: Dockerfile.firmware_manager
     image: jpo_firmware_manager:latest
-    restart: always
+    restart: on-failure:3
 
     ports:
       - '8089:8080'
@@ -86,10 +143,16 @@ services:
       GCP_PROJECT: ${GCP_PROJECT}
       GOOGLE_APPLICATION_CREDENTIALS: '/google/gcp_credentials.json'
 
-      LOGGING_LEVEL: ${API_LOGGING_LEVEL}
+      FW_EMAIL_RECIPIENTS: ${FW_EMAIL_RECIPIENTS}
+      SMTP_SERVER_IP: ${SMTP_SERVER_IP}
+      SMTP_EMAIL: ${SMTP_EMAIL}
+      SMTP_USERNAME: ${SMTP_USERNAME}
+      SMTP_PASSWORD: ${SMTP_PASSWORD}
+      LOGGING_LEVEL: ${FIRMWARE_MANAGER_LOGGING_LEVEL}
     volumes:
       - ${GOOGLE_APPLICATION_CREDENTIALS}:/google/gcp_credentials.json
+      - ${HOST_BLOB_STORAGE_DIRECTORY}:/mnt/blob_storage
     logging:
       options:
         max-size: '10m'
-        max-file: '5'
+        max-file: '5'

docker-compose-webapp-deployment.yml

@@ -12,6 +12,7 @@ services:
         MAPBOX_TOKEN: ${MAPBOX_TOKEN}
         KEYCLOAK_HOST_URL: ${KEYCLOAK_DOMAIN} # e.g. http://localhost
         COUNT_MESSAGE_TYPES: ${COUNTS_MSG_TYPES}
+        VIEWER_MESSAGE_TYPES: ${VIEWER_MSG_TYPES}
         DOT_NAME: ${DOT_NAME}
         MAPBOX_INIT_LATITUDE: ${MAPBOX_INIT_LATITUDE}
         MAPBOX_INIT_LONGITUDE: ${MAPBOX_INIT_LONGITUDE}

docker-compose.yml

@@ -22,14 +22,14 @@ services:
       MONGO_DB_NAME: ${MONGO_DB_NAME}
 
       COUNTS_MSG_TYPES: ${COUNTS_MSG_TYPES}
-      COUNTS_DB_TYPE: ${COUNTS_DB_TYPE}
-      COUNTS_DB_NAME: ${COUNTS_DB_NAME}
       GOOGLE_APPLICATION_CREDENTIALS: '/google/gcp_credentials.json'
 
-      BSM_DB_NAME: ${BSM_DB_NAME}
+      GEO_DB_NAME: ${GEO_DB_NAME}
       SSM_DB_NAME: ${SSM_DB_NAME}
       SRM_DB_NAME: ${SRM_DB_NAME}
 
+      MAX_GEO_QUERY_RECORDS: ${MAX_GEO_QUERY_RECORDS}
+
       FIRMWARE_MANAGER_ENDPOINT: ${FIRMWARE_MANAGER_ENDPOINT}
 
       WZDX_API_KEY: ${WZDX_API_KEY}
@@ -66,6 +66,7 @@ services:
         MAPBOX_TOKEN: ${MAPBOX_TOKEN}
         KEYCLOAK_HOST_URL: http://${KEYCLOAK_DOMAIN}:8084/
         COUNT_MESSAGE_TYPES: ${COUNTS_MSG_TYPES}
+        VIEWER_MESSAGE_TYPES: ${VIEWER_MSG_TYPES}
         DOT_NAME: ${DOT_NAME}
         MAPBOX_INIT_LATITUDE: ${MAPBOX_INIT_LATITUDE}
         MAPBOX_INIT_LONGITUDE: ${MAPBOX_INIT_LONGITUDE}
@@ -118,13 +119,15 @@ services:
       KEYCLOAK_ADMIN: ${KEYCLOAK_ADMIN}
       KEYCLOAK_ADMIN_PASSWORD: ${KEYCLOAK_ADMIN_PASSWORD}
       WEBAPP_ORIGIN: http://${WEBAPP_DOMAIN}
+      WEBAPP_CM_ORIGIN: http://${WEBAPP_CM_DOMAIN}
       KC_HEALTH_ENABLED: true
       KC_DB: postgres
       KC_DB_URL: jdbc:postgresql://${PG_DB_HOST}/postgres?currentSchema=keycloak
       KC_DB_USERNAME: ${PG_DB_USER}
       KC_DB_PASSWORD: ${PG_DB_PASS}
       KC_HOSTNAME: ${KEYCLOAK_DOMAIN}
       KEYCLOAK_API_CLIENT_SECRET_KEY: ${KEYCLOAK_API_CLIENT_SECRET_KEY}
+      KEYCLOAK_CM_API_CLIENT_SECRET_KEY: ${KEYCLOAK_CM_API_CLIENT_SECRET_KEY}
       GOOGLE_CLIENT_ID: ${GOOGLE_CLIENT_ID}
       GOOGLE_CLIENT_SECRET: ${GOOGLE_CLIENT_SECRET}
     command:

docs/Release_notes.md

@@ -1,5 +1,32 @@
 ## JPO CV Manager Release Notes
 
+## Version 1.3.0
+
+### **Summary**
+This release includes enhanced MongoDB support, replacing GCP BigQuery in the CV Manager and integrating with the existing [Conflict Visualizer](https://github.com/usdot-jpo-ode/jpo-conflictvisualizer) MongoDB deployment. The web application now meets WCAG accessibility standards, featuring improved V2X data visualization and CV counts. Key updates include a daily aggregate of CV counts for better MongoDB query performance, Keycloak token refresh optimization, SNMP configurations pulled from PostgreSQL, support for PSM and TIM messages and new services like the RSU Status Checker. Additional enhancements include email alerts for firmware manager failures, a 'Contact Support' button on the 'Help' page and a filter for RSU vendors. The project now fully supports Python 3.12.2, includes various bug fixes and introduces several performance improvements across different modules.
+
+Enhancements in this release:
+
+- CDOT PR 69: Keycloak token refresh timer increased to reduce the frequency of site refreshes.
+- CDOT PR 67: Daily aggregate CV counts to improve CV Manager count query performance in MongoDB.
+- CDOT PR 66: Email alerts on firmware manager fail cases.
+- CDOT PR 62: 'Contact Support' button now present on the 'Help' page.
+- CDOT PR 61: CV Manager SNMP configurations now pull from PostgreSQL instead of directly from RSUs for performance.
+- CDOT PR 60: PSM message visualization support and changing 'BSM Visualizer' to 'V2X Visualizer'.
+- CDOT PR 59: CV Manager support for TIM messages.
+- CDOT PR 57: Firmware Manager upgrade queue for handling excessive numbers of simultaneous upgrades.
+- CDOT PR 56: Firmware Manager post-upgrade bash script support.
+- CDOT PR 54: CV Manager full support of MongoDB instead of GCP BigQuery.
+- CDOT PR 52: Rework the existing counter to utilize MongoDB.
+- CDOT PR 51: RSU vendor filter added to the CV Manager web application.
+- CDOT PRs 45-50: Updates to visual elements of the CV Manager web application to meet WCAG standard requirements for accessibility.
+- CDOT PR 44: Project updated to fully support Python 3.12.2.
+- CDOT PR 42: Adds support for a unique encryption SNMP password separate from the authentication password.
+- CDOT PR 38: RSU status check service added to perform regular, automated pings and SNMP message forwarding configuration checks on RSUs within PostgreSQL.
+- CDOT PR 37: URL page routing for the CV Manager web application.
+- CDOT PR 36: Keycloak realm updates to support the Conflict Visualizer realm within the same Keycloak deployment as the CV Manager.
+- Additional bug fixes
+
 ## Version 1.2.0
 
 ### **Summary**