mock-cve-data

Getting Started

• npm install from the root of the repository will install the needed dependencies
• In the extensions pane, search for @recommended, and install the recommended extensions
  • All team code linting / formatting will happen upon saving
• npm run watch will spin up the api in watch mode (code changes will refresh)

Testing

Endpoints are tested with jest and the supertest library. An example is available in the app.test.ts file.

• npm run test:watch will run all *.test.ts files in watch mode
• npm run view:coverage will startup a server to serve the coverage report for you to view in your browser

Deployment

Deployment to the azure environment happens manually with the following commands. (You'll need to be configured to have azure cluster access in advance)

• Build a docker image: docker build . -t smartcitiesdata/mock-cve-data:latest
• Push that new image to docker hub: docker push smartcitiesdata/mock-cve-data:latest
• Remove existing mock-cve-data release: helm uninstall mock-cve-data -n urban-os-demo-services
• Helm install a new release: helm install mock-cve-data ./chart -n urban-os-demo-services --create-namespace

note: It'd be nice if we could just upgrade the helm release, but that hasn't been pulling a new image even though the pull policy in the chart is set to Always

• Upgrading the api command: helm upgrade mock-cve-data ./chart -n urban-os-demo-services

UI

The UI is served by the application from the root (:3000/), and is accessible once the application is running.

API Dependencies Note

• Anything needed by typescript to build (including @types packages) need to be included in the dependencies, not the devDependencies. This could be changed to be more conventional, but the dockerfile steps would also need to be updated. This is because at this time, the docker image performs a typescript build in addition to starting the app.

Where data came from: cve_path_json and scripts/clean_cve_json.ts

Here's an outline of how mock CVE data was gathered for usage in this API. Might be helpful if we end up needing to pull new CVE data for new paths.

• IDs queried (sql_queries.md)
• Messages for those IDs queried (sql_queries.md)
• visualize the resulting messages in kepler
• if it looks reasonable, use the rectangle select to filter a layer, export only that region
• convert filtered csvs to .json for easier use in javascript (https://csvjson.com/)
• those json files are to be placed in the CVE_PATH_JSON directory with the name {path_id}.json
• Update the PATH_IDS variable in the configuration file to include the new additions
• Run npm run script:cleancve
  • this will iterate through all the PATH_IDS and create a {path_id}_cleaned.json file
  • this file is what the Mock API loads in to serve on the /data endpoint

Note: Currently the "cleaned" CVE data is sorted by time. When we eventually visualize cars traveling through this data, if we see jumps in travel, we may need to adjust our SQL query to return data for an ID within a range of time. This would ensure that data is more likely to have been collected within a single trip of travel.

Example of data available on the /data endpoint

[{"lat":"40.0606922","long":"-82.9771919","speed":23.6160391458816,"speed_unit":"kph","timestamp":"2021-08-16T16:06:52.850Z","path_id":"7635341","vehicle_id":"1494ff0a-ba3e-4cbb-86be-0c4f3c3e06de","is_crashed":false,"crash_type":null},{"lat":"40.017488","long":"-83.0120681","speed":29.8080494097408,"speed_unit":"kph","timestamp":"2021-08-31T18:25:39.450Z","path_id":"603497707","vehicle_id":"6d10e21b-91ea-4399-8f15-4706452616b7","is_crashed":false,"crash_type":null} ... ]