Improve info parsing and consistency check.

Summary

Make several small improvements based on real-life testing by @time-trader.

Contents (#25)

Enhancements

• No longer output timestamp check warnings in case of IMU sensor deviations to reduce log clutter
• Re-initialize previous timestamps when the wake info packet is received
• Exiting info packet parsing after the specific info packet is parsed

Fixes

• Reinitialize sensor_time_offset upon handle update

Add ability to add sensor types to BigQuery dataset via CLI

Summary

Add the ability to add sensor types to the BigQuery dataset via the CLI.

Note: For some reason, the coverage calculations showing on this PR are incorrect e.g. packet_reader.py is untouched.

Contents (#40)

New features

• Add add-sensor-type CLI command

Enhancements

• Convert CLI log messages to print statements

Fixes

• Add "Diff_Baros" sensor type to BigQuery sensor mapping

Operations

• Add add-sensor-type cloud function deployment to cloudbuild.yaml

Dependencies

• Loosen python-slugify dependency

Testing

• Factor out repeated environment patching in test classes

Quality Checklist

• New features are fully tested (No matter how much Coverage Karma you have)
• [v0.2 onward] New features are included in the documentation
• [v0.2 onward] Breaking changes are documented with clear instructions of what

Coverage Karma

• If your PR decreases test coverage, do you feel you have built enough Coverage Karma* to justify it?

*Coverage Karma can be earned by disproportionally increasing test coverage in the rest of your contributions.
It's an honesty policy - you keep count of your own.

Paramaterize DataGateway tests

Contents (#38)

Testing

• Paramaterize DataGateway tests

Add multi-processing

Summary

Convert data-gateway to run on multiple processes and make it more efficient to avoid serial port buffer overflows.

Note: the test coverage has appeared to drop because the test runner isn't taking into account code run in other processes - this will increase back to current levels once the runner is configured to do this.

Contents (#30)

New features

• Run packet reading and packet parsing in different processes, using a process-safe queue to send packets from the reader to the parser

Enhancements

• Adjust log levels for cleaner output
• Continue running if an unknown packet is read from serial port
• Allow specific time after which to stop gateway if when no new data received
• Add process name to logging context
• Use multiprocessing logger throughout data_gateway package
• Log the location(s) that data will be persisted to
• Stop the closing and re-opening of the serial port on buffer overflow
• Log a warning if no routine is provided in non-interactive mode
• Stop the gateway if there's an error in either of the commands threads

Fixes

• Add force_persist method to NoOperationContextManager
• Stop packet reader when a routine finishes
• Ensure packet reader has correct default output directory
• Ensure routines containing a stop command only schedule and run the commands up until and including the stop command
• Ensure routines don't stop the gateway if/when they time out

Refactoring

• Move logic for starting gateway into new DataGateway class
• Rename PacketReader.parse_payload to parse_packets
• Confine notion of session_subdirectory to PacketReader and remove it from the persistence module

Testing

• Use simpler 0 and 1 values for RUN_DEPLOYMENT_TESTS environment variable
• Avoid potentially missing environment variable when skipping test
• Add missing bucket and project arguments in CLI tests
• Remove CredentialsEnvironmentVariableAsFile
• Mock BiqQuery client in tests
• Only save locally in CLI tests
• Remove problematic test (Windows)
• Move non-Cloud Function tests into test_data_gateway subpackage
• Remove use of temporary directories in DataGateway tests to avoid colons in cloud paths on Windows

Use separate threads for serial port reading and packet parsing

Summary

Separate serial port reading from packet parsing and data persistence by running them in different threads. This helps avoid serial port buffer overflow. Unlike #15 and #22, this pull request doesn't alter the way packets are read from the serial port.

Contents (#23)

New features

• Use separate threads for serial port reading and packet parsing

Enhancements

• Add thread name to logging context
• Set thread names to useful values

Fixes

• Raise errors from parser thread in reader thread

Operations

• Remove MacOS test runner from CI workflow
• Make release depend on tests passing after merge into main
• Run pull request description update on all branches but skip it if the skip indicator appears in the pull request description

Quality Checklist

• New features are fully tested (No matter how much Coverage Karma you have)

Avoid mutating configuration when adding to BigQuery

Contents (#24)

Fixes

• Avoid mutating configuration objects when adding to BigQuery table

Refactoring

• Factor out table names to instance attributes in BigQueryDataset

Operations

• Add deployment of test cloud function to cloudbuild.yaml
• Remove MacOS from CI test runners

Testing

• Add deployment test for test-ingress-eu cloud function

Add ability to provide sensor command routines

Summary

Add the ability to provide routines of sensor commands to run once or periodically. A routine can provided via a JSON file to the CLI via the new --routine-file option in the start command.

Contents (#12)

IMPORTANT: There is 1 breaking change.

New features

• Allow user to provide a routine of sensor commands to the CLI via a JSON routine file

Enhancements

• BREAKING CHANGE: Separate interactive mode from whether data is saved locally or to the cloud
• Add CLI option to save data locally
• Add CLI option to not upload data to the cloud (i.e. uploading to the cloud is the default)
• Add Routine class for scheduling commands once or periodically
• Warn if serial port buffer size cannot be set due to not being on Windows

Fixes

• Ensure commands file is always written to disk locally

Refactoring

• Always run the packet reader in a separate thread so that commands can be sent to the serial port either in interactive mode or by a routine
• Factor out logic in CLI start command into functions
• Remove unused --stop-when-no-more-data option from CLI

Testing

• Use --use-dummy-serial-port CLI option instead of mocking serial port in CLI tests

Quality Checklist

• New features are fully tested (No matter how much Coverage Karma you have)
• [v0.2 onward] New features are included in the documentation

Open source data-gateway under BSD license

Summary

Add BSD license so the repository can be made open-source.

Contents (#17)

Operations

• Increment version for new license release

Other

• Add the BSD-4-clause license

Fix various bugs and improve variable naming

Summary

Various fixes uncovered during testing.

Contents (#10)

Enhancements

• Update periods on IMU sensors:
  Resolves false positive packet loss when IMU period drifts
• Reset previous time-stamp if in sleep mode:
  Resolves false positive packet loss on wake up
• Set warning for buffer overflow, close and reopen serial port:
  Resolves time-traveling packets from the future when the input buffer is in overflow.
  This partially addresses #6

Fixes

• Correct sleep state boolean
• Set positive loss for delayed packets

Operations

• Version bump

Refactoring

• Rename sensor_type to packet_type or sensor_name
• Define sensor names in configuration

Testing

• Update valid configuration

Other

• Improve descriptions
• Add comments

Use correct column name when checking configuration uniqueness

Contents (#11)

Fixes

• Use correct column name when checking configuration uniqueness