Support multiple nodes

Summary

Upgrade data-gateway to handle data from multiple nodes instead of one. This pull request pulls together the changes from the following pull requests - view each one for their individual release notes:

• Update Configuration class to support multiple nodes #70
• Convert packet reader to allow multiple nodes #71
• Fix bugs, simplify/fix dependencies, and improve deployment process #60
• Improve Raspberry Pi installation process #72
• Manage Raspberry Pi deployment via Balena #81

Breaking changes and changes separate from these PRs are listed below.

Contents (#69)

IMPORTANT: There are 2 breaking changes.

New features

• 💥 BREAKING CHANGE: Update configuration for multiple nodes (see #64)

Operations

• Prevent serialisation error in window handler debugging statement
• Make workflows apply correct memory
• Remove old cloudbuild.yaml file

Refactoring

• 💥 BREAKING CHANGE: Remove incorrectly parameterised default sensor name

---

Upgrade instructions

💥 Remove incorrectly parameterised default sensor name

MICROPHONE_SENSOR_NAME is no longer exported

This was removed because it was a misnomer (actually should have been called DEFAULT_MICROPHONE_SENSOR_NAME).

It's very unlikely this will cause any onward breaking changes, but if using that variable externally, the user should instead access data_gateway.configuration.DEFAULT_SENSOR_NAMES[0] which is the same thing.

💥 Update configuration for multiple nodes per #64

Configuration values nested and separated by concerns.

• Users should consult docs for the new configuration format. Older configuration.json files will no longer work so need to be updated.

• Periods are now calculated rather than being configurable, to eliminate duplication. Users should
  configure the relevant frequency parameters for the sensors.

• The session_data key was converted to session

• The ambiguous hardware_version key was removed and replaced with gateway.receiver_firmware_versionandnodes[node_id].node_firmware_version`

• The packet_key is no longer present, instead use Configuration().get_packet_key(node_id)

Improve documentation

Summary

Update and improve the documentation.

Contents (#58)

Documentation

• Hide table of contents on index page
• Update documentation
• Format code blocks properly
• Fix headings in cloud functions doc
• Remove preprocessing doc
• Remove API doc

Fix documentation and improve CLI load speed

Summary

Fix documentation so it can be built on Read the Docs and lazily import packages in the CLI to improve speed and clean up its help messages.

Contents (#56)

Enhancements

• Import packages lazily in CLI for faster loading and cleaner help messages

Fixes

• Ensure log handler is applied in parser process

Operations

• Add @cortadocodes as author

Dependencies

• Use octue==0.27.0
• Update docs dependencies

Other

• Correct spelling/casing of Aerosense
• Add documentation link to README
• Remove GitHub pages config file

Update README and installation documentation to reflect new code host and package manager

Summary

Contents (#51)

Other

• Update README and installation documentation with the new code host (GitHub) and package manager poetry

Fix microphone data upload

Summary

Fix upload of microphone data to cloud storage, update code using deprecated features of octue, and upgrade the package manager to give deterministic dependency resolution.

Contents (#50)

Enhancements

• Remove unused project name parameters
• Stop using deprecated method of providing cloud paths
• Log upload of microphone data to cloud storage

Fixes

• Fix window cloud path in WindowHandler
• Write microphone data to HDF5 file as a numpy array

Operations

• Switch package management to poetry
• Use workload identity federation to authenticate with Google Cloud in CI

Dependencies

• Upgrade octue==0.16.0

Testing

• Test microphone data upload
• Add microphone data to deployment test
• Stop test fixture being mutated

Update handles in parser process instead of reader process

Contents (#49)

Fixes

• Update handles in the Parser process instead of the Reader process
• Closes #42

Only persist configuration file locally

Contents (#47)

Enhancements

• Only save configuration file locally, not to the cloud (there's no need as the configuration is saved as metadata on each file uploaded to the cloud)

Ensure sensor commands always end in a newline

Contents (#44)

Enhancements

• Make parsing of non-sensor sleep command more robust

Fixes

• Ensure sensor commands always end in a newline

Send sensor stop commands on gateway exit

Contents (#39)

Note: the coverage calculation is again wrong on this PR (I'll be looking into this with CodeCov)

Enhancements

• Store available sensor commands in Configuration
• Send sensor stop commands on exit of DataGateway by default
• Add CLI option to not stop sensors on gateway exit
• Clarify gateway stop message

Fixes

• Apply log handler to multiprocessing logger in DataGateway instantiation instead of at data_gateway module import time to avoid unnecessary logs when using gateway start CLI command

Remove preprocessing from the cloud function

Summary

Remove preprocessing from gateway cloud function. This means that there is no more:

• Attempt to convert the window samples to a constant time-step
• Searching for missing data and NAN padding
• Conversion of fixed point numbers to physical variables

All of these will be done separately on the stored raw data (in order not to mess with ingress pipeline) each time new a sampling procedure is adopted or a different calibration is done.

Contents (#41)

Enhancements

• Remove pre-processing from ingress-eu cloud function
• Convert timestamps with convert_window_timestamps_to_unix_time method instead of preprocessor

Operations

• Give more time for cloud function builds to complete