diff --git a/README.md b/README.md index 79057611..38b012b2 100644 --- a/README.md +++ b/README.md @@ -2,22 +2,25 @@ [![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit) [![black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black) [![python-template](https://img.shields.io/badge/template-python--library-blue)](https://github.com/thclark/python-library-template) - +[![docs](https://readthedocs.org/projects/aerosense-data-gateway/badge/?version=latest)](https://aerosense-data-gateway.readthedocs.io/en/latest/) # data-gateway -Read the docs [here.](https://aerosense-data-gateway.readthedocs.io/en/latest/) +Usage: Read the documentation [here.](https://aerosense-data-gateway.readthedocs.io/en/latest/) -*Note that the test coverage figure is more like 90% - the recent addition of multiprocessing has made it difficult to -measure the true coverage across multiple processes.* +_Note that the test coverage figure is more like 90% - the recent addition of multiprocessing has made it difficult to +measure the true coverage across multiple processes._ ## Installation and usage + To install, run: + ```shell pip install git+https://github.com/aerosense-ai/data-gateway.git ``` The command line interface (CLI) can then be accessed via: + ```shell gateway --help ``` @@ -50,19 +53,25 @@ Commands: ### Installation #### Poetry + We're using `poetry` instead of `pip` to manage the package to take advantage of the `poetry.lock` file [among other useful features](https://python-poetry.org/). In terms of developer experience, this just means there are some slightly different commands to run than usual. `data-gateway` can still be `pip`-installed by anyone anywhere, but dependency resolution and dependency specification for `data-gateway` developers is improved by using `poetry` locally. #### Architecture-specific installations + Due to some (most likely temporary) constraints with `poetry` and the need to run and develop the gateway on Linux, Windows, M1 Macs, and Raspberry Pis, the need has arisen for some slightly different installation procedures on these -different architectures/platforms. Instructions are detailed below - [click here](https://github.com/aerosense-ai/data-gateway/issues/65) +different architectures/platforms. [click here](https://github.com/aerosense-ai/data-gateway/issues/65) to read more. +For Raspberry Pis, overwhelmingly the simple thing to do is use balena instead ([see the docs](https://aerosense-data-gateway.readthedocs.io/en/latest/)). + #### Clone the repository + First, clone the repository and `cd` into it: + ```shell git clone https://github.com/aerosense-ai/data-gateway.git cd data-gateway @@ -70,8 +79,10 @@ cd data-gateway Then follow the instructions for your platform below. -#### Install on MacOS and Linux (except on Raspberry Pi) +#### Install on MacOS and Linux + Run the following from the repository root: + ```shell pip install poetry @@ -80,25 +91,31 @@ poetry install ``` This will editably install `data-gateway` in a `poetry`-managed virtual environment, meaning: + - Any local changes you make to it will be automatically used when running it locally - It won't be affected by changes to other python packages you have installed on your system, making development much easier and more deterministic #### Install on Raspberry Pi -Run the following from the repository root: + +It's best to use [balena](https://aerosense-data-gateway.readthedocs.io/en/latest/) but if you do need to run the install, do the following from the repository root: + ```shell pip install -r requirements-pi-dev.txt ``` #### Install on Windows -This workflow works for Windows using Powershell. + +This workflow works for Windows using Powershell, if you're the most masochistic of people. Prerequisites: + 1. Make sure to have python not installed from [python.org](https://www.python.org/) 2. Install [pyenv-win](https://github.com/pyenv-win/pyenv-win) via pip method -3. Execute ```pip install virtualenv``` +3. Execute `pip install virtualenv` Installation: + ```shell pyenv install 3.8.0 # (or higher) pyenv local 3.8.0 @@ -110,25 +127,32 @@ poetry install ``` Every time you enter the repo over powershell again, make sure to activate the venv using + ```shell ./venv/Scripts/activate ``` #### Troubleshooting + If there are problems reading the serial port, try running this (or the equivalent on non-Linux platforms) and retrying: + ```shell sudo apt-get update sudo apt-get install libhdf5-dev libhdf5-serial-dev ``` ### Testing + These environment variables need to be set to run the tests: -* `GOOGLE_APPLICATION_CREDENTIALS=/absolute/path/to/service/account/file.json` + +- `GOOGLE_APPLICATION_CREDENTIALS=/absolute/path/to/service/account/file.json` Then, from the repository root, run + ```bash tox ``` ## Contributing + Take a look at our [contributing](/docs/contributing.md) page. diff --git a/docs/source/cloud_functions.rst b/docs/source/cloud/cloud_functions.rst similarity index 100% rename from docs/source/cloud_functions.rst rename to docs/source/cloud/cloud_functions.rst diff --git a/docs/source/deployment.rst b/docs/source/cloud/creating_service_accounts.rst similarity index 56% rename from docs/source/deployment.rst rename to docs/source/cloud/creating_service_accounts.rst index fd866c4b..d6a6dc7b 100644 --- a/docs/source/deployment.rst +++ b/docs/source/cloud/creating_service_accounts.rst @@ -1,51 +1,53 @@ -.. _deployment: -========== -Deployment -========== +.. _creating_service_accounts: -To deploy the Aerosense system, you must: +========================= +Creating Service Accounts +========================= -- Register the deployment -- Create a deployment Service Account -- Configure and run the gateway (at the on-nacelle processor) +The gateway that you install on a turbine needs to upload data to Aerosense Cloud. However, we don't want "just anybody" +to be able to write data to the store - that leaves us vulnerable to a wide range of attacks. So the gateway must +authenticate itself with the store prior to upload. +To enable the gateway to authenticate itself, we use a **service account**, which is a bit like a user account (it has +an email address and can be given certain permissions) but for a non-human. -.. _register_the_deployment: +Here, we will create a service account for a deployment - this will result in a single credentials file that we can +reuse across the gateways (turbines) in the deployment to save administrative overhead maintaining all the credentials. -Register the deployment -======================= +Choose your service account name +================================ -A "deployment" is the act of commissioning the Aerosense system on a group of turbines (typically the turbines in a -particular wind farm). +When creating service accounts, whilst any name will work, sticking to a naming convention is very helpful. +There are three kinds of service account names: -For experimental and test purposes, there will typically be only one turbine (and therefore one gateway) per deployment. +**For deployment with Balena** -TODO Registration process documentation - See Tom and Yuriy +If you've deployed gateway code using Balena, the newly added device will be given a "coolname" like ``fried-firefly`` or ``holy-lake``. +Use this name as your service account name, which makes it super easy to diagnose any problems, or restrict permissions +if a device is lost. +**For manual deployment** -.. _create_a_service_account: +Use a name that's prefixed with ``as-deployment-``, eg ``as-deployment-tommasso``. These service accounts should be set up +to have consistent permissions as with a balena deployment but simply indicate its for manual trialling of uploads from the gateway. -Create a Service Account -======================== +**For development** -The gateway that you install on a turbine needs to upload data to Aerosense Cloud. However, we don't want "just anybody" -to be able to write data to the store - that leaves us vulnerable to a wide range of attacks. So the gateway must -authenticate itself with the store prior to upload. +Developers will work across the entire stack of cloud functions, gateway and other aspects of the project like dashboard or tools. +Thus developers' service accounts are expected to have a wide and varied range of permissions. +The name should be in the form ``developer-``, eg ``developer-thclark``. -To enable the gateway to authenticate itself, we use a **service account**, which is a bit like a user account (it has -an email address and can be given certain permissions) but for a non-human. -Here, we will create a service account for a deployment - this will result in a single credentials file that we can -reuse across the gateways (turbines) in the deployment to save administrative overhead maintaining all the credentials. +Create account and credentials +============================== Log in to the ``aerosense-twined`` project on Google Cloud Platform (GCP) and work through the following steps: **1. Go to IAM > Service Accounts > Create** - -.. figure:: images/creating-a-service-account/1-go-to-iam-service-accounts.png +.. figure:: /images/creating-a-service-account/1-go-to-iam-service-accounts.png :width: 600px :align: center :figclass: align-center @@ -55,17 +57,17 @@ Log in to the ``aerosense-twined`` project on Google Cloud Platform (GCP) and wo **2. Create the service account** -.. figure:: images/creating-a-service-account/2-create-service-account.png +.. figure:: /images/creating-a-service-account/2-create-service-account.png :width: 600px :align: center :figclass: align-center - The service account name should contain your deployment id (from above) in the pattern - ``as-deployment-``. In this case, ``deploymentId = gatewaytest`` + The service account name should be per the naming guidelines above. + In this image, ``as-deployment-gatewaytest`` was used. **3. Skip assignation of optional roles and users (for now)** -.. figure:: images/creating-a-service-account/3-no-grants-or-users.png +.. figure:: /images/creating-a-service-account/3-no-grants-or-users.png :width: 600px :align: center :figclass: align-center @@ -74,21 +76,21 @@ Log in to the ``aerosense-twined`` project on Google Cloud Platform (GCP) and wo **4. Create and download a private JSON key for this Service Account** -.. figure:: images/creating-a-service-account/4a-create-key.png +.. figure:: /images/creating-a-service-account/4a-create-key.png :width: 600px :align: center :figclass: align-center Find your newly created service account in the list (you may have to search) and click 'Create Key'. -.. figure:: images/creating-a-service-account/4b-key-should-be-json.png +.. figure:: /images/creating-a-service-account/4b-key-should-be-json.png :width: 600px :align: center :figclass: align-center Choose the default JSON key type. -.. figure:: images/creating-a-service-account/4c-key-will-be-saved.png +.. figure:: /images/creating-a-service-account/4c-key-will-be-saved.png :width: 600px :align: center :figclass: align-center @@ -97,7 +99,7 @@ Log in to the ``aerosense-twined`` project on Google Cloud Platform (GCP) and wo **5. Locate the ingress bucket in the storage browser, and click on "Add Member"** -.. figure:: images/creating-a-service-account/5-locate-aerosense-ingress-bucket.png +.. figure:: /images/creating-a-service-account/5-locate-aerosense-ingress-bucket.png :width: 600px :align: center :figclass: align-center @@ -107,7 +109,7 @@ Log in to the ``aerosense-twined`` project on Google Cloud Platform (GCP) and wo **6. Assign ``Storage Object Creator`` permission** -.. figure:: images/creating-a-service-account/5-locate-aerosense-ingress-bucket.png +.. figure:: /images/creating-a-service-account/6-add-storage-object-creator.png :width: 600px :align: center :figclass: align-center @@ -120,15 +122,8 @@ And you're done! Keep that downloaded permission file for later. .. ATTENTION:: - Do not add to a docker image, email, skype/meet/zoom, dropbox, whatsapp, commit to git, post in an issue, or - whatever, this private credentials file. + Do not add this private credentials file to a docker image, email, skype/meet/zoom, dropbox, whatsapp, git commit, + post in an issue, or anywhere else. Doing so will earn you the penance of flushing and rotating all the system credentials. - -.. _configure_and_run_the_gateway: - -Configure and run the Gateway -============================= - -See :doc:`using the gateway. ` diff --git a/docs/source/cloud/index.rst b/docs/source/cloud/index.rst new file mode 100644 index 00000000..9f292b60 --- /dev/null +++ b/docs/source/cloud/index.rst @@ -0,0 +1,17 @@ +.. _cloud: + +===== +Cloud +===== + +The majority of aspects of the aerosense cloud are documented in other +repositories (eg aerosense-tools, aerosense-dashboard). However, on the +gateway / ingress side it's worth collecting some notes here. + +.. toctree:: + :caption: Cloud Contents + :maxdepth: 1 + + cloud_functions + creating_service_accounts + uploads \ No newline at end of file diff --git a/docs/source/cloud/uploads.rst b/docs/source/cloud/uploads.rst new file mode 100644 index 00000000..4b530b2c --- /dev/null +++ b/docs/source/cloud/uploads.rst @@ -0,0 +1,11 @@ +.. _uploads: + +======= +Uploads +======= + +Once running ``gateway start``, windows are uploaded to the cloud ingress bucket (unless the ``--no-upload-to-cloud`` option is used). + +If the connection to Google Cloud fails, windows will be written to the hidden directory +``.//.backup`` where they will stay until the connection resumes. +Backup files are deleted upon successful cloud upload. diff --git a/docs/source/deployment/check.rst b/docs/source/deployment/check.rst new file mode 100644 index 00000000..e924e3c1 --- /dev/null +++ b/docs/source/deployment/check.rst @@ -0,0 +1,22 @@ +.. _check: + +===== +Check +===== + +Once the above steps are complete, in the balena (or your own, for a manual installation) terminal, check the installation by typing: + +.. code-block:: shell + + gateway --help + +.. figure:: /images/balena/gateway-help.png + :width: 600px + :align: center + :figclass: align-center + + If the gateway is correctly installed, you should see this. + + + + diff --git a/docs/source/deployment/configuration.rst b/docs/source/deployment/configuration.rst new file mode 100644 index 00000000..26595e4f --- /dev/null +++ b/docs/source/deployment/configuration.rst @@ -0,0 +1,120 @@ +.. _configuration: + +============= +Configuration +============= + +.. note:: + + The majority of users will use Balena, so we'll give examples in balena here. + Configuring in a manually installed environment is broadly the same. + + +Add a service account +===================== + +Using the name generated by balena when the device was added, create +a dedicated service account by following :ref:`these instructions `. + +Once you have the ``json`` file containing credentials on your computer, +select your new device from the fleet and open a terminal (on ``main``, not on the host OS) and do: + +.. code-block:: shell + + nano $GOOGLE_APPLICATION_CREDENTIALS + +Then paste the contents of the credentials file. Press ``ctrl-x`` then ``y`` to save and exit. +This file will persist over reboots of the device; you shouldn't need to touch it again. + +.. figure:: /images/balena/opening-a-terminal.png + :width: 600px + :align: center + :figclass: align-center + + Open a terminal to the 'main' target and you can shell into the machine directly. + +.. _add_routine_and_configuration_files: + +Add routine and configuration files +=================================== + +We'll start with an empty routine file, which you can change later (see :ref:`routine_files`), and a basic configuration file. + +.. code-block:: shell + + mkdir -p /data/routines && echo "[]" > /data/routines/my-routine.json + mkdir -p /data/configurations && nano /data/configurations/my-configuration.json + +Then paste in the following JSON (a basic configuration), update the values and save it: + +.. code-block:: json + + { + "gateway": { + "installation_reference": "my-installation-reference", // change this to a meaningful value, eg "aventa-initial-deployment" + "latitude": 0, + "longitude": 0, + "receiver_firmware_version": "unknown" + }, + "nodes": { + "1": {}, + "2": {}, + "3": {}, + "4": {}, + "5": {} // Remove nodes if you know you don't need them + }, + "measurement_campaign": { + "label": "test-campaign", + "description": "This field can be used to label and describe different measurement campaigns" + } + } + +.. warning:: + + You can store as many different configurations and routines as you want, but you should always save + them somewhere in the ``/data/`` folder, because it persists over restarts. + + +Set device variables +==================== + +When you run the gateway, you'll be able to able to specify a routine file and a configuration file. +However, that makes for a lot of typing, especially when you're trying to debug things. + +To ease frustration, the best thing to do is to set these values as environment variables. +Go to the "Device Variables" tab and add two variables, ``GATEWAY_ROUTINE_FILE`` and ``GATEWAY_CONFIG_FILE``, +whose values match the paths you set for the two files :ref:`above `. + +.. figure:: /images/balena/device-variables.png + :width: 600px + :align: center + :figclass: align-center + + Set device variables to change the default ``GATEWAY_ROUTINE_FILE`` and ``GATEWAY_CONFIG_FILE``. + + +Check the installation +====================== + +In the balena (or your own, for a manual installation) terminal, check by typing: + +.. code-block:: shell + + gateway --help + +.. figure:: /images/balena/gateway-help.png + :width: 600px + :align: center + :figclass: align-center + + If the gateway is correctly installed, you should see this. + +.. tip:: + + You can always use the ``$GATEWAY_CONFIG_FILE`` or ``$GATEWAY_ROUTINE_FILE`` to see the paths and their contents: + + .. code-block:: shell + + echo The config file is at $GATEWAY_CONFIG_FILE and it contains... + cat $GATEWAY_CONFIG_FILE + diff --git a/docs/source/deployment/index.rst b/docs/source/deployment/index.rst new file mode 100644 index 00000000..e42f2c82 --- /dev/null +++ b/docs/source/deployment/index.rst @@ -0,0 +1,34 @@ +.. _deployment: + +========== +Deployment +========== + +To deploy the Aerosense system, you must: + +- Install gateway code to the raspberry pi in the base station +- Install and connect the physical base station +- Configure the gateway +- Register this installation to the aerosense database + +Go through the following steps: + +1. :ref:`Install using balena ` (strongly recommended) or :ref:`manually `. + +2. :ref:`Configure ` the gateway for the first time. + +3. :ref:`Register ` your installation. + +4. :ref:`Check ` the gateway. + +Once complete, move on to :ref:`using the gateway `. + +.. toctree:: + :caption: Deployment Contents + :maxdepth: 2 + + using_balena + manual_installation + configuration + registration + check diff --git a/docs/source/installation.rst b/docs/source/deployment/manual_installation.rst similarity index 64% rename from docs/source/installation.rst rename to docs/source/deployment/manual_installation.rst index 2a475fba..f814b879 100644 --- a/docs/source/installation.rst +++ b/docs/source/deployment/manual_installation.rst @@ -1,16 +1,26 @@ -.. _installation: +.. _manual_installation: -============ -Installation -============ +=================== +Manual Installation +=================== + +.. note:: + Once installed, you'll need to configure with a service account; follow the :ref:`setup steps ` using your own terminal and system environment variables. + +.. warning:: + + It's possible to manually install gateway code to any machine (eg your own laptop) + but **by far the easiest way**, even for development purposes in the lab, is to use balena to deploy the + code straight to a raspberry pi. + .. _installing_on_a_raspberry_pi: -Installing on a Raspberry Pi 4 -============================== +Installing manually (on a Raspberry Pi 4) +========================================= Although **data-gateway** can be run on a wide range of hardware, it's generally aimed at being run on a Raspberry Pi -on board a turbine nacelle. +on board a turbine (in the base station box on the tower). It's anticipated that you're using: - Raspberry Pi 4 @@ -23,6 +33,7 @@ You'll need to install Raspberry Pi OS (formerly "Raspbian", which was a much be When booted into your **pi**, use the following commands to install... .. code-block:: shell + sudo apt-get update sudo apt-get install libhdf5-dev libhdf5-serial-dev @@ -38,8 +49,9 @@ This installs the CLI :ref:`gateway_cli`, which enables you to start the gateway Installing on Other Hardware ============================ -There's no reason **data-gateway** can't be run on a wide range of hardware, although we've only tested it for the -Raspberry Pi 4, which has a quad-core processor. +There's no reason **data-gateway** can't be run on a wide range of hardware, or your own development laptop in the lab. + +However, we've only tested it for the Raspberry Pi 4, which has a quad-core processor and is unix-based. The main consideration when choosing other hardware is that a dual-core CPU is probably a conservative choice: **data-gateway** uses three processes and multiple threads. Additional vCPUs will always reduce the likelihood of the @@ -53,5 +65,5 @@ tests prior to field deployment if you go down this route! Installation for developers =========================== -If you're developing **data-gateway** you'll need to follow the instructions for developers in the +If you're developing **data-gateway** code itself you'll need to follow the instructions for developers in the `repo's README.md file `_. diff --git a/docs/source/deployment/registration.rst b/docs/source/deployment/registration.rst new file mode 100644 index 00000000..4dee030c --- /dev/null +++ b/docs/source/deployment/registration.rst @@ -0,0 +1,31 @@ +.. _registration: + +Registration +============ + +When installing the physical Aerosense system onto one or more turbines (typically the turbines in a +particular wind farm), you need to register the installation in the aerosense database. + +For experimental and test purposes, there will generally be only one turbine (and therefore one gateway) per deployment. + +Either way, once you've :ref:`installed ` and :ref:`configured ` the gateway, +you need to register the installation. + +.. tip:: + + Make sure you've chosen a sensible value for the ``installation_reference`` value in your + configuration file. This should be unique to your installation (an error will occur in registration + if it's been used before), and this will be what you use to refer to later when you want to filter results + in ``dashboard`` and ``aerosense-tools``. + +To register, do: + +.. code-block:: shell + + gateway create-installation --config-file $GATEWAY_CONFIG_FILE + +And follow the instructions. After this, the gateway should be ready for use. + + + + diff --git a/docs/source/deployment/using_balena.rst b/docs/source/deployment/using_balena.rst new file mode 100644 index 00000000..ce415b4d --- /dev/null +++ b/docs/source/deployment/using_balena.rst @@ -0,0 +1,67 @@ +.. _using_balena: + +============ +Using Balena +============ + +A fleet of devices (under the ``aerosense`` organisation on BalenaCloud) is managed by Balena, +and this is by far the preferred way of doing things. + +Balena manages device installation, health and continuous deployment of code +(whenever the main repository is updated, a fleet-wide update is triggered). +It also works as a portal to the device, allowing you to log in and view device status as well +as opening a terminal shell to individual devices for test and diagnostic purposes. + +You can :ref:`join the balena organisation with your github account `. + +.. _installation_with_balena: + +Installation with balena +======================== + +With a fresh new raspberry pi (or an old one with a wiped SSD card!) you'll want to install +the gateway code. You do this by Adding a Device. + +.. tip:: + Technically, the "device" is tied to the SSD card, not the raspberry pi itself. + SSDs are notorious for failing after a short cycle time, so it's well worth buying high quality SanDisk + SSD cards rather than the cheap equivalents, or you may find yourself dealing with a failure in the field. + + +Add a device +------------ + +Follow the balenaCloud instructions to install balena on the SSD card and add it to the ``gateways`` fleet (in the ``aerosense`` organization): + +.. figure:: /images/balena/adding-a-device.png + :width: 600px + :align: center + :figclass: align-center + + Hit 'add a device' and balena will take you through the process of installing and deploying. + +Once added, the device will appear in your balena dashboard with a coolname, like ``fried-firefly`` or +``holy-lake``. + +Labelmaking is the way +---------------------- + +Steal the labelmaker from PBL, and label your raspberry pi so you know which one is which. + +Once added, you can follow the instructions to :ref:`configure ` your device. + + +.. _balena_organization_admins: + +Balena Organization Admins +========================== + +Use your github account to register with balena. Existing admins for the aerosense organisation +can log into BalenaCloud then invite you. + +.. figure:: /images/balena/current-admins.png + :width: 600px + :align: center + :figclass: align-center + + The current list of admins in balena. diff --git a/docs/source/images/balena/adding-a-device.png b/docs/source/images/balena/adding-a-device.png new file mode 100644 index 00000000..0c336262 Binary files /dev/null and b/docs/source/images/balena/adding-a-device.png differ diff --git a/docs/source/images/balena/current-admins.png b/docs/source/images/balena/current-admins.png new file mode 100644 index 00000000..ddd1df68 Binary files /dev/null and b/docs/source/images/balena/current-admins.png differ diff --git a/docs/source/images/balena/device-variables.png b/docs/source/images/balena/device-variables.png new file mode 100644 index 00000000..63c6a48f Binary files /dev/null and b/docs/source/images/balena/device-variables.png differ diff --git a/docs/source/images/balena/fleets.png b/docs/source/images/balena/fleets.png new file mode 100644 index 00000000..f583bc2f Binary files /dev/null and b/docs/source/images/balena/fleets.png differ diff --git a/docs/source/images/balena/gateway-help.png b/docs/source/images/balena/gateway-help.png new file mode 100644 index 00000000..ac6e0a3f Binary files /dev/null and b/docs/source/images/balena/gateway-help.png differ diff --git a/docs/source/images/balena/gateways-fleet.png b/docs/source/images/balena/gateways-fleet.png new file mode 100644 index 00000000..966f32ea Binary files /dev/null and b/docs/source/images/balena/gateways-fleet.png differ diff --git a/docs/source/images/balena/opening-a-terminal.png b/docs/source/images/balena/opening-a-terminal.png new file mode 100644 index 00000000..3cf1cd73 Binary files /dev/null and b/docs/source/images/balena/opening-a-terminal.png differ diff --git a/docs/source/images/database/installations-table-schema.png b/docs/source/images/database/installations-table-schema.png new file mode 100644 index 00000000..4220727d Binary files /dev/null and b/docs/source/images/database/installations-table-schema.png differ diff --git a/docs/source/index.rst b/docs/source/index.rst index aae230ff..90f34f4f 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,6 +1,3 @@ -.. ATTENTION:: - This library is in experimental stages! Please pin deployments to a specific release, and consider every release as breaking. - ============ Data Gateway ============ @@ -8,6 +5,12 @@ Data Gateway .. epigraph:: *"Data Gateway" ~ reads data from an Aerosense receiver and shoves it into the cloud.* +Get Started Quick +================= + +- :ref:`Deploy a gateway `. + +- :ref:`Run an already-deployed gateway `. Data Flow ========= @@ -17,8 +20,8 @@ The data flow from the aerosense sensor modules looks like this: .. code-block:: Node (edge processor on-blade) - -> Receiver (bluetooth equipment in-nacelle) - ---> Gateway (data manager and uploader on-nacelle) + -> Base Station (bluetooth equipment on-tower) + ---> Gateway (data manager and uploader on-tower) -----> Ingress (Cloud Function to receive data on-cloud) -------> Google Cloud BigQuery + Google Cloud Store (database / object storage system) |----> Digital Twin (data analysis system) @@ -26,7 +29,7 @@ The data flow from the aerosense sensor modules looks like this: |----> Dashboard (data visualisation for researchers and system installers) -A ``Node`` streams data to the ``Receiver`` via bluetooth. The ``Receiver`` writes the bytestream directly to a serial +A ``Node`` streams data to the ``Base Station`` via bluetooth. The ``Base Station`` writes the bytestream directly to a serial port. The ``Gateway`` (this library) reads the bytestream from the serial port, decodes it and buffers it in local storage. The ``Gateway`` then is responsible for: @@ -38,17 +41,12 @@ internet outages. The code for the Cloud Function ``Ingress`` is also included in this repository. - .. toctree:: + :caption: Table of Contents :maxdepth: 2 - :hidden: - - installation - using_the_gateway - output_data - routines - deployment - cloud_functions + + deployment/index + usage/index + cloud/index hardware_and_firmware_versions - information_for_developers version_history diff --git a/docs/source/output_data.rst b/docs/source/output_data.rst deleted file mode 100644 index 2681672d..00000000 --- a/docs/source/output_data.rst +++ /dev/null @@ -1,24 +0,0 @@ -.. _output_data: - -=========== -Output data -=========== - -Here is some example output data from the data gateway when reading random data from the serial port (random apart from -the first three bytes, which represent the packet key, sensor type, and length): - -.. code-block:: json - - { - "sensor_time_offset": 1624381890.662965, - "sensor_data": { - "Mics": [[46999.553926635745, 30722, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55412663574, 26251, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55432663574, -2675, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.554526635744, -22745, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55472663574, -24821, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55492663574, -12457, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55512663574, 11929, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.555326635746, -23664, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55552663574, 28160, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55572663574, -5574, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.555926635745, 20057, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55612663574, 23103, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55632663574, 3867, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.556526635744, 4562, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55672663574, 77, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55692663574, -16036, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.557126635744, 5807, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55732663574, -1491, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55752663574, 31852, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55772663574, 13585, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.557926635745, 16099, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55812663574, -2227, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55832663574, 28940, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.558526635745, -7450, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55872663574, -19446, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55892663574, -16620, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.559126635744, 14178, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55932663574, -26675, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55952663574, -10641, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.55972663574, 17836, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.559926635746, 3202, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56012663574, -25480, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56032663574, 27970, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.560526635745, 29664, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56072663574, -26117, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56092663574, -5154, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.561126635745, 6296, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56132663574, 17782, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56152663574, -25610, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.561726635744, -14316, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56192663574, -14633, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56212663574, -24990, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56232663574, -6835, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.562526635746, -9433, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56272663574, 26196, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56292663574, 11844, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.563126635745, -12533, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56332663574, -9070, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56352663574, 925, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.563726635744, 3546, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56392663574, -31505, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56412663574, -18456, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.564326635744, -9315, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56452663574, 20906, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56472663574, 25479, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56492663574, -13736, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.565126635745, -32552, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56532663574, -31562, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56552663574, -15731, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.565726635745, -11667, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56592663574, -30121, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56612663574, -18400, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.566326635744, -30323, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56652663574, 21741, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56672663574, 27578, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56692663574, 10594, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.567126635746, 9208, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56732663574, -15228, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56752663574, -18521, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.567726635745, -4332, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56792663574, -13432, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56812663574, 11392, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.568326635745, 5296, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56852663574, -14029, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56872663574, 15607, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.568926635744, -19025, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56912663574, -28329, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56932663574, 4969, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56952663574, -5504, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.569726635746, -25426, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56992663574, -32083, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57012663574, 2801, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.570326635745, -7499, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57052663574, -17006, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57072663574, -22466, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.570926635744, -28542, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57112663574, -31731, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57132663574, 16235, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.571526635744, -13985, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57172663574, 30893, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57192663574, 13793, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57212663574, 9411, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.572326635745, -19779, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57252663574, -22400, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57272663574, 24897, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.572926635745, 6780, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57312663574, 23648, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57332663574, -1615, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.573526635744, -9921, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57372663574, 19967, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57392663574, 790, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57412663574, 30743, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.574326635746, 16197, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57452663574, -18157, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57472663574, -5384, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.574926635745, 14537, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57512663574, -28831, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57532663574, -17150, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.575526635745, -28932, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57572663574, 2823, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57592663574, -15995, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.576126635744, -15606, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57632663574, -18784, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57652663574, -12599, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57672663574, 9530, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.576926635746, 25739, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57712663574, 5731, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57732663574, 25488, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.577526635745, 5866, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.57772663574, -17361, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - "Baros_P": [[46999.45792663574, 9140226, 10954741, 10080087, 7209123, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.46792663574, 4148825, 1167887, 11518300, 8154362, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.477926635744, 5062371, 14870129, 6471444, 14053271, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.487926635746, 7867522, 7594093, 10021854, 10221125, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.49792663574, 6473431, 14362597, 732740, 237020, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.50792663574, 15238383, 5352155, 14207576, 12750212, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.517926635745, 2132567, 5565833, 16263522, 12036036, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.52792663574, 8440712, 13185812, 5748143, 15368211, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.53792663574, 15827629, 12423906, 888962, 13197119, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.54792663574, 12793313, 11042994, 6298236, 14237689, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.557926635745, 1508118, 12129087, 6371529, 9370813, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56792663574, 704901, 13552054, 6513803, 1501795, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - "Baros_T": [[46999.45792663574, -29338, -24821, -28626, -5574, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.46792663574, 7002, 77, 11542, 13585, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.477926635744, 3319, -19446, -13001, 17836, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.487926635746, 17052, -26117, 30232, -14316, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.49792663574, 19870, 26196, -27953, 3546, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.50792663574, -25161, 25479, -18816, -11667, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.517926635745, -29256, 27578, -31709, -4332, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.52792663574, -20436, 15607, 27025, -25426, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.53792663574, -19190, -22466, 27524, 30893, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.54792663574, -17116, 24897, -20132, 19967, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.557926635745, 17784, -5384, 655, 2823, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [46999.56792663574, -24381, 9530, -28650, -17361, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], - "Acc": [[33155.02135620117, 30722, 26251, -2675], [33155.03135620117, -22745, -24821, -12457], [33155.041356201174, 11929, -23664, 28160], [33155.05135620117, -5574, 20057, 23103], [33155.06135620117, 3867, 4562, 77], [33155.07135620117, -16036, 5807, -1491], [33155.081356201175, 31852, 13585, 16099], [33155.09135620117, -2227, 28940, -7450], [33155.10135620117, -19446, -16620, 14178], [33155.111356201174, -26675, -10641, 17836], [33155.12135620117, 3202, -25480, 27970], [33155.13135620117, 29664, -26117, -5154], [33155.14135620117, 6296, 17782, -25610], [33155.151356201175, -14316, -14633, -24990], [33155.16135620117, -6835, -9433, 26196], [33155.17135620117, 11844, -12533, -9070], [33155.181356201174, 925, 3546, -31505], [33155.19135620117, -18456, -9315, 20906], [33155.20135620117, 25479, -13736, -32552], [33155.21135620117, -31562, -15731, -11667], [33155.221356201175, -30121, -18400, -30323], [33155.23135620117, 21741, 27578, 10594], [33155.24135620117, 9208, -15228, -18521], [33155.25135620117, -4332, -13432, 11392], [33155.26135620117, 5296, -14029, 15607], [33155.27135620117, -19025, -28329, 4969], [33155.28135620117, -5504, -25426, -32083], [33155.291356201174, 2801, -7499, -17006], [33155.30135620117, -22466, -28542, -31731], [33155.31135620117, 16235, -13985, 30893], [33155.32135620117, 13793, 9411, -19779], [33155.331356201175, -22400, 24897, 6780], [33155.34135620117, 23648, -1615, -9921], [33155.35135620117, 19967, 790, 30743], [33155.361356201174, 16197, -18157, -5384], [33155.37135620117, 14537, -28831, -17150], [33155.38135620117, -28932, 2823, -15995], [33155.39135620117, -15606, -18784, -12599], [33155.401356201175, 9530, 25739, 5731], [33155.41135620117, 25488, 5866, -17361]], - "Gyro": [[33155.02135620117, 30722, 26251, -2675], [33155.03135620117, -22745, -24821, -12457], [33155.041356201174, 11929, -23664, 28160], [33155.05135620117, -5574, 20057, 23103], [33155.06135620117, 3867, 4562, 77], [33155.07135620117, -16036, 5807, -1491], [33155.081356201175, 31852, 13585, 16099], [33155.09135620117, -2227, 28940, -7450], [33155.10135620117, -19446, -16620, 14178], [33155.111356201174, -26675, -10641, 17836], [33155.12135620117, 3202, -25480, 27970], [33155.13135620117, 29664, -26117, -5154], [33155.14135620117, 6296, 17782, -25610], [33155.151356201175, -14316, -14633, -24990], [33155.16135620117, -6835, -9433, 26196], [33155.17135620117, 11844, -12533, -9070], [33155.181356201174, 925, 3546, -31505], [33155.19135620117, -18456, -9315, 20906], [33155.20135620117, 25479, -13736, -32552], [33155.21135620117, -31562, -15731, -11667], [33155.221356201175, -30121, -18400, -30323], [33155.23135620117, 21741, 27578, 10594], [33155.24135620117, 9208, -15228, -18521], [33155.25135620117, -4332, -13432, 11392], [33155.26135620117, 5296, -14029, 15607], [33155.27135620117, -19025, -28329, 4969], [33155.28135620117, -5504, -25426, -32083], [33155.291356201174, 2801, -7499, -17006], [33155.30135620117, -22466, -28542, -31731], [33155.31135620117, 16235, -13985, 30893], [33155.32135620117, 13793, 9411, -19779], [33155.331356201175, -22400, 24897, 6780], [33155.34135620117, 23648, -1615, -9921], [33155.35135620117, 19967, 790, 30743], [33155.361356201174, 16197, -18157, -5384], [33155.37135620117, 14537, -28831, -17150], [33155.38135620117, -28932, 2823, -15995], [33155.39135620117, -15606, -18784, -12599], [33155.401356201175, 9530, 25739, 5731], [33155.41135620117, 25488, 5866, -17361]], - "Mag": [[33152.221356201175, 30722, 26251, -2675], [33152.30135620117, -22745, -24821, -12457], [33152.38135620117, 11929, -23664, 28160], [33152.46135620117, -5574, 20057, 23103], [33152.541356201174, 3867, 4562, 77], [33152.62135620117, -16036, 5807, -1491], [33152.70135620117, 31852, 13585, 16099], [33152.78135620117, -2227, 28940, -7450], [33152.861356201174, -19446, -16620, 14178], [33152.94135620117, -26675, -10641, 17836], [33153.02135620117, 3202, -25480, 27970], [33153.10135620117, 29664, -26117, -5154], [33153.181356201174, 6296, 17782, -25610], [33153.26135620117, -14316, -14633, -24990], [33153.34135620117, -6835, -9433, 26196], [33153.42135620117, 11844, -12533, -9070], [33153.50135620117, 925, 3546, -31505], [33153.581356201175, -18456, -9315, 20906], [33153.66135620117, 25479, -13736, -32552], [33153.74135620117, -31562, -15731, -11667], [33153.82135620117, -30121, -18400, -30323], [33153.901356201175, 21741, 27578, 10594], [33153.98135620117, 9208, -15228, -18521], [33154.06135620117, -4332, -13432, 11392], [33154.14135620117, 5296, -14029, 15607], [33154.221356201175, -19025, -28329, 4969], [33154.30135620117, -5504, -25426, -32083], [33154.38135620117, 2801, -7499, -17006], [33154.46135620117, -22466, -28542, -31731], [33154.541356201174, 16235, -13985, 30893], [33154.62135620117, 13793, 9411, -19779], [33154.70135620117, -22400, 24897, 6780], [33154.78135620117, 23648, -1615, -9921], [33154.861356201174, 19967, 790, 30743], [33154.94135620117, 16197, -18157, -5384], [33155.02135620117, 14537, -28831, -17150], [33155.10135620117, -28932, 2823, -15995], [33155.181356201174, -15606, -18784, -12599], [33155.26135620117, 9530, 25739, 5731], [33155.34135620117, 25488, 5866, -17361]], - "Analog Vbat": [[46993.57792663574, 1720.416258], [46993.67792663574, 2804.413837], [46993.77792663574, 3478.626059], [46993.877926635745, 2744.135321], [46993.977926635744, 3929.697792], [46994.07792663574, 1514.098265], [46994.17792663574, 298.979099], [46994.27792663574, 3244.032077], [46994.377926635745, 4197.258927], [46994.477926635744, 890.338412], [46994.57792663574, 4149.034723], [46994.67792663574, 3806.753036], [46994.77792663574, 3205.805066], [46994.877926635745, 2546.808674], [46994.977926635744, 1168.954991], [46995.07792663574, 2625.113218], [46995.17792663574, 1944.087874], [46995.27792663574, 3957.234171], [46995.377926635745, 1165.367448], [46995.477926635744, 3356.793846], [46995.57792663574, 2657.273559], [46995.67792663574, 3676.824909], [46995.77792663574, 776.23458], [46995.877926635745, 3700.608779], [46995.977926635744, 232.391581], [46996.07792663574, 3085.468911], [46996.17792663574, 1370.151837], [46996.27792663574, 3394.790279], [46996.377926635745, 2226.553048], [46996.477926635744, 3530.408589], [46996.57792663574, 3089.140311], [46996.67792663574, 1424.853389], [46996.77792663574, 694.315962], [46996.877926635745, 3296.994296], [46996.977926635744, 4011.112359], [46997.07792663574, 746.638216], [46997.17792663574, 3375.568048], [46997.27792663574, 3048.160503], [46997.377926635745, 325.685591], [46997.477926635744, 2628.708992], [46997.57792663574, 183.599789], [46997.67792663574, 3180.520117], [46997.77792663574, 2424.481854], [46997.877926635745, 1064.010765], [46997.977926635744, 2024.655199], [46998.07792663574, 616.773089], [46998.17792663574, 2827.006653], [46998.27792663574, 444.358977], [46998.377926635745, 4189.150304], [46998.477926635744, 1308.612927], [46998.57792663574, 2014.774038], [46998.67792663574, 3105.046341], [46998.77792663574, 952.756984], [46998.877926635745, 3171.061601], [46998.977926635744, 185.044732], [46999.07792663574, 3272.262021], [46999.17792663574, 3469.325984], [46999.27792663574, 1686.840634], [46999.377926635745, 1670.387299], [46999.477926635744, 3157.202666]], - "Constat": [[46997.77792663574, 3.2931134726145695e+23, -115, -11], [46997.822926635745, -2.9572660095179736e-20, 87, -49], [46997.86792663574, -1.5632246158845554e-17, 0, 110], [46997.91292663574, 914001536.0, 63, 90], [46997.95792663574, 3.3141467483151178e-28, 77, 0], [46998.002926635745, 2.839480176439744e-25, 45, -6], [46998.04792663574, 5.41977669854532e-07, -29, 62], [46998.09292663574, 6.980299147944369e+29, -26, -30], [46998.13792663574, -0.5808721780776978, 98, 55], [46998.182926635745, -65858814607360.0, -84, 69], [46998.227926635744, -8.207248273401886e-22, 66, 109], [46998.27292663574, -2.5999593258636195e-23, -34, -21], [46998.31792663574, 3937.537109375, -10, -101], [46998.362926635746, -27620.0390625, 98, -98], [46998.407926635744, -4.725843998592205e+16, 84, 102], [46998.45292663574, -2335065088.0, -110, -36], [46998.49792663574, 1.3436157157864966e-30, -17, -124], [46998.54292663574, -8.878755680773734e+16, -86, 81], [46998.587926635744, -3545313.75, -40, -128], [46998.63292663574, -70.75920104980469, 109, -46], [46998.67792663574, -3.827581167570315e-05, -115, -119], [46998.72292663574, 4.505225062224551e+26, 98, 41], [46998.767926635745, -1057.1240234375, -89, -73], [46998.81292663574, -17948200.0, -128, 44], [46998.85792663574, -733515.0, -9, 60], [46998.90292663574, -1.7016495042028876e-28, 105, 19], [46998.947926635745, -1.1574970956001797e-21, -83, -126], [46998.99292663574, -1.6698245443755977e+21, -110, -67], [46999.03792663574, -5.153517697281814e-29, 13, -124], [46999.08292663574, -914422.6875, -83, 120], [46999.127926635745, 8.46590438713809e-17, -67, -78], [46999.17292663574, 2.232727069263334e+20, 124, 26], [46999.21792663574, -1.1511376644347523e+35, 63, -39], [46999.26292663574, 4.417057328043474e-37, 23, 120], [46999.307926635745, -0.0001404258218826726, -8, -22], [46999.352926635744, -1.1104292902952337e-29, 2, -67], [46999.39792663574, 2.610762373727143e-32, -123, -63], [46999.44292663574, -4.791077117261011e-06, -55, -50], [46999.487926635746, 2.053423906218944e+22, 99, 22], [46999.532926635744, 3.7867571704188447e-25, 47, -68]] - } - } diff --git a/docs/source/usage/configuration_files.rst b/docs/source/usage/configuration_files.rst new file mode 100644 index 00000000..797f1f10 --- /dev/null +++ b/docs/source/usage/configuration_files.rst @@ -0,0 +1,333 @@ +.. _configuration_files: + +=================== +Configuration Files +=================== + +Configuration options for the gateway are supplied via a configuration file, +which was set up in :ref:`configuration`. + +Once the ``gateway start`` command is invoked, the configuration is saved with +the output data (if saving data locally). The configuration is also added to the +metadata on the output files uploaded to the cloud, where it is used by the cloud +ingress to populate the database. + +Specifying other configuration files +==================================== + +The easiest way of specifying the file is to set the ``GATEWAY_CONFIG_FILE`` +environment variable. + +But, there are other ways. If the environment variable is not set, ``data-gateway`` +looks for a file named ``config.json`` in the working directory, or the file path +can be overridden in the CLI options (also see ``gateway start --help``): + +.. code-block:: shell + + gateway start --config-file= + + +Useful customisations +===================== + +The most useful customisation is to add a ``measurement_campaign_reference`` field: + +.. code-block:: javascript + + { + "gateway": { + // ... + }, + "nodes": { + // ... + } + "measurement_campaign": { + // If you leave out this reference, a new one is created every + // time you start the gateway. That allows you to then filter + // down results in the dashboard to the exact run you're doing + // right now. + // But, if you're doing a series of related runs and want all + // the results to be able to be merged, the set the reference + // value here for continuity across runs. Don't forget to change + // or remove this if you reuse the file for something else, though. + "measurement_campaign_reference": "my-measurement-campaign", + // You can enable further sorting of data by adding + // campaign-specific labels + "label": "run-1", + // And add notes as aide-memoires. + "description": "It's windy right now and the battery charged up overnight so we're taking the opportunity to run with mic and diff baros turned on." + } + } + +Further customisation +===================== + +Any of the options in the ``data-gateway`` `configuration module `_ +can be customised by updating entries in the configuration file. + +.. warning:: + Moving off the beaten track, especially customising things like handles and packet keys, you really have to know what you're doing! + +Here is an example of a more extensive configuration file. + +.. code-block:: javascript + + { + "gateway": { + "baudrate": 2300000, + "endian": "little", + "installation_reference": "my_installation_reference", + "latitude": 0, + "longitude": 0, + "packet_key": 254, + "packet_key_offset": 245, + "receiver_firmware_version": "1.2.3", + "serial_buffer_rx_size": 100000, + "serial_buffer_tx_size": 1280, + "turbine_id": "unknown" + }, + "nodes": { + "0": { + "acc_freq": 100, + "acc_range": 16, + "analog_freq": 16384, + "baros_bm": 1023, + "baros_freq": 100, + "blade_id": "0", + "constat_period": 45, + "battery_info_period": 3600, + "decline_reason": { + "0": "Bad block detection ongoing", + "1": "Task already registered, cannot register again", + "2": "Task is not registered, cannot de-register", + "3": "Connection Parameter update unfinished" + }, + "diff_baros_freq": 1000, + "initial_node_handles": { + "34": "Abs. baros", + "36": "Diff. baros", + "38": "Mic 0", + "40": "Mic 1", + "42": "IMU Accel", + "44": "IMU Gyro", + "46": "IMU Magnetometer", + "48": "Analog1", + "50": "Analog2", + "52": "Constat", + "54": "Cmd Decline", + "56": "Sleep State", + "58": "Info Message" + }, + "gyro_freq": 100, + "gyro_range": 2000, + "remote_info_type": { + "0": "Battery info" + }, + "mag_freq": 12.5, + "mics_freq": 15625, + "mics_bm": 1023, + "max_timestamp_slack": 0.005, + "max_period_drift": 0.02, + "node_firmware_version": "unknown", + "number_of_sensors": { + "Mics": 10, + "Baros_P": 40, + "Baros_T": 40, + "Diff_Baros": 5, + "Acc": 3, + "Gyro": 3, + "Mag": 3, + "Analog Vbat": 1, + "Constat": 4, + "battery_info": 3 + }, + "periods": { + "Mics": 6.4e-5, + "Baros_P": 0.01, + "Baros_T": 0.01, + "Diff_Baros": 0.001, + "Acc": 0.01, + "Gyro": 0.01, + "Mag": 0.08, + "Analog Vbat": 6.103515625e-5, + "Constat": 0.045, + "battery_info": 3600 + }, + "samples_per_packet": { + "Mics": 8, + "Diff_Baros": 24, + "Baros_P": 1, + "Baros_T": 1, + "Acc": 40, + "Gyro": 40, + "Mag": 40, + "Analog Vbat": 60, + "Constat": 24, + "battery_info": 1 + }, + "sensor_conversion_constants": { + "Mics": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + "Diff_Baros": [1, 1, 1, 1, 1], + "Acc": [1, 1, 1], + "Gyro": [1, 1, 1], + "Mag": [1, 1, 1], + "Analog Vbat": [1], + "Constat": [1, 1, 1, 1], + "battery_info": [1e6, 100, 256] + }, + "sensor_coordinates": { + "Mics": [ + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0] + ], + "Baros_P": [ + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0] + ], + "Baros_T": [ + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0] + ], + "Diff_Baros": [ + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0] + ], + "Acc": [ + [0, 0, 0], + [0, 0, 0], + [0, 0, 0] + ], + "Gyro": [ + [0, 0, 0], + [0, 0, 0], + [0, 0, 0] + ], + "Mag": [ + [0, 0, 0], + [0, 0, 0], + [0, 0, 0] + ], + "Analog Vbat": [[0, 0, 0]], + "Constat": [ + [0, 0, 0], + [0, 0, 0], + [0, 0, 0], + [0, 0, 0] + ], + "battery_info": [ + [0, 0, 0], + [0, 0, 0], + [0, 0, 0] + ] + }, + "sensor_names": [ + "Mics", + "Baros_P", + "Baros_T", + "Diff_Baros", + "Acc", + "Gyro", + "Mag", + "Analog Vbat", + "Constat", + "battery_info" + ], + "sleep_state": { + "0": "Exiting sleep", + "1": "Entering sleep" + } + } + }, + "measurement_campaign": { + "label": "my-test-1", + "description": null + } + } + diff --git a/docs/source/usage/daemonising.rst b/docs/source/usage/daemonising.rst new file mode 100644 index 00000000..5965eeaa --- /dev/null +++ b/docs/source/usage/daemonising.rst @@ -0,0 +1,57 @@ +.. _daemonising: + +=========== +Daemonising +=========== + +.. warning:: + + Daemonisation cannot happen reliably until :ref:`_ is solved. + +During the aerosense project, Balena has made it so convenient +to shell in and manage sessions that it's the only thing we've actually done. + +However, this sometimes means babysitting the gateway and takes up time - that's fine +in the very early days, but if you are setting up a longer-term deployment of aerosense +test rig) you should *daemonise* the ``gateway start`` command. + +This basically means set the system up to: + + - start the gateway along with the rest of the OS on boot + - restart the gateway program if it crashes + +There are lots of ways of doing this but we **strongly** recommend using `supervisord `_, +which, as the name suggests, is a supervisor for daemonised processes. + +Install supervisord on your system: + +.. code-block:: shell + + # Ensure you've got the latest version of supervisord installed + sudo apt-get install --update supervisord + +Configure supervisord to (`more info here `_) run +the gateway as a daemonised service: + +.. warning:: + + We've not actualy done this (see the warning above) but it should look very similar to this: + +.. code-block:: shell + + sudo gateway supervisord-conf >> /etc/supervisord.conf + +Restarting your system, at this point, should start the gateway process at boot time. + +You can use `supervisorctl `_ to check gateway status: + +.. code-block:: shell + + supervisorctl status AerosenseGateway + +Similarly, you can stop and start the daemon with: + +.. code-block:: shell + + supervisorctl stop AerosenseGateway + supervisorctl start AerosenseGateway diff --git a/docs/source/usage/gateway_cli.rst b/docs/source/usage/gateway_cli.rst new file mode 100644 index 00000000..d9ab3fd1 --- /dev/null +++ b/docs/source/usage/gateway_cli.rst @@ -0,0 +1,81 @@ +.. _gateway_cli: + +=========== +Gateway CLI +=========== + +The gateway has a CLI which means you can call it just like any other unix command. + +It is called simply ``gateway``. Once the code is :ref:`deployed/installed `, you can see the options and +help by typing: + +.. code-block:: shell + + gateway --help + +Or see more detailed help on a subcommand (eg ``start``) with: + +.. code-block:: shell + + gateway start --help + + +Start +===== + +The ``start`` subcommand is overwhelmingly the most common you'll use. + +Once started, data is read continuously from the serial port, parsed, +processed, batched into time windows, and either: + +- :ref:`uploaded ` to an ingress Google Cloud storage bucket where it is cleaned and + forwarded to another bucket for storage, or + +- saved locally as JSON files, or + +- both. + +The start command also allows you to send commands to the base station (which will +broadcast them to the nodes). The sequence of commands you send is called a "routine" and the +commands can be sent automatically (for long term acquisition) or interactivel (for debug/test). + +Automatic mode +-------------- + +Running the gateway in automatic mode doesn't allow further commands to be passed +to the serial port. Instead, a :ref:`routine file ` must be specified, +and the commands in it are issues automatically on your behalf, looping indefinitely. + +Assuming you have your configuration and routine files set up per :ref:`the instructions here `, +to start this mode, type: + +.. code-block:: shell + + gateway start + +You can stop the gateway by pressing ``Ctrl + C``. + +Interactive mode +---------------- + +Running the gateway in interactive mode allows commands to be sent to the serial port while the gateway is +running. A routine file can't be provided if using this mode. Any commands entered interactively are logged to a +``commands.txt`` file in the output directory. + +To start this mode, type: + +.. code-block:: shell + + gateway start --interactive + +Typing ``stop`` or pressing ``Ctrl + C`` will stop the session. + + +Other options +------------- + +* The window size (default 600 seconds) can be set by using ``--window-size=`` after the `start` command +* You can store data locally instead of or at the same time as it is sent to the cloud by using the ``--save-locally`` option +* To avoid sending any data to the cloud, provide the ``--no-upload-to-cloud`` option + + diff --git a/docs/source/usage/index.rst b/docs/source/usage/index.rst new file mode 100644 index 00000000..176c3ea0 --- /dev/null +++ b/docs/source/usage/index.rst @@ -0,0 +1,20 @@ +.. _usage: + +===== +Usage +===== + +A :ref:`gateway Command Line Interface (CLI) ` is provided to +streamline gateway management and use. We recommend you get started with +the gateway by looking at the following sections in order. Be sure to check +out the :ref:`tips `! + +.. toctree:: + :caption: Usage Contents + :maxdepth: 1 + + gateway_cli + configuration_files + routine_files + tips + daemonising diff --git a/docs/source/routines.rst b/docs/source/usage/routine_files.rst similarity index 61% rename from docs/source/routines.rst rename to docs/source/usage/routine_files.rst index 1229f7fc..dff80ee6 100644 --- a/docs/source/routines.rst +++ b/docs/source/usage/routine_files.rst @@ -1,8 +1,8 @@ -.. _sensor_command_routines: +.. _routine_files: -======================= -Sensor command routines -======================= +============= +Routine files +============= Commands can be sent to the sensors in two ways: @@ -11,7 +11,7 @@ Commands can be sent to the sensors in two ways: Creating and providing a routine file -------------------------------------- +===================================== A routine file looks like this: @@ -37,7 +37,7 @@ exist and the ``--routine-file`` option isn't provided, the command assumes ther Routine file schema -------------------- +=================== - The ``commands`` key in the file should be a list of two-element lists. Each two-element list should comprise a valid string command to send to the sensors and a delay in seconds from the gateway starting to run the command. @@ -46,49 +46,49 @@ Routine file schema Example routine files ---------------------- +===================== -The following routine file instructs the gateway to .. code-block:: shell - { - "commands": [ - [ - "startDiffBaros", - 60 - ], - [ - "startIMU", - 65 - ], - [ - "getBattery", - 70 - ], - [ - "stopDiffBaros", - 660 - ], - [ - "startBaros", - 670 + { + "commands": [ + [ + "startDiffBaros", + 60 + ], + [ + "startIMU", + 65 + ], + [ + "getBattery", + 70 + ], + [ + "stopDiffBaros", + 660 ], - [ - "startMics", - 1265 - ], - [ - "stopBaros", - 1270 - ], - [ - "stopIMU", - 1275 - ], - [ - "stopMics", + [ + "startBaros", + 670 + ], + [ + "startMics", + 1265 + ], + [ + "stopBaros", + 1270 + ], + [ + "stopIMU", + 1275 + ], + [ + "stopMics", 1280 - ] - ], - "period": 3600 - } + ] + ], + "period": 3600 + } + diff --git a/docs/source/information_for_developers.rst b/docs/source/usage/tips.rst similarity index 71% rename from docs/source/information_for_developers.rst rename to docs/source/usage/tips.rst index e976a7db..e07d01c3 100644 --- a/docs/source/information_for_developers.rst +++ b/docs/source/usage/tips.rst @@ -1,15 +1,40 @@ -.. _information_for_developers: +.. _tips: + +==== +Tips +==== + +.. _editing_files: + +Editing configuration and routine files +======================================= + +Often, whilst testing or getting set up, you'll want to edit files on a device itself, +particularly a configuration or routine file that you're just trying out. + +For this, the ``nano`` editor has been installed in the built containers so you can do: + +.. code-block:: shell + + $ nano $GATEWAY_ROUTINE_FILE + +or + +.. code-block:: shell + + $ nano $GATEWAY_CONFIG_FILE + +to edit either the routine or the configuration files. -========================== -Information for Developers -========================== .. _scp_of_files: SCP of files to/from BalenaOS ============================= -Sometimes, for debugging, you might run the gateway with the ``-l`` and/or ``--save-local-logs`` options, then want to +For 99% of the time, using nano to edit (or paste from your own preferred IDE) will be fine. +Occasionally though, you'll want to get files on/off a device. In particular, for debugging, +you might run the gateway with the ``-l`` and/or ``--save-local-logs`` options, then want to directly inspect the data files that result. However, the ``balena`` cli doesn't support ``scp`` that well out of the box (although there are workarounds using tunneling). @@ -18,7 +43,7 @@ To copy files between ``/data`` directory (on a container deployed by Balena) an #. `Install the balena CLI `_ and do ``balena login`` -#. Add your public ssh key to BalenaCloud and make sure you can use ``balena ssh` correctly, `following these instructions `_ +#. Add your public ssh key to BalenaCloud and make sure you can use ``balena ssh`` correctly, `following these instructions `_ #. Check `this GitHub issue `_. If closed with a new balena CLI feature, then follow those instructions instead. Otherwise use the following workaround. @@ -27,6 +52,7 @@ To copy files between ``/data`` directory (on a container deployed by Balena) an #. Get the full UUID of the device: .. code-block:: shell + $ balena devices ID UUID DEVICE NAME DEVICE TYPE FLEET STATUS IS ONLINE SUPERVISOR VERSION OS VERSION DASHBOARD URL 7294376 4bfe19d fried-firefly raspberrypi4-64 aerosense/gateways Idle true 13.1.11 balenaOS 2.98.33 https://dashboard.balena-cloud.com/devices/4bfe19d3651d27dc89d4b1a8c95061fa/summary @@ -41,6 +67,7 @@ To copy files between ``/data`` directory (on a container deployed by Balena) an #. Get the App ID (which is actually the Fleet ID), in this case ``1945598``: .. code-block:: shell + balena fleets ID NAME SLUG DEVICE TYPE ONLINE DEVICES DEVICE COUNT 1945598 gateways aerosense/gateways raspberrypi4-64 1 3 @@ -48,11 +75,14 @@ To copy files between ``/data`` directory (on a container deployed by Balena) an #. We'll be copying from balena's Host OS, not from the container. The ``/data`` directory `isn't mounted in the same place as when you're inside the container `_. So the root of the data folder is: .. code-block:: shell + /var/lib/docker/volumes/_resin-data/_data/ #. So for example, to copy a file from within the ``/data`` folder from remote to local, we do: .. code-block:: shell + scp-uuid 4bfe19d3651d27dc89d4b1a8c95061fa.balena:/var/lib/docker/volumes/1945598_resin-data/_data/gateway/20221122T100229/window-2.json . #. The scp command should work recursively with folders, but take care because they can be large if a long session has taken place. + diff --git a/docs/source/using_the_gateway.rst b/docs/source/using_the_gateway.rst deleted file mode 100644 index 0e8ba7ef..00000000 --- a/docs/source/using_the_gateway.rst +++ /dev/null @@ -1,528 +0,0 @@ -.. _using_the_gateway: - -================= -Using the Gateway -================= - -.. _gateway_cli: - -Gateway CLI -=========== - -The gateway has a CLI which means you can call it just like any other unix command. - -It is called simply ``gateway`` and, once :ref:`installed `, you can see the options and -help by typing: - -.. code-block:: shell - - gateway --help - -The main command to start the gateway running is: - -.. code-block:: shell - - gateway start - -You can see help about this command by executing: - -.. code-block:: shell - - gateway start --help - - -Automatic mode --------------------------------- -Running the gateway in automatic mode doesn't allow further commands to be passed to the serial port. Instead, a -:doc:`routine JSON file ` can be provided via the ``--routine-file`` option. Data from the serial port is -processed, batched into time windows, and uploaded to an ingress Google Cloud storage bucket where it is cleaned and -forwarded to another bucket for storage. This is the mode you'll want to deploy in production. - -Before starting this mode, Google application credentials must be provided. To start this mode, type: - -.. code-block:: shell - - export GOOGLE_APPLICATION_CREDENTIALS=/absolute/path/to/service/account/file.json - - gateway start --gcp-bucket-name=my-bucket --output-dir=path/to/output-directory-in-cloud-bucket - -If the connection to Google Cloud fails, windows will be written to the hidden directory -``.//.backup`` where they will stay until the connection resumes. Backup files are deleted upon -successful cloud upload. - -You can stop the gateway by pressing ``Ctrl + C``. - - -Interactive mode ----------------- -Running the gateway in interactive mode allows commands to be sent to the serial port while the gateway is -running. A routine file can't be provided if using this mode. Any commands entered interactively are logged to a -``commands.txt`` file in the output directory. - -To start this mode, type: - -.. code-block:: shell - - export GOOGLE_APPLICATION_CREDENTIALS=/absolute/path/to/service/account/file.json - - gateway start --interactive --output-dir= - -Typing ``stop`` or pressing ``Ctrl + C`` will stop the session. - - -Other options -------------- -* The window size (default 600 seconds) can be set by using ``--window-size=`` after the `start` command -* You can store data locally instead of or at the same time as it is sent to the cloud by using the ``--save-locally`` option -* To avoid sending any data to the cloud, provide the ``--no-upload-to-cloud`` option - - -.. _configuring: - - -Configuring the Gateway -======================= - -Configuration options for the gateway can be supplied via a configuration file. By default, **data-gateway** looks for -a file named ``config.json`` in the working directory, although the CLI allows this to be overridden to use a specific -configuration file. Here is the contents of an example configuration file: - -.. code-block:: json - - { - "gateway": { - "baudrate": 2300000, - "endian": "little", - "initial_gateway_handles": { - "64": "Local Info Message" - }, - "installation_reference": "my_installation_reference", - "latitude": 0, - "local_info_types": { - "0": "Synchronization not ready as not every sensor node is connected", - "1": "Time synchronization info", - "2": "Time sync exception", - "4": "Time sync coarse data record error", - "8": "Time sync alignment error", - "16": "Time sync coarse data time diff error", - "32": "Device not connected", - "64": "Select message destination successful", - "128": "Time sync success", - "129": "Coarse sync finish", - "130": "Time sync msg sent" - }, - "longitude": 0, - "packet_key": 254, - "packet_key_offset": 245, - "receiver_firmware_version": "1.2.3", - "serial_buffer_rx_size": 100000, - "serial_buffer_tx_size": 1280, - "turbine_id": "unknown" - }, - "nodes": { - "0": { - "acc_freq": 100, - "acc_range": 16, - "analog_freq": 16384, - "baros_bm": 1023, - "baros_freq": 100, - "blade_id": "0", - "constat_period": 45, - "battery_info_period": 3600, - "decline_reason": { - "0": "Bad block detection ongoing", - "1": "Task already registered, cannot register again", - "2": "Task is not registered, cannot de-register", - "3": "Connection Parameter update unfinished" - }, - "diff_baros_freq": 1000, - "initial_node_handles": { - "34": "Abs. baros", - "36": "Diff. baros", - "38": "Mic 0", - "40": "Mic 1", - "42": "IMU Accel", - "44": "IMU Gyro", - "46": "IMU Magnetometer", - "48": "Analog1", - "50": "Analog2", - "52": "Constat", - "54": "Cmd Decline", - "56": "Sleep State", - "58": "Info Message" - }, - "gyro_freq": 100, - "gyro_range": 2000, - "remote_info_type": { - "0": "Battery info" - }, - "mag_freq": 12.5, - "mics_freq": 15625, - "mics_bm": 1023, - "max_timestamp_slack": 0.005, - "max_period_drift": 0.02, - "node_firmware_version": "unknown", - "number_of_sensors": { - "Mics": 10, - "Baros_P": 40, - "Baros_T": 40, - "Diff_Baros": 5, - "Acc": 3, - "Gyro": 3, - "Mag": 3, - "Analog Vbat": 1, - "Constat": 4, - "battery_info": 3 - }, - "periods": { - "Mics": 6.4e-5, - "Baros_P": 0.01, - "Baros_T": 0.01, - "Diff_Baros": 0.001, - "Acc": 0.01, - "Gyro": 0.01, - "Mag": 0.08, - "Analog Vbat": 6.103515625e-5, - "Constat": 0.045, - "battery_info": 3600 - }, - "samples_per_packet": { - "Mics": 8, - "Diff_Baros": 24, - "Baros_P": 1, - "Baros_T": 1, - "Acc": 40, - "Gyro": 40, - "Mag": 40, - "Analog Vbat": 60, - "Constat": 24, - "battery_info": 1 - }, - "sensor_commands": { - "start": ["startBaros", "startDiffBaros", "startIMU", "startMics"], - "stop": ["stopBaros", "stopDiffBaros", "stopIMU", "stopMics"], - "configuration": [ - "configBaros", - "configAccel", - "configGyro", - "configMics" - ], - "utilities": [ - "getBattery", - "setConnInterval", - "tpcBoostIncrease", - "tpcBoostDecrease", - "tpcBoostHeapMemThr1", - "tpcBoostHeapMemThr2", - "tpcBoostHeapMemThr4" - ] - }, - "sensor_conversion_constants": { - "Mics": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], - "Diff_Baros": [1, 1, 1, 1, 1], - "Baros_P": [ - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96, - 40.96 - ], - "Baros_T": [ - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100, - 100 - ], - "Acc": [1, 1, 1], - "Gyro": [1, 1, 1], - "Mag": [1, 1, 1], - "Analog Vbat": [1], - "Constat": [1, 1, 1, 1], - "battery_info": [1e6, 100, 256] - }, - "sensor_coordinates": { - "Mics": [ - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0] - ], - "Baros_P": [ - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0] - ], - "Baros_T": [ - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0] - ], - "Diff_Baros": [ - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0] - ], - "Acc": [ - [0, 0, 0], - [0, 0, 0], - [0, 0, 0] - ], - "Gyro": [ - [0, 0, 0], - [0, 0, 0], - [0, 0, 0] - ], - "Mag": [ - [0, 0, 0], - [0, 0, 0], - [0, 0, 0] - ], - "Analog Vbat": [[0, 0, 0]], - "Constat": [ - [0, 0, 0], - [0, 0, 0], - [0, 0, 0], - [0, 0, 0] - ], - "battery_info": [ - [0, 0, 0], - [0, 0, 0], - [0, 0, 0] - ] - }, - "sensor_names": [ - "Mics", - "Baros_P", - "Baros_T", - "Diff_Baros", - "Acc", - "Gyro", - "Mag", - "Analog Vbat", - "Constat", - "battery_info" - ], - "sleep_state": { - "0": "Exiting sleep", - "1": "Entering sleep" - } - } - }, - "measurement_campaign": { - "label": "my-test-1", - "description": null - } - } - - -A default configuration is used if a ``config.json`` file is not specified and one is not found in the working -directory. If a configuration file is specified, all of the fields seen above must be present for it to be valid. - -One configuration is used per run of the ``start`` command. A copy is saved with the output data if saving data -locally. The configuration is saved as metadata on the output files uploaded to the cloud. To supply the configuration -file and start the gateway, type the following, supplying any other options you need: - -.. code-block:: shell - - gateway start --config-file= - - -.. _daemonising_the_installation: - -Daemonising the installation -============================ - -If you are setting up a deployment of aerosense (on a turbine nacelle, rather than on prototype equipment or a -test rig) you should *daemonise* the gateway. - -This basically means set the system up to: - - - start the gateway along with the rest of the OS on boot - - restart the gateway program if it crashes - -There are lots of ways of doing this but we **strongly** recommend using `supervisord `_, -which, as the name suggests, is a supervisor for daemonised processes. - -Install supervisord on your system: - -.. code-block:: shell - - # Ensure you've got the latest version of supervisord installed - sudo apt-get install --update supervisord - -Configure supervisord to (`more info here `_) run -the gateway as a daemonised service: - -.. code-block:: shell - - sudo gateway supervisord-conf >> /etc/supervisord.conf - # Or, if you want to set up the daemon with a specific configuration file - sudo gateway supervisord-conf --config-file = /path/to/my/config.json >> /etc/supervisord.conf - -Restarting your system, at this point, should start the gateway process at boot time. - -You can use `supervisorctl `_ to check gateway status: - -.. code-block:: shell - - supervisorctl status AerosenseGateway - -Similarly, you can stop and start the daemon with: - -.. code-block:: shell - - supervisorctl stop AerosenseGateway - supervisorctl start AerosenseGateway diff --git a/examples/routines/pbl-test-paper_three.json b/examples/routines/pbl-test-paper_three.json index b5bc7e4a..5cca1eb5 100644 --- a/examples/routines/pbl-test-paper_three.json +++ b/examples/routines/pbl-test-paper_three.json @@ -1,29 +1,29 @@ { "commands": [ - ["selMsgDest 1", 5], + ["selMsgDest 1", 5], ["selMsgDest 1", 6], - ["startMics", 7], - ["selMsgDest 2", 8], + ["startMics", 7], + ["selMsgDest 2", 8], ["selMsgDest 2", 9], ["startMics", 10], - ["selMsgDest 3", 11], + ["selMsgDest 3", 11], ["selMsgDest 3", 12], ["startMics", 13], ["stopMics", 43], - ["selMsgDest 2", 44], + ["selMsgDest 2", 44], ["selMsgDest 2", 45], ["stopMics", 46], - ["selMsgDest 1", 47], + ["selMsgDest 1", 47], ["selMsgDest 1", 48], ["stopMics", 49], ["readMics", 50], - ["selMsgDest 2", 350], - ["selMsgDest 2", 351], - ["readMics", 352], - ["selMsgDest 3", 653], - ["selMsgDest 3", 654], - ["readMics", 655], - ["selMsgDest 1", 955] + ["selMsgDest 2", 550], + ["selMsgDest 2", 551], + ["readMics", 552], + ["selMsgDest 3", 1052], + ["selMsgDest 3", 1054], + ["readMics", 1055], + ["selMsgDest 1", 1555] ], - "period": 1000 -} \ No newline at end of file + "period": 2000 +} diff --git a/pyproject.toml b/pyproject.toml index 6382b210..0b6394b9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "data-gateway" -version = "0.16.1" +version = "0.16.2" repository = "https://github.com/aerosense-ai/data-gateway" description = "A data gateway that runs on-nacelle for relaying data streams from aerosense nodes to cloud." readme = "README.md"