Merge pull request #120 from aerosense-ai/doc/update-documentation

Update documentation

README.md

@@ -2,22 +2,25 @@
 [![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
 [![black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)
 [![python-template](https://img.shields.io/badge/template-python--library-blue)](https://github.com/thclark/python-library-template)
-
+[![docs](https://readthedocs.org/projects/aerosense-data-gateway/badge/?version=latest)](https://aerosense-data-gateway.readthedocs.io/en/latest/)
 
 # data-gateway
 
-Read the docs [here.](https://aerosense-data-gateway.readthedocs.io/en/latest/)
+Usage: Read the documentation [here.](https://aerosense-data-gateway.readthedocs.io/en/latest/)
 
-*Note that the test coverage figure is more like 90% - the recent addition of multiprocessing has made it difficult to
-measure the true coverage across multiple processes.*
+_Note that the test coverage figure is more like 90% - the recent addition of multiprocessing has made it difficult to
+measure the true coverage across multiple processes._
 
 ## Installation and usage
+
 To install, run:
+
 ```shell
 pip install git+https://github.com/aerosense-ai/data-gateway.git
 ```
 
 The command line interface (CLI) can then be accessed via:
+
 ```shell
 gateway --help
 ```
@@ -50,28 +53,36 @@ Commands:
 ### Installation
 
 #### Poetry
+
 We're using `poetry` instead of `pip` to manage the package to take advantage of the `poetry.lock` file [among other
 useful features](https://python-poetry.org/). In terms of developer experience, this just means there are some slightly
 different commands to run than usual. `data-gateway` can still be `pip`-installed by anyone anywhere, but dependency
 resolution and dependency specification for `data-gateway` developers is improved by using `poetry` locally.
 
 #### Architecture-specific installations
+
 Due to some (most likely temporary) constraints with `poetry` and the need to run and develop the gateway on Linux,
 Windows, M1 Macs, and Raspberry Pis, the need has arisen for some slightly different installation procedures on these
-different architectures/platforms. Instructions are detailed below - [click here](https://github.com/aerosense-ai/data-gateway/issues/65)
+different architectures/platforms. [click here](https://github.com/aerosense-ai/data-gateway/issues/65)
 to read more.
 
+For Raspberry Pis, overwhelmingly the simple thing to do is use balena instead ([see the docs](https://aerosense-data-gateway.readthedocs.io/en/latest/)).
+
 #### Clone the repository
+
 First, clone the repository and `cd` into it:
+
 ```shell
 git clone https://github.com/aerosense-ai/data-gateway.git
 cd data-gateway
 ```
 
 Then follow the instructions for your platform below.
 
-#### Install on MacOS and Linux (except on Raspberry Pi)
+#### Install on MacOS and Linux
+
 Run the following from the repository root:
+
 ```shell
 pip install poetry
 
@@ -80,25 +91,31 @@ poetry install
 ```
 
 This will editably install `data-gateway` in a `poetry`-managed virtual environment, meaning:
+
 - Any local changes you make to it will be automatically used when running it locally
 - It won't be affected by changes to other python packages you have installed on your system, making development much
   easier and more deterministic
 
 #### Install on Raspberry Pi
-Run the following from the repository root:
+
+It's best to use [balena](https://aerosense-data-gateway.readthedocs.io/en/latest/) but if you do need to run the install, do the following from the repository root:
+
 ```shell
 pip install -r requirements-pi-dev.txt
 ```
 
 #### Install on Windows
-This workflow works for Windows using Powershell.
+
+This workflow works for Windows using Powershell, if you're the most masochistic of people.
 
 Prerequisites:
+
 1. Make sure to have python not installed from [python.org](https://www.python.org/)
 2. Install [pyenv-win](https://github.com/pyenv-win/pyenv-win) via pip method
-3. Execute ```pip install virtualenv```
+3. Execute `pip install virtualenv`
 
 Installation:
+
 ```shell
 pyenv install 3.8.0  # (or higher)
 pyenv local 3.8.0
@@ -110,25 +127,32 @@ poetry install
 ```
 
 Every time you enter the repo over powershell again, make sure to activate the venv using
+
 ```shell
 ./venv/Scripts/activate
 ```
 
 #### Troubleshooting
+
 If there are problems reading the serial port, try running this (or the equivalent on non-Linux platforms) and retrying:
+
 ```shell
 sudo apt-get update
 sudo apt-get install libhdf5-dev libhdf5-serial-dev
 ```
 
 ### Testing
+
 These environment variables need to be set to run the tests:
-* `GOOGLE_APPLICATION_CREDENTIALS=/absolute/path/to/service/account/file.json`
+
+- `GOOGLE_APPLICATION_CREDENTIALS=/absolute/path/to/service/account/file.json`
 
 Then, from the repository root, run
+
 ```bash
 tox
 ```
 
 ## Contributing
+
 Take a look at our [contributing](/docs/contributing.md) page.

docs/source/deployment.rst → docs/source/cloud/creating_service_accounts.rst

@@ -1,51 +1,53 @@
-.. _deployment:
 
-==========
-Deployment
-==========
+.. _creating_service_accounts:
 
-To deploy the Aerosense system, you must:
+=========================
+Creating Service Accounts
+=========================
 
-- Register the deployment
-- Create a deployment Service Account
-- Configure and run the gateway (at the on-nacelle processor)
+The gateway that you install on a turbine needs to upload data to Aerosense Cloud. However, we don't want "just anybody"
+to be able to write data to the store - that leaves us vulnerable to a wide range of attacks. So the gateway must
+authenticate itself with the store prior to upload.
 
+To enable the gateway to authenticate itself, we use a **service account**, which is a bit like a user account (it has
+an email address and can be given certain permissions) but for a non-human.
 
-.. _register_the_deployment:
+Here, we will create a service account for a deployment - this will result in a single credentials file that we can
+reuse across the gateways (turbines) in the deployment to save administrative overhead maintaining all the credentials.
 
-Register the deployment
-=======================
+Choose your service account name
+================================
 
-A "deployment" is the act of commissioning the Aerosense system on a group of turbines (typically the turbines in a
-particular wind farm).
+When creating service accounts, whilst any name will work, sticking to a naming convention is very helpful.
+There are three kinds of service account names:
 
-For experimental and test purposes, there will typically be only one turbine (and therefore one gateway) per deployment.
+**For deployment with Balena**
 
-TODO Registration process documentation - See Tom and Yuriy
+If you've deployed gateway code using Balena, the newly added device will be given a "coolname" like ``fried-firefly`` or ``holy-lake``.
+Use this name as your service account name, which makes it super easy to diagnose any problems, or restrict permissions
+if a device is lost.
 
+**For manual deployment**
 
-.. _create_a_service_account:
+Use a name that's prefixed with ``as-deployment-``, eg ``as-deployment-tommasso``. These service accounts should be set up
+to have consistent permissions as with a balena deployment but simply indicate its for manual trialling of uploads from the gateway.
 
-Create a Service Account
-========================
+**For development**
 
-The gateway that you install on a turbine needs to upload data to Aerosense Cloud. However, we don't want "just anybody"
-to be able to write data to the store - that leaves us vulnerable to a wide range of attacks. So the gateway must
-authenticate itself with the store prior to upload.
+Developers will work across the entire stack of cloud functions, gateway and other aspects of the project like dashboard or tools.
+Thus developers' service accounts are expected to have a wide and varied range of permissions.
+The name should be in the form ``developer-<github-handle>``, eg ``developer-thclark``.
 
-To enable the gateway to authenticate itself, we use a **service account**, which is a bit like a user account (it has
-an email address and can be given certain permissions) but for a non-human.
 
-Here, we will create a service account for a deployment - this will result in a single credentials file that we can
-reuse across the gateways (turbines) in the deployment to save administrative overhead maintaining all the credentials.
+Create account and credentials
+==============================
 
 Log in to the ``aerosense-twined`` project on Google Cloud Platform (GCP) and work through the following steps:
 
 
 **1. Go to IAM > Service Accounts > Create**
 
-
-.. figure:: images/creating-a-service-account/1-go-to-iam-service-accounts.png
+.. figure:: /images/creating-a-service-account/1-go-to-iam-service-accounts.png
     :width: 600px
     :align: center
     :figclass: align-center
@@ -55,17 +57,17 @@ Log in to the ``aerosense-twined`` project on Google Cloud Platform (GCP) and wo
 
 **2. Create the service account**
 
-.. figure:: images/creating-a-service-account/2-create-service-account.png
+.. figure:: /images/creating-a-service-account/2-create-service-account.png
     :width: 600px
     :align: center
     :figclass: align-center
 
-    The service account name should contain your deployment id (from above) in the pattern
-    ``as-deployment-<deploymentId>``. In this case, ``deploymentId = gatewaytest``
+    The service account name should be per the naming guidelines above.
+    In this image, ``as-deployment-gatewaytest`` was used.
 
 **3. Skip assignation of optional roles and users (for now)**
 
-.. figure:: images/creating-a-service-account/3-no-grants-or-users.png
+.. figure:: /images/creating-a-service-account/3-no-grants-or-users.png
     :width: 600px
     :align: center
     :figclass: align-center
@@ -74,21 +76,21 @@ Log in to the ``aerosense-twined`` project on Google Cloud Platform (GCP) and wo
 
 **4. Create and download a private JSON key for this Service Account**
 
-.. figure:: images/creating-a-service-account/4a-create-key.png
+.. figure:: /images/creating-a-service-account/4a-create-key.png
     :width: 600px
     :align: center
     :figclass: align-center
 
     Find your newly created service account in the list (you may have to search) and click 'Create Key'.
 
-.. figure:: images/creating-a-service-account/4b-key-should-be-json.png
+.. figure:: /images/creating-a-service-account/4b-key-should-be-json.png
     :width: 600px
     :align: center
     :figclass: align-center
 
     Choose the default JSON key type.
 
-.. figure:: images/creating-a-service-account/4c-key-will-be-saved.png
+.. figure:: /images/creating-a-service-account/4c-key-will-be-saved.png
     :width: 600px
     :align: center
     :figclass: align-center
@@ -97,7 +99,7 @@ Log in to the ``aerosense-twined`` project on Google Cloud Platform (GCP) and wo
 
 **5. Locate the ingress bucket in the storage browser, and click on "Add Member"**
 
-.. figure:: images/creating-a-service-account/5-locate-aerosense-ingress-bucket.png
+.. figure:: /images/creating-a-service-account/5-locate-aerosense-ingress-bucket.png
     :width: 600px
     :align: center
     :figclass: align-center
@@ -107,7 +109,7 @@ Log in to the ``aerosense-twined`` project on Google Cloud Platform (GCP) and wo
 
 **6. Assign ``Storage Object Creator`` permission**
 
-.. figure:: images/creating-a-service-account/5-locate-aerosense-ingress-bucket.png
+.. figure:: /images/creating-a-service-account/6-add-storage-object-creator.png
     :width: 600px
     :align: center
     :figclass: align-center
@@ -120,15 +122,8 @@ And you're done! Keep that downloaded permission file for later.
 
 .. ATTENTION::
 
-   Do not add to a docker image, email, skype/meet/zoom, dropbox, whatsapp, commit to git, post in an issue, or
-   whatever, this private credentials file.
+   Do not add this private credentials file to a docker image, email, skype/meet/zoom, dropbox, whatsapp, git commit,
+   post in an issue, or anywhere else.
 
    Doing so will earn you the penance of flushing and rotating all the system credentials.
 
-
-.. _configure_and_run_the_gateway:
-
-Configure and run the Gateway
-=============================
-
-See :doc:`using the gateway. <using_the_gateway>`

docs/source/cloud/index.rst

@@ -0,0 +1,17 @@
+.. _cloud:
+
+=====
+Cloud
+=====
+
+The majority of aspects of the aerosense cloud are documented in other 
+repositories (eg aerosense-tools, aerosense-dashboard). However, on the 
+gateway / ingress side it's worth collecting some notes here.
+
+.. toctree::
+   :caption: Cloud Contents
+   :maxdepth: 1
+
+   cloud_functions
+   creating_service_accounts
+   uploads

docs/source/cloud/uploads.rst

@@ -0,0 +1,11 @@
+.. _uploads:
+
+=======
+Uploads
+=======
+
+Once running ``gateway start``, windows are uploaded to the cloud ingress bucket (unless the ``--no-upload-to-cloud`` option is used).
+
+If the connection to Google Cloud fails, windows will be written to the hidden directory
+``./<output_directory>/.backup`` where they will stay until the connection resumes.
+Backup files are deleted upon successful cloud upload.

docs/source/deployment/check.rst

@@ -0,0 +1,22 @@
+.. _check:
+
+=====
+Check
+=====
+
+Once the above steps are complete, in the balena (or your own, for a manual installation) terminal, check the installation by typing:
+
+.. code-block:: shell
+   
+   gateway --help
+
+.. figure:: /images/balena/gateway-help.png
+    :width: 600px
+    :align: center
+    :figclass: align-center
+
+    If the gateway is correctly installed, you should see this.
+
+
+
+