Development

🚧 Under Construction 🚧

This Wiki is currently being developed.

Please refer to the README for information about SCOS Sensor in the meantime.

---

This page contains helpful information for developers wishing to contribute to SCOS Sensor.

Running the Sensor in Development

The following techniques can be used to make local modifications. Sections are in order, so "Running Tests" assumes you've done the setup steps in “Requirements and Configuration”.

Requirements and Configuration

It is highly recommended that you first initialize a virtual development environment using a tool such a conda or venv. The following commands create a virtual environment using venv and install the required dependencies for development and testing.

python3 -m venv ./venv
source venv/bin/activate
python3 -m pip install --upgrade pip # upgrade to pip>=18.1
python3 -m pip install -r src/requirements-dev.txt

Using pip-tools

It is recommended to keep direct dependencies in a separate file. The direct dependencies are in the requirements.in and requirements-dev.in files. Then pip-tools can be used to generate files with all the dependencies and transitive dependencies (sub-dependencies). The files containing all the dependencies are in requirements.txt and requirements-dev.txt. Run the following in the virtual environment to install pip-tools.

python -m pip install pip-tools

To update requirements.txt after modifying requirements.in:

pip-compile requirements.in

To update requirements-dev.txt after modifying requirements.in or requirements-dev.in:

pip-compile requirements-dev.in

Use pip-sync to match virtual environment to requirements-dev.txt:

pip-sync requirements.txt requirements-dev.txt

For more information about pip-tools, see https://pip-tools.readthedocs.io/en/latest/#

Running Tests

Ideally, you should add a test that covers any new feature that you add. If you've done that, then running the included test suite is the easiest way to check that everything is working. In any case, all tests should be run after making any local modifications to ensure that you haven't caused a regression.

scos-sensor uses pytest and pytest-django for testing. Tests are organized by application , so tests related to the scheduler are in ./src/scheduler/tests. tox is a tool that can run all available tests in a virtual environment against all supported versions of Python. Running pytest directly is faster, but running tox is a more thorough test.

The following commands install the sensor's development requirements. We highly recommend you initialize a virtual development environment using a tool such a conda or venv first.

cd src
pytest          # faster, but less thorough
tox             # tests code in clean virtualenv
tox --recreate  # if you change `requirements.txt`
tox -e coverage # check where test coverage lacks

Running Docker with Local Changes

The docker-compose file and application code look for information from the environment when run, so it's necessary to source the following file in each shell that you intend to launch the sensor from. (HINT: it can be useful to add the source command to a post-activate file in whatever virtual environment you're using).

cp env.template env     # modify if necessary, defaults are okay for testing
source ./env

Then, build the API docker image locally, which will satisfy the smsntia/scos-sensor and smsntia/autoheal images in the Docker compose file and bring up the sensor.

docker-compose down
docker-compose build
docker-compose up -d
docker-compose logs --follow api

Running Development Server (Not Recommended)

Running the sensor API outside of Docker is possible but not recommended, since Django is being asked to run without several security features it expects. See Common Issues for some hints when running the sensor in this way. The following steps assume you've already set up some kind of virtual environment and installed python dev requirements from Requirements and Configuration.

docker-compose up -d db
cd src
export MOCK_SIGAN=1 MOCK_SIGAN_RANDOM=1 # if running without signal analyzer attached
./manage.py makemigrations
./manage.py migrate
./manage.py createsuperuser
./manage.py runserver

Common Issues

• The development server serves on localhost:8000, not :80
• If you get a Forbidden (403) error, close any tabs and clear any cache and cookies related to SCOS Sensor and try again
• If you're using a virtual environment and your signal analyzer driver is installed outside of it, you may need to allow access to system sitepackages. For example, if you're using a virtualenv called scos-sensor, you can remove the following text file: rm -f ~/.virtualenvs/scos-sensor/lib/python3.7/no-global-site-packages.txt, and thereafter use the ignore-installed flag to pip: pip install -I -r requirements.txt. This should let the devserver fall back to system sitepackages for the signal analyzer driver only.

Committing

Besides running the test suite and ensuring that all tests are passed, we also expect all Python code that's checked in to have been run through an auto-formatter.

This project uses a Python auto-formatter called Black. You probably won't like every decision it makes, but our continuous integration test-runner will reject your commit if it's not properly formatted.

Additionally, import statement sorting is handled by isort.

The continuous integration test-runner verifies the code is auto-formatted by checking that neither isort nor Black would recommend any changes to the code. Occasionally, this can fail if these two autoformatters disagree. The only time I've seen this happen is with a commented-out import statement, which isort parses, and Black treats as a comment. Solution: don't leave commented-out import statements in the code.

There are several ways to autoformat your code before committing. First, IDE integration with on-save hooks is very useful. Second, there is a script, scripts/autoformat_python.sh, that will run both isort and Black over the codebase. Lastly, if you've already pip-installed the dev requirements from the section above, you already have a utility called pre-commit installed that will automate setting up this project's git pre-commit hooks. Simply type the following once, and each time you make a commit, it will be appropriately autoformatted.

pre-commit install

You can manually run the pre-commit hooks using the following command.

pre-commit run --all-files