Development

This page contains helpful information for developers wishing to contribute to SCOS Sensor.

The following techniques can be used to make and test local modifications to SCOS Sensor. Sections are in order, so Running Tests assumes you've done the setup steps in Requirements and Setup.

Requirements and Setup

It is highly recommended that you first initialize a virtual development environment using a tool such a conda or venv. The following commands create a virtual environment using venv and install the required dependencies for development and testing.

python3 -m venv ./venv
source venv/bin/activate
python3 -m pip install --upgrade pip # upgrade to pip>=18.1
python3 -m pip install -r src/requirements-dev.txt

Using pip-tools

Direct dependencies of SCOS Sensor are declared in src/requirements.in and src/requirements-dev.in. From these files, pip-tools can be used to compile files (src/requirements.txt and src/requirements-dev.txt) with all the dependencies and transitive dependencies (sub-dependencies) pinned to specific versions. Run the following in the virtual environment to install pip-tools:

python -m pip install pip-tools

To update requirements.txt after modifying requirements.in:

pip-compile requirements.in

To update requirements-dev.txt after modifying either requirements.in or requirements-dev.in:

pip-compile requirements-dev.in

Use pip-sync to match your virtual environment to requirements-dev.txt:

pip-sync requirements.txt requirements-dev.txt

Running Tests

Ideally, you should add a unit test that covers any new feature that you add. If you've done that, then running the included test suite is the easiest way to check that everything is working. In any case, all tests should be run after making any local modifications to ensure that you haven't caused a regression.

SCOS Sensor uses pytest and pytest-django for testing. Tests are organized by application, so tests related to the scheduler are in src/scheduler/tests. tox is a tool that can run all available tests in a virtual environment against all supported versions of Python. Running pytest directly is faster, but running tox is a more thorough test.

The following commands install the sensor's development requirements. We highly recommend you initialize a virtual development environment using a tool such as conda or venv first.

cd src
pytest          # faster, but less thorough
tox             # tests code in clean virtualenv
tox --recreate  # if you change `requirements.txt`
tox -e coverage # check where test coverage lacks

The full tox test suite is run using GitHub Actions when pull requests are made. It is recommended to run tests yourself before opening a pull request to help identify and resolve errors as early as possible.

Running Docker with Local Changes

The Compose file and application code look for information from the environment when run, so it's necessary to source the following file in each shell that you intend to launch the sensor from. (HINT: it can be useful to add the source command to a post-activate file in whatever virtual environment you're using).

cp env.template env     # modify if necessary, defaults are okay for testing
source ./env

Then, build the API docker image locally, which will satisfy the smsntia/scos-sensor and smsntia/autoheal images in the Compose file and bring up the sensor.

docker-compose down
docker-compose build
docker-compose up -d
docker-compose logs --follow api

Running Development Server (Not Recommended)

Running the SCOS Sensor API outside of Docker is possible but not recommended, since Django is being asked to run without several security features it expects. See Common Issues for some hints when running the sensor in this way. The following steps assume you've already set up some kind of virtual environment and installed development requirements from Requirements and Setup.

docker-compose up -d db
cd src
export MOCK_SIGAN=1 MOCK_SIGAN_RANDOM=1 # if running without signal analyzer attached
./manage.py makemigrations
./manage.py migrate
./manage.py createsuperuser
./manage.py runserver

Common Issues

• The development server serves on localhost:8000, not :80
• If you get a Forbidden (403) error, close any tabs and clear any cache and cookies related to SCOS Sensor and try again
• If you're using a virtual environment and your signal analyzer driver is installed outside of it, you may need to allow access to system sitepackages. For example, if you're using a virtualenv called scos-sensor, you can remove the following text file: rm -f ~/.virtualenvs/scos-sensor/lib/python3.7/no-global-site-packages.txt, and thereafter use the ignore-installed flag to pip: pip install -I -r requirements.txt. This should let the devserver fall back to system sitepackages for the signal analyzer driver only.

Committing

Besides running the test suite and ensuring that all tests are passed, we also expect all Python code that's checked in to have been run through an auto-formatter.

This project uses a Python auto-formatter called Black. You probably won't like every decision it makes, but our continuous integration test-runner will reject your commit if it's not properly formatted. TODO: this currently isn't true, but should be!

Additionally, import statement sorting is handled by isort.

The continuous integration test-runner verifies the code is auto-formatted by checking that neither isort nor Black would recommend any changes to the code. Occasionally, this can fail if these two autoformatters disagree. The only time I've seen this happen is with a commented-out import statement, which isort parses, and Black treats as a comment. Solution: don't leave commented-out import statements in the code.

There are several ways to autoformat your code before committing. First, IDE integration with on-save hooks is very useful. Second, there is a script, scripts/autoformat_python.sh, that will run both isort and Black over the codebase. Lastly, if you've already pip-installed the dev requirements from the section above, you already have a utility called pre-commit installed that will automate setting up this project's git pre-commit hooks. Simply type the following once, and each time you make a commit, it will be appropriately autoformatted.

pre-commit install

You can manually run the pre-commit hooks using the following command.

pre-commit run --all-files