From 4e83d6a56df689b829f791560f2265ba3b61adeb Mon Sep 17 00:00:00 2001 From: Stuart MacKay Date: Sun, 8 Oct 2023 11:29:32 +0100 Subject: [PATCH] Use sensible defaults so everything runs out of the box Change the Django settings so the demo site runs without having to define any environment variables. Similarly change the docker-compose.yml file to provide sensible defaults so the images are built and the containers run without any additional configuration. The .env and .env.docker files are replaced by .env.example which contains all the environment variables used in the project. Now you only need to create .env when you want to change the way the project is configured, for example, running the database in a container when you also have PostgreSQL installed natively. As a result the getting started guide is simplified with the steps for adding PostgreSQL and RabbitMQ accounts are now options and are moved to the end of the guide. --- .env | 39 --------- .env.docker | 31 ------- .env.example | 86 +++++++++++++++++++ .gitignore | 3 +- demo/settings.py | 19 +++-- docker-compose.yml | 52 +++++++----- docs/howtos/get-started-with-development.md | 93 +++++++++++++-------- 7 files changed, 188 insertions(+), 135 deletions(-) delete mode 100644 .env delete mode 100644 .env.docker create mode 100644 .env.example diff --git a/.env b/.env deleted file mode 100644 index 9b2fe5c..0000000 --- a/.env +++ /dev/null @@ -1,39 +0,0 @@ -# -# Environment variables used for development using a virtualenv -# - -# This assumes you've installed postgresql and rabbitmq on your -# machine and have added the accounts need to connect to them: -# -# sudo -u postgres psql -# postgres=# create database feeds; -# postgres=# create user feeds with createdb encrypted password 'feeds'; -# postgres=# grant all privileges on database feeds to feeds; -# -# sudo rabbitmqctl add_user feeds feeds -# sudo rabbitmqctl add_vhost feeds -# sudo rabbitmqctl set_permissions -p feeds feeds ".*" ".*" ".*" - -# Set environment variables used by the postgresql commands to connect -# to the database. This is the admin account not the account used by -# Django to connect to the database. PGPASSWORD is particularly useful -# to avoid having to enter it for each command. You will need to configure -# the server to allow username/password (md5) authentication on local -# (socket) connections and so avoid having to su to the postgres user first. -# -# /etc/postgresql//main/pg_hba.conf -# local all all md5 -# -# For more info on environment variables see, -# https://www.postgresql.org/docs/current/libpq-envars.html - -PGHOST=localhost -PGPORT=5432 -PGUSER=postgres -PGPASSWORD=postgres - -# Environment variables used by Django - -CELERY_BROKER_URL=amqp://feeds:feeds@localhost:5672/feeds - -DB_HOST=localhost diff --git a/.env.docker b/.env.docker deleted file mode 100644 index 83637ca..0000000 --- a/.env.docker +++ /dev/null @@ -1,31 +0,0 @@ -# -# Environment variables used for running the demo site using docker -# - -# Service: db -# Rather than set POSTGRES_PASSWORD (required) and leave POSTGRES_USER and -# so POSTGRES_DB to default to "postgres" we set the variables to the name -# of the app so it mirrors the configuration for locally installed services -# which might be shared between sites. However, the value of doing this is -# that you easily initialize the database with dumps from production where -# the ownership of the tables is defined. - -PGHOST=db -POSTGRES_DB=feeds -POSTGRES_USER=feeds -POSTGRES_PASSWORD=feeds - -# Service: broker -# This, like the db service, simply mirrors the configuration for rabbitmq -# being installed locally and possibly shared between sites. - -RABBITMQ_DEFAULT_USER=feeds -RABBITMQ_DEFAULT_PASS=feeds -RABBITMQ_DEFAULT_VHOST=feeds - -# Services: web, celery, celery-beat -# Environment variables used in Django settings - -CELERY_BROKER_URL=amqp://feeds:feeds@broker:5672/feeds - -DB_HOST=db diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..2231f39 --- /dev/null +++ b/.env.example @@ -0,0 +1,86 @@ +# +# Environment variables used for running the demo site +# +# Django settings and docker-compose.yml are configured with sensible default +# so whether you run the demo with a virtualenv or containers everything works +# out of the box. You will only need to set environment variables if you want +# to change the configuration in some way. For example, you have postgres +# installed natively but you want to use a container instead. In this case you +# need to set DOCKER_POSTGRES_PORT_FORWARD so the port does not clash with the +# post used by the native postgres. + +# Configuration for PostgreSQL + +# Both POSTGRES_USER and POSTGRES_PASSWORD are required when building the +# postgres image. Both are set to use the default value of 'postgres' in +# docker-compose.yml. Override them here, along with any other configuration +# changes you want. + +#POSTGRES_USER=postgres +#POSTGRES_PASSWORD=postgres +#POSTGRES_DB=postgres + +# This is the external port visible locally. POSTGRES_PORT is the port in +# the docker virtual network. + +#DOCKER_POSTGRES_PORT_FORWARD=5432 + +# Configuration for RabbitMQ + +#RABBITMQ_DEFAULT_USER=guest +#RABBITMQ_DEFAULT_PASS=guest +#RABBITMQ_DEFAULT_VHOST= + +# Ports on the local network so the rabbitmq service and flower webserver +# are accessible. + +#DOCKER_RABBITMQ_PORT_FORWARD=5672 +#DOCKER_FLOWER_PORT_FORWARD=5555 + +# Configuration for Celery + +#CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672/ + +# Django-feeds settings + +# Crontab setting the schedule on which the Celery task, load_feeds, that +# checks for Feeds to be loaded, runs. + +#FEEDS_TASK_SCHEDULE=0 * * * * + +# Crontab setting the default schedule a Feed is loaded on. This schedule +# can be set on Feeds individually. This schedule must coincide with the +# FEEDS_TASK_SCHEDULE otherwise no feed will be loaded. The Celery task +# runs every hour on the hour however you can set the default schedule to +# every hour on the hour between 8am and 8pm for example. That way you +# can limit fetches to the times that new posts are more likely to be +# published. + +#FEEDS_LOAD_SCHEDULE=0 * * * * + +# Extra. Accessing postgres on the command line. +# +# This is useful if you want to define Makefile targets for managing the +# database. +# +# Set environment variables used by the postgresql commands, psql, createdb, +# dropdb, etc. to connect to the database. PGPASSWORD is particularly useful +# to avoid having to enter it for each command. You will need to configure +# the server to allow username/password (md5) authentication on local (socket) +# connections and so avoid having to su to the postgres user first. +# +# /etc/postgresql//main/pg_hba.conf +# local all all md5 +# +# For more info on environment variables see, +# https://www.postgresql.org/docs/current/libpq-envars.html +# +# The values must match the variables defined above, obviously. + +#PGUSER=postgres +#PGPASSWORD=postgres +#PGDATABASE=postgres + +# PGPORT must match DOCKER_POSTGRES_PORT_FORWARD + +#PGPORT=5432 diff --git a/.gitignore b/.gitignore index 97afc28..bc13181 100644 --- a/.gitignore +++ b/.gitignore @@ -11,8 +11,7 @@ __pycache__/ !.dockerignore !.editorconfig !.envrc -!.env -!.env.docker +!.env.example !.flake8 !.gitattributes !.github diff --git a/demo/settings.py b/demo/settings.py index 0033b65..1d8e65a 100644 --- a/demo/settings.py +++ b/demo/settings.py @@ -1,6 +1,11 @@ """ Demo site settings +The database and celery settings assume everything is running +on localhost. When using a virtualenv, no environment variables +need to be set to run the demo site. When running the site using +docker the variables are defined in the docker-compose file. + """ import logging import os @@ -58,11 +63,11 @@ DATABASES = { "default": { "ENGINE": "django.db.backends.postgresql_psycopg2", - "NAME": "feeds", - "USER": "feeds", - "PASSWORD": "feeds", - "HOST": os.environ["DB_HOST"], - "PORT": 5432, + "NAME": os.environ.get("POSTGRES_DB", "postgres"), + "USER": os.environ.get("POSTGRES_USER", "postgres"), + "PASSWORD": os.environ.get("POSTGRES_PASS", "postgres"), + "HOST": os.environ.get("POSTGRES_HOST", "localhost"), + "PORT": os.environ.get("POSTGRES_PORT", 5432), } } @@ -74,7 +79,9 @@ USE_TZ = True -CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL", "") +CELERY_BROKER_URL = os.environ.get( + "CELERY_BROKER_URL", "amqp://guest:guest@localhost:5672/" +) CELERY_TASK_ALWAYS_EAGER = CELERY_BROKER_URL == "" diff --git a/docker-compose.yml b/docker-compose.yml index 9cb6b9d..d1e65ad 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,19 +1,23 @@ version: '3.1' services: - db: + postgres: image: postgres:15 ports: - - "15432:5432" + - "${DOCKER_POSTGRES_PORT_FORWARD-5432}:5432" volumes: - - ${DB_DATA_DIR-./.data/db}:/var/lib/postgresql/data - env_file: - - .env.docker + - ${PGDATA-./.data/db}:/var/lib/postgresql/data + environment: + POSTGRES_USER: "${POSTGRES_USER-postgres}" + POSTGRES_PASSWORD: "${POSTGRES_PASSWORD-postgres}" + POSTGRES_DB: "${POSTGRES_DB-postgres}" - broker: + rabbitmq: image: rabbitmq:3.11.9-management - env_file: - - .env.docker + environment: + RABBITMQ_DEFAULT_USER: "${RABBITMQ_DEFAULT_USER-guest}" + RABBITMQ_DEFAULT_PASS: "${RABBITMQ_DEFAULT_PASS-guest}" + RABBITMQ_DEFAULT_VHOST: "${RABBITMQ_DEFAULT_VHOST}" web: stdin_open: true @@ -23,35 +27,37 @@ services: volumes: &web-volumes - ./:/code ports: - - "8000:8000" + - "${DOCKER_WEB_PORT_FORWARD-8000}:8000" depends_on: &web-depends-on - - db - - broker - env_file: - - .env.docker + - postgres + - rabbitmq + environment: &web-environment + POSTGRES_USER: "${POSTGRES_USER-postgres}" + POSTGRES_PASSWORD: "${POSTGRES_PASSWORD-postgres}" + POSTGRES_DB: "${POSTGRES_DB-postgres}" + POSTGRES_HOST: "${POSTGRES_HOST-postgres}" + POSTGRES_PORT: "${POSTGRES_PORT-5432}" + CELERY_BROKER_URL: "${CELERY_BROKER_URL-amqp://guest:guest@rabbitmq:5672/}" flower: image: mher/flower ports: - - "15555:5555" + - "${DOCKER_FLOWER_PORT_FORWARD-5555}:5555" command: celery flower --broker_api=http://feeds:feeds@broker:15672/api/feeds depends_on: - - broker - env_file: - - .env.docker + - rabbitmq + environment: *web-environment celery-beat: build: . - command: celery --app feeds beat --loglevel debug + command: celery --app demo beat --loglevel debug volumes: *web-volumes depends_on: *web-depends-on - env_file: - - .env.docker + environment: *web-environment celery-worker: build: . - command: celery --app feeds worker --loglevel debug + command: celery --app demo worker --loglevel debug volumes: *web-volumes depends_on: *web-depends-on - env_file: - - .env.docker + environment: *web-environment diff --git a/docs/howtos/get-started-with-development.md b/docs/howtos/get-started-with-development.md index d4d29ce..4d9df72 100644 --- a/docs/howtos/get-started-with-development.md +++ b/docs/howtos/get-started-with-development.md @@ -7,33 +7,6 @@ The assumption is that you are working in a Linux environment with PostgreSQL and RabbitMQ installed natively. If you would rather not install these, then see below, where various options are listed. -If you encounter any problems, please check the Troubleshooting section below, -for a solution. - -Create a database account: - -```shell -sudo -u postgres psql -``` - -```shell -postgres=# create database feeds; -postgres=# create user feeds with createdb encrypted password 'feeds'; -postgres=# grant all privileges on database feeds to feeds; -postgres=# \q -``` - -If you want to use another account, be sure to change the DATABASES setting -in `demo/settings.py` to match. - -Create a message broker account: - -```shell -sudo rabbitmqctl add_user feeds feeds -sudo rabbitmqctl add_vhost feeds -sudo rabbitmqctl set_permissions -p feeds feeds ".*" ".*" ".*" -``` - Checkout the repository: ```shell @@ -55,12 +28,6 @@ pip install pip-tools pip-sync requirements/dev.txt ``` -Set environment variables used by Django: - -```shell -source .env -``` - Run the tests: ```shell @@ -76,6 +43,13 @@ python manage.py createsuperuser python manage.py runserver ``` +## Setting environment variables + +Whether you create a virtualenv or use docker the demo site runs out of the +box. If you need to change the configuration the `env.example` file contains +a complete list of variables that are used to configure the docker containers +or are used for Django settings. + ## Using direnv If you install [direnv](https://direnv.net/), then whenever you cd to the project @@ -122,7 +96,7 @@ all the components: db, celery, web, etc. That way you can use a mix of services running natively or in containers: ```shell -docker-compose up db broker +docker-compose up postgres rabbitmq ``` If you use an IDE that supports docker then just run everything in containers: @@ -130,3 +104,54 @@ If you use an IDE that supports docker then just run everything in containers: ```shell docker-compose up ``` + +## Using a different database + +The configuration uses the default `postgres` database. If you want to keep +the databases for different projects separate then run the following commands: + +```shell +sudo -u postgres psql +``` + +```shell +postgres=# create database feeds; +postgres=# create user feeds with createdb encrypted password 'feeds'; +postgres=# grant all privileges on database feeds to feeds; +postgres=# \q +``` + +There's no need to create another user, you could still use the default +`postgres` account. You will also need to change the following environment +variables in an `.env` file: + +```shell +POSTGRES_USER=feeds +POSTGRES_PASSWORD=feeds +POSTGRES_DB=feeds +``` + +If you use postgreSQL's command line tools for managing the database then +remember to also change the following: + +```shell +PGDATABASE=feeds +``` + +## Using a different RabbitMQ account and vhost + +For the same reasons for keeping the project database separate you may +also want to create a separate account and vhost for RabbitMQ. + +```shell +sudo rabbitmqctl add_user feeds feeds +sudo rabbitmqctl add_vhost feeds +sudo rabbitmqctl set_permissions -p feeds feeds ".*" ".*" ".*" +``` + +You will need to set the environment variables in `.env` so celery +can connect to RabbitMQ: + +```shell +CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672/ +```