The undeployed versions of the code are meant for development and debugging purposes. Any branch that is not mapped to one of the deployed environments can be used to create a local development environment (the same can be done with dev
and master
, but since they are not intended for direct modification, we exclude them from the scope of this discussion).
Install Docker desktop software:
Install other CLI dependencies:
- Stringer allows for automated creation of methods that satisfy the
fmt.Stringer
interface. Rungo install golang.org/x/tools/cmd/stringer@latest
- Jade PUG template compiler
go install github.com/quible-io/vendors-jade/cmd/jade@latest
Create .env
file based on content from .env.sample
and edit it to define values of the listed variables:
ENV_JWT_SECRET
passphrase for JWT signing/verificationENV_ABLY_KEY
API key for Ably serviceENV_RAPIDAPI_KEY
API key forBasketAPI
data providerENV_POSTMARK_API_KEY
API key for Postmark email delivery service (server key)WEB_CLIENT_URL
holds the URL of the associated web clientPOSTGRES_USER
DB user to be setup and used for connecting microservices to DBPOSTGRES_PASSWORD
password of the DB user (arbitrary good password)POSTGRES_DB
DB name (arbitrary good name)AUTH_PORT
TCP port to runauth-service
, should not conflict with existing host portsAPP_PORT
TCP port to runapp-service
, should not conflict with existing host portsIS_DEV
when set to1
allows differentiating behavior onprod
anddev
deployments
Database migrations are defined in the cmd
module. They are automatically executed up to the highest version when:
- one of the microservices is started in docker environment (docker hosted DB is targeted/used for migrations)
- one of the microservices is deployed to Render (external DB hosted in Render is targeted/used for migrations)
We use goose
library to perform DB migrations. It supports 2 kinds of migration files:
- pure SQL (for simple migrations)
- Go sources (for complex use cases)
Every migration file satisfies the following conditions
- it is stored in
cmd/migrations
- it is generated by a migration script
cmd/run.sh migrate create <migration name> <migration type>
and should not be edited unless it is a Go migration - it has a filename containing
id
(timestamp) andname
(matching<migration name>
from the above command). Theid
portion is used to keep the track of applied migrations. Thename
portion is ignored and is used for readability
To create/apply/undo/redo migrations use the shell script cmd/run.sh migrate
and feed it with the commands followed by command's arguments (some usage examples can be found in the official goose
documentation):
up
to apply all available migrationsup-to
migrate up to a specific version (timestamp or migrationid
)up-by-one
migrate up a single migration from the current versiondown
roll back a single migration from the current versiondown-to
roll back migrations to a specific version (use0
as a pseudo version number, corresponding to the initial DB state)redo
roll back the most recently applied migration, then run it againstatus
print the status of all migrationsversion
print the current version of the database:
Important: the migration utility uses environment variable ENV_DSN
to locate the DB instance and connect to it. This variable should either be "exported" explicitly or otherwise its value will be implied from .env
file located in the repo's root.
A special docker service migrations
is defined to perform all pending DB migrations every time when it is executed explicitly:
docker-compose up migrations --build
or one of the microservices is started in docker environment (see below).
Let's say we have a DB with 2 applied migrations. To see those applied migrations we need
- Run DB service in docker environment:
docker-compose up db
- Run the following command (assuming your current directory matches the repo root)
cmd/run.sh migrate status
- It should print something similar to
from where we see that 2 migrations have been previously applied and no other migrations are pending
2023/12/04 18:04:54 Applied At Migration 2023/12/04 18:04:54 ======================================= 2023/12/04 18:04:54 Tue Dec 5 00:20:11 2023 -- 20231204150816_init.sql 2023/12/04 18:04:54 Tue Dec 5 00:20:11 2023 -- 20231204164132_add_image_column_to_users.sql
Having initial state of the DB confirmed, we need to keep DB server running in the container and run the following command to create a placeholder for a new SQL migration
cmd/run.sh migrate create mytest sql
This command creates a new file named xxx_mytest.sql
in cmd/migrations
. We can re-run the status
command to see it in pending status
cmd/run.sh migrate status
2023/12/04 18:12:18 Applied At Migration
2023/12/04 18:12:18 =======================================
2023/12/04 18:12:18 Tue Dec 5 00:20:11 2023 -- 20231204150816_init.sql
2023/12/04 18:12:18 Tue Dec 5 00:20:11 2023 -- 20231204164132_add_image_column_to_users.sql
2023/12/04 18:12:18 Pending -- 20231204181114_mytest.sql
The migration file usually needs to be edited to implement the migration logic, but even without editing it can be applied and rolled back as follows:
To apply run:
cmd/run.sh migrate up
2023/12/04 18:14:34 OK 20231204181114_mytest.sql (28.74ms)
2023/12/04 18:14:34 goose: successfully migrated database to version: 20231204181114
To roll back run
cmd/run.sh migrate down
2023/12/04 18:15:12 OK 20231204181114_mytest.sql (9.43ms)
Then to confirm the original state run
cmd/run.sh migrate status
2023/12/04 18:16:06 Applied At Migration
2023/12/04 18:16:06 =======================================
2023/12/04 18:16:06 Tue Dec 5 00:20:11 2023 -- 20231204150816_init.sql
2023/12/04 18:16:06 Tue Dec 5 00:20:11 2023 -- 20231204164132_add_image_column_to_users.sql
2023/12/04 18:16:06 Pending -- 20231204181114_mytest.sql
which shows that our newly introduced migration is in pending state (since we rolled it back after applying)
Run docker-compose up --build
to build and start all microservices. The operation will run in the foreground and can be gracefully terminated by hitting Ctrl+C
. All services will send their logs to the same console. The service prefix can help identify specific service logs.
Below is the log of an exampled run:
...
[+] Running 4/4
✔ Container quible-api-db-1 Created
✔ Container quible-api-migrations-1 Recreated
✔ Container quible-api-auth-1 Recreated
✔ Container quible-api-app-1 Recreated
Attaching to quible-api-app-1, quible-api-auth-1, quible-api-db-1, quible-api-migrations-1
quible-api-db-1 |
quible-api-db-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
quible-api-db-1 |
quible-api-db-1 | 2023-12-05 01:25:40.458 UTC [1] LOG: starting PostgreSQL 16.1 (Debian 16.1-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
quible-api-db-1 | 2023-12-05 01:25:40.458 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
quible-api-db-1 | 2023-12-05 01:25:40.459 UTC [1] LOG: listening on IPv6 address "::", port 5432
quible-api-db-1 | 2023-12-05 01:25:40.464 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
quible-api-db-1 | 2023-12-05 01:25:40.503 UTC [31] LOG: database system was shut down at 2023-12-05 01:25:28 UTC
quible-api-db-1 | 2023-12-05 01:25:40.557 UTC [1] LOG: database system is ready to accept connections
quible-api-migrations-1 | 2023/12/05 01:26:51 goose: no migrations to run. current version: 20231204164132
quible-api-migrations-1 exited with code 0
quible-api-auth-1 | 2023/12/05 01:26:51 running in docker...
quible-api-auth-1 | 2023/12/05 01:26:52 starting server on port: 8001
quible-api-app-1 | 2023/12/05 01:26:52 running in docker...
quible-api-app-1 | 2023/12/05 01:26:52 starting server on port: 8002
with quible-api-db-1
, quible-api-auth-1
, quible-api-app-1
and quible-api-migrations-1
being log prefixes of individual services.
There are two ways to configure the composed Docker environment regarding which Dockerfile
will be picked up by individual microservices.
- Shared
Dockerfile
from the repo root - Microservice-specific
Dockerfile
hosted at the root level of the microservice directory.
The former option (shared config) can be activated by defining the build
section of the specific microservice in the docker-compose.yml
as follows (see auth
service for example):
build:
context: ./auth-service
dockerfile: ../Dockerfile
...
Here, the dockerfile: ../Dockerfile
references Dockerfile
from the repo root.
Alternatively, especially when the microservice is not compatible with the shared Dockerfile
, the build section can be defined as follows (see migrations
service for example):
build:
context: ./cmd
dockerfile: ./Dockerfile
...
where dockerfile: ./Dockerfile
references Dockerfile
file inside microservice directory.
Running all microservices in Docker containers proves their inter-operational compatibility. But there are situations when local development is focused on aspects of one (or only a few) microservice(s), and it can be safely localized without affecting compatibility. In such cases, the overhead of re-building and re-running the docker containers could seriously degrade the development pace.
The only service that is meant to continue running in Docker is DB. You can start it separately (without modifying any files) by running:
docker-compose up db -d
Once the DB service is started in a Docker container, it exposes the TCP:5432
port on the localhost
interface. If you now start a microservice outside the Docker environment, it will still be able to connect to DB to access data. Under the hood, it will read .env
file from the root of the repo and will compile the value of ENV_DSN
environment variable from the variables exposed in that file (see implementation of this logic in lib/env/setup.go
)
Without loss of generality, start the service by changing into its root directory running the default source as shown below
cd auth-service
go run .