Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve README #86

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

improve README #86

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
186 changes: 127 additions & 59 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,51 @@
# `pg_upgrade`, Docker style

This is a PoC for using `pg_upgrade` inside Docker -- learn from it, adapt it for your needs; don't expect it to work as-is!
This is a PoC for using `pg_upgrade` inside Docker to upgrade from an old PostgreSQL data directory to a new one -- learn from it, adapt it for your needs; don't expect it to work as-is!

(Source for this image is available at [https://github.com/tianon/docker-postgres-upgrade](https://github.com/tianon/docker-postgres-upgrade).)

Tags of this image are of the format `OLD-to-NEW`, where `OLD` represents the version of PostgreSQL you are _currently_ running, and `NEW` represents the version of PostgreSQL you would like to upgrade to.

In order to get good performance, it is recommended to run this image with `docker run image --link` (see [`pg_upgrade` documentation](https://www.postgresql.org/docs/9.5/static/pgupgrade.html) for more details).
In order to get good performance, it is recommended to run this Docker image with the `--link` option (see [`pg_upgrade` documentation](https://www.postgresql.org/docs/9.5/static/pgupgrade.html) for more details, and the below sections for examples).

For this to be feasible, your directory structure should look something like this: (if yours does not, either adjust it, or scroll down to see the alternate pattern for running this image)
## Prerequisites

For a migration, you only need to have Docker installed, know where your current Postgres data lives, and where you want to create the new data directory.

> [!WARNING]
> Always create a backup of the data before attempting a migration so you can restore in the event of data loss.

## How does it work?

This image runs `pg_upgrade` to migrate the data from an old to a newer version of Postgres. Note that it does not use your existing Postgres Docker image that runs the actual database.

For the upgrade procedure, you have to mount two directories into the container:

* The old PostgreSQL data directory, from which you want to upgrade
* The new (empty) PostgreSQL data directory, to which the upgraded data will be written (by this image)

There are two ways your existing data may be laid out on your file system already:

1. A directory with subdirectories for each Postgres version (e.g. `/mnt/bigdrive/postgresql/9.4` and `/mnt/bigdrive/postgresql/9.5`).
2. A single directory containing just the Postgres data (e.g. `/mnt/bigdrive/postgresql/`)

The examples in this README are always Docker bind mounts, but the same applies to Docker named volumes.

Depending on which method you use, follow any of the below sections.

### Subdirectories for each Postgres version

For this to be feasible, your directory structure should look something like this:

```console
$ find DIR -mindepth 2 -maxdepth 2
DIR/OLD/data
DIR/NEW/data

$ docker run --rm \
-v DIR:/var/lib/postgresql \
tianon/postgres-upgrade:OLD-to-NEW \
--link
-v DIR:/var/lib/postgresql \
tianon/postgres-upgrade:OLD-to-NEW \
--link

...
```
Expand All @@ -31,24 +58,26 @@ $ find /mnt/bigdrive/postgresql -mindepth 2 -maxdepth 2
/mnt/bigdrive/postgresql/9.5/data

$ docker run --rm \
-v /mnt/bigdrive/postgresql:/var/lib/postgresql \
tianon/postgres-upgrade:9.4-to-9.5 \
--link
-v /mnt/bigdrive/postgresql:/var/lib/postgresql \
tianon/postgres-upgrade:9.4-to-9.5 \
--link

...
```

(which assumes that your previous `postgres:9.4` instance was running with `-v /mnt/bigdrive/postgresql/9.4/data:/var/lib/postgresql/data`, and your new `postgres:9.5` instance will run with `-v /mnt/bigdrive/postgresql/9.5/data:/var/lib/postgresql/data`)
This assumes that your previous `postgres:9.4` Docker container was running with `-v /mnt/bigdrive/postgresql/9.4/data:/var/lib/postgresql/data`, and your new `postgres:9.5` container will run with `-v /mnt/bigdrive/postgresql/9.5/data:/var/lib/postgresql/data`.

Note that this uses `--link` and will be faster than the single directory method below.

---
### Single directory with Postgres data

If your two directories (denoted below as `PGDATAOLD` and `PGDATANEW`) do not follow this structure, then the following may also be used (but will be slower):
If you previously only had a single directory for Postgres data (denoted below as `PGDATAOLD`), you must create a new one and (`PGDATANEW`). This will, however, be slower:

```console
$ docker run --rm \
-v PGDATAOLD:/var/lib/postgresql/OLD/data \
-v PGDATANEW:/var/lib/postgresql/NEW/data \
tianon/postgres-upgrade:OLD-to-NEW
-v PGDATAOLD:/var/lib/postgresql/OLD/data \
-v PGDATANEW:/var/lib/postgresql/NEW/data \
tianon/postgres-upgrade:OLD-to-NEW

...
```
Expand All @@ -57,57 +86,96 @@ More concretely, assuming `OLD` of `9.4`, `NEW` of `9.5`, `PGDATAOLD` of `/mnt/b

```console
$ docker run --rm \
-v /mnt/bigdrive/postgresql-9.4:/var/lib/postgresql/9.4/data \
-v /mnt/bigdrive/postgresql-9.5:/var/lib/postgresql/9.5/data \
tianon/postgres-upgrade:9.4-to-9.5
-v /mnt/bigdrive/postgresql-9.4:/var/lib/postgresql/9.4/data \
-v /mnt/bigdrive/postgresql-9.5:/var/lib/postgresql/9.5/data \
tianon/postgres-upgrade:9.4-to-9.5

...
```

(which assumes that your previous `postgres:9.4` instance was running with `-v /mnt/bigdrive/postgresql-9.4:/var/lib/postgresql/data`, and your new `postgres:9.5` instance will run with `-v /mnt/bigdrive/postgresql-9.5:/var/lib/postgresql/data`)
This assumes that your previous `postgres:9.4` container was running with `-v /mnt/bigdrive/postgresql-9.4:/var/lib/postgresql/data`, and your new `postgres:9.5` container will run with `-v /mnt/bigdrive/postgresql-9.5:/var/lib/postgresql/data`.

---
### Usage with Docker Named Volumes

Putting it all together:
If your Postgres data is stored in Docker named volumes, you can use the same commands as above. Just replace the directory paths with the names of your named volumes. For example, if you have a named volume called `pgdata-9.4` and another called `pgdata-9.5`, you can use the following commands:

```console
$ mkdir -p postgres-upgrade-testing
$ cd postgres-upgrade-testing
$ OLD='9.4'
$ NEW='9.5'

$ docker pull "postgres:$OLD"
$ docker run -dit \
--name postgres-upgrade-testing \
-e POSTGRES_PASSWORD=password \
-v "$PWD/$OLD/data":/var/lib/postgresql/data \
"postgres:$OLD"
$ sleep 5
$ docker logs --tail 100 postgres-upgrade-testing

$ # let's get some testing data in there
$ docker exec -it \
-u postgres \
postgres-upgrade-testing \
pgbench -i -s 10

$ docker stop postgres-upgrade-testing
$ docker rm postgres-upgrade-testing

$ docker run --rm \
-v "$PWD":/var/lib/postgresql \
"tianon/postgres-upgrade:$OLD-to-$NEW" \
--link

$ docker pull "postgres:$NEW"
$ docker run -dit \
--name postgres-upgrade-testing \
-e POSTGRES_PASSWORD=password \
-v "$PWD/$NEW/data":/var/lib/postgresql/data \
"postgres:$NEW"
$ sleep 5
$ docker logs --tail 100 postgres-upgrade-testing

$ # can now safely remove "$OLD"
$ sudo rm -rf "$OLD"
-v pgdata-9.4:/var/lib/postgresql/OLD/data \
-v pgdata-9.5:/var/lib/postgresql/NEW/data \
tianon/postgres-upgrade:OLD-to-NEW
```

## Example

Putting it all together, you can use this image as follows, assuming you want to upgrade from Postgres 9.4 to 9.5.

We first create a directory for our test and set the old and new versions as variables in our current shell session:

```bash
mkdir -p postgres-upgrade-testing
cd postgres-upgrade-testing
OLD='9.4'
NEW='9.5'
```

Then, we pull the old Postgres image (from which we want to migrate) and create a container for the old version named `postgres-upgrade-testing`:

```bash
docker pull "postgres:$OLD"
docker run -dit \
--name postgres-upgrade-testing \
-e POSTGRES_PASSWORD=password \
-v "$PWD/$OLD/data":/var/lib/postgresql/data \
"postgres:$OLD"
docker logs -f postgres-upgrade-testing
```

We should now see some logs indicating that the database is ready to accept connections. When that is the case, press `Ctrl-C` to exit the logs.

Let's create some test data:

```bash
docker exec -it \
-u postgres \
postgres-upgrade-testing \
pgbench -i -s 10
```

Now, we can stop and remove the Postgres container – your data still lives in `$OLD/data`.

```bash
docker stop postgres-upgrade-testing
docker rm postgres-upgrade-testing
```

We begin the migration by pulling the image from this repository and running the migration container:

```bash
docker run --rm \
-v "$PWD":/var/lib/postgresql \
"tianon/postgres-upgrade:$OLD-to-$NEW" \
--link
```

When this succeeded, we can start the new Postgres version in a container and check that the data is still there:

```bash
docker pull "postgres:$NEW"
docker run -dit \
--name postgres-upgrade-testing \
-e POSTGRES_PASSWORD=password \
-v "$PWD/$NEW/data":/var/lib/postgresql/data \
"postgres:$NEW"
docker logs -f postgres-upgrade-testing
```

This should show the same logs as before. `Ctrl-C` to exit the logs.

Now you can safely remove `$OLD`:

```bash
sudo rm -rf "$OLD"
```

That's it!