diff --git a/Testing.md b/Testing.md index 4cb2d0f..8647fbf 100644 --- a/Testing.md +++ b/Testing.md @@ -1,12 +1,12 @@ ## Testing Backup and Restore -To test the project we need to simulate running a live system , taking it down and restoring the system. For this test we have two Docker Compose files for the system `docker-compose-live.yaml` and `docker-compose-restore-test.yaml` in a real restore scenario we will only have on Docker Compose file as we expect the restore is being done on a separate instance. We also have separate files for Backup and Restore (`docker-compose-backup-service.yaml` and `docker-compose-restore-service.yaml`) +To test the project we need to simulate running a live system , taking it down and restoring the system. For this test we have two Docker Compose files for the system `docker-compose-live.yaml` and `docker-compose-live-two.yaml` in a real restore scenario we will only have on Docker Compose file as we expect the restore is being done on a separate instance. We also have separate files for Backup and Restore (`docker-compose-backup-service.yaml` and `docker-compose-restore-service.yaml`) Run: -```./start-live-system.sh``` +```./start-live.sh``` This will start the example live system; Wait for > 5 minutes for the first backup to happen. You can confirm by running -`docker logs -f live-system-backup-1` +`docker logs -f live-backup-1` You should see logs similar to: @@ -14,7 +14,7 @@ You should see logs similar to: crond 4.5 dillon's cron daemon, started with loglevel notice 2024-10-21 03:50:01,338 - INFO: Starting backup container 2024-10-21 03:50:01,647 - INFO: Backup process container: compassionate_ganguly -2024-10-21 03:50:01,848 - INFO: 2024-10-21 03:50:01,848 - INFO: Status for compose project 'live-system' +2024-10-21 03:50:01,848 - INFO: 2024-10-21 03:50:01,848 - INFO: Status for compose project 'live' 2024-10-21 03:50:01,848 - INFO: 2024-10-21 03:50:01,848 - INFO: Repository: '/restic_data' 2024-10-21 03:50:01,849 - INFO: 2024-10-21 03:50:01,848 - INFO: Backup currently running?: True 2024-10-21 03:50:01,849 - INFO: 2024-10-21 03:50:01,848 - INFO: Include project name in backup path?: False @@ -28,28 +28,28 @@ crond 4.5 dillon's cron daemon, started with loglevel notice 2024-10-21 03:50:03,933 - INFO: 2024-10-21 03:50:03,933 - INFO: Successfully initialized repository: /restic_data 2024-10-21 03:50:03,933 - INFO: 2024-10-21 03:50:03,933 - INFO: ------------------------- Detected Config ------------------------- 2024-10-21 03:50:03,934 - INFO: 2024-10-21 03:50:03,933 - INFO: service: postgres -2024-10-21 03:50:03,934 - INFO: 2024-10-21 03:50:03,934 - INFO: Container NAME live-system-postgres-1 -2024-10-21 03:50:03,940 - INFO: 2024-10-21 03:50:03,939 - INFO: Container NAME live-system-postgres-1 +2024-10-21 03:50:03,934 - INFO: 2024-10-21 03:50:03,934 - INFO: Container NAME live-postgres-1 +2024-10-21 03:50:03,940 - INFO: 2024-10-21 03:50:03,939 - INFO: Container NAME live-postgres-1 2024-10-21 03:50:03,940 - INFO: 2024-10-21 03:50:03,940 - INFO: - postgres (is_ready=True) -> /backup/databases/postgres/test-postgres.sql 2024-10-21 03:50:03,940 - INFO: 2024-10-21 03:50:03,940 - INFO: service: mariadb -2024-10-21 03:50:03,941 - INFO: 2024-10-21 03:50:03,940 - INFO: Container NAME live-system-mariadb-1 +2024-10-21 03:50:03,941 - INFO: 2024-10-21 03:50:03,940 - INFO: Container NAME live-mariadb-1 2024-10-21 03:50:03,948 - INFO: 2024-10-21 03:50:03,948 - INFO: - mariadb (is_ready=True) -> /backup/databases/mariadb/all_databases.sql 2024-10-21 03:50:03,948 - INFO: 2024-10-21 03:50:03,948 - INFO: service: web 2024-10-21 03:50:03,949 - INFO: 2024-10-21 03:50:03,948 - INFO: - volume: /workspace/restic-compose-backup/src/tests -> /backup/volumes/web/srv/tests 2024-10-21 03:50:03,949 - INFO: 2024-10-21 03:50:03,948 - INFO: ------------------------------------------------------------------- 2024-10-21 03:50:03,949 - INFO: 2024-10-21 03:50:03,948 - INFO: Backing up databases 2024-10-21 03:50:03,949 - INFO: 2024-10-21 03:50:03,948 - INFO: Backing up postgres in service postgres -2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-system-postgres-1 -2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-system-postgres-1 +2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-postgres-1 +2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-postgres-1 2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Backing PostgresContainer container postgres /backup/databases/postgres/test-postgres.sql -2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-system-postgres-1 -2024-10-21 03:50:03,951 - INFO: 2024-10-21 03:50:03,949 - INFO: Container NAME live-system-postgres-1 -2024-10-21 03:50:03,951 - INFO: 2024-10-21 03:50:03,949 - INFO: Container NAME live-system-postgres-1 +2024-10-21 03:50:03,950 - INFO: 2024-10-21 03:50:03,948 - INFO: Container NAME live-postgres-1 +2024-10-21 03:50:03,951 - INFO: 2024-10-21 03:50:03,949 - INFO: Container NAME live-postgres-1 +2024-10-21 03:50:03,951 - INFO: 2024-10-21 03:50:03,949 - INFO: Container NAME live-postgres-1 2024-10-21 03:50:04,145 - INFO: 2024-10-21 03:50:04,144 - INFO: Database backup exit code: =======> 0 2024-10-21 03:50:04,145 - INFO: 2024-10-21 03:50:04,145 - INFO: Exit code: 0 2024-10-21 03:50:04,146 - INFO: 2024-10-21 03:50:04,145 - INFO: Backing up mariadb in service mariadb -2024-10-21 03:50:04,146 - INFO: 2024-10-21 03:50:04,145 - INFO: Container NAME live-system-mariadb-1 -2024-10-21 03:50:04,146 - INFO: 2024-10-21 03:50:04,145 - INFO: Container NAME live-system-mariadb-1 +2024-10-21 03:50:04,146 - INFO: 2024-10-21 03:50:04,145 - INFO: Container NAME live-mariadb-1 +2024-10-21 03:50:04,146 - INFO: 2024-10-21 03:50:04,145 - INFO: Container NAME live-mariadb-1 2024-10-21 03:50:04,348 - INFO: 2024-10-21 03:50:04,347 - INFO: Database backup exit code: =======> 0 2024-10-21 03:50:04,348 - INFO: 2024-10-21 03:50:04,347 - INFO: Exit code: 0 2024-10-21 03:50:04,348 - INFO: 2024-10-21 03:50:04,347 - INFO: Backup files from ===> /backup/ @@ -62,7 +62,7 @@ crond 4.5 dillon's cron daemon, started with loglevel notice ### Validate database restore -```docker exec -it live-system-mariadb-1 bash``` +```docker exec -it live-mariadb-1 bash``` ```mysql -uroot -pmy-secret-pw``` @@ -85,15 +85,15 @@ crond 4.5 dillon's cron daemon, started with loglevel notice Stop the live system to avoid conflicts as the backup service may try to backup the restored system (The backup and restore services are not intended to run on the same system) -```./stop-live-system.sh``` +```./stop-live.sh``` Run the restore -```./restore-backup.sh``` +```./start-live-two-with-restore.sh``` Check the restore logs -```docker logs -f restore-test-restore-1``` +```docker logs -f live-two-restore-1``` You should see logs similar to: @@ -102,7 +102,7 @@ You should see logs similar to: 2024-10-21 03:58:19,579 - INFO: Starting Restore container 2024-10-21 03:58:19,864 - INFO: Restore process container: charming_cohen -2024-10-21 03:58:20,074 - INFO: 2024-10-21 03:58:20,073 - INFO: Status for compose project 'restore-test' +2024-10-21 03:58:20,074 - INFO: 2024-10-21 03:58:20,073 - INFO: Status for compose project 'live-two' 2024-10-21 03:58:20,074 - INFO: 2024-10-21 03:58:20,073 - INFO: Repository: '/restic_data' 2024-10-21 03:58:20,074 - INFO: 2024-10-21 03:58:20,073 - INFO: Restore currently running?: True 2024-10-21 03:58:20,074 - INFO: 2024-10-21 03:58:20,074 - INFO: Include project name in restore path?: False @@ -110,20 +110,20 @@ You should see logs similar to: 2024-10-21 03:58:20,698 - INFO: 2024-10-21 03:58:20,697 - INFO: ------------------------- Detected Config ------------------------- 2024-10-21 03:58:20,698 - INFO: 2024-10-21 03:58:20,697 - INFO: service: web 2024-10-21 03:58:20,698 - INFO: 2024-10-21 03:58:20,697 - INFO: service: postgres -2024-10-21 03:58:20,698 - INFO: 2024-10-21 03:58:20,698 - INFO: Container NAME restore-test-postgres-1 -2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: Container NAME restore-test-postgres-1 +2024-10-21 03:58:20,698 - INFO: 2024-10-21 03:58:20,698 - INFO: Container NAME live-two-postgres-1 +2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: Container NAME live-two-postgres-1 2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: - postgres (is_ready=True) -> /backup/databases/postgres/test-postgres.sql 2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: service: mariadb -2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: Container NAME restore-test-mariadb-1 +2024-10-21 03:58:20,704 - INFO: 2024-10-21 03:58:20,703 - INFO: Container NAME live-two-mariadb-1 2024-10-21 03:58:20,709 - INFO: 2024-10-21 03:58:20,708 - INFO: - mariadb (is_ready=True) -> /backup/databases/mariadb/all_databases.sql 2024-10-21 03:58:20,709 - INFO: 2024-10-21 03:58:20,708 - INFO: ------------------------------------------------------------------- 2024-10-21 03:58:20,709 - INFO: 2024-10-21 03:58:20,708 - INFO: Restoring databases 2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Restoring up postgres in service postgres -2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME restore-test-postgres-1 -2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME restore-test-postgres-1 -2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME restore-test-postgres-1 -2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME restore-test-postgres-1 -2024-10-21 03:58:20,711 - INFO: 2024-10-21 03:58:20,709 - INFO: Database restore_command: =======> restic -r /restic_data dump latest /backup/databases/postgres/test-postgres.sql | psql --host=restore-test-postgres-1 --port=5432 --username=pguser --dbname=postgres +2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME live-two-postgres-1 +2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME live-two-postgres-1 +2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME live-two-postgres-1 +2024-10-21 03:58:20,710 - INFO: 2024-10-21 03:58:20,709 - INFO: Container NAME live-two-postgres-1 +2024-10-21 03:58:20,711 - INFO: 2024-10-21 03:58:20,709 - INFO: Database restore_command: =======> restic -r /restic_data dump latest /backup/databases/postgres/test-postgres.sql | psql --host=live-two-postgres-1 --port=5432 --username=pguser --dbname=postgres 2024-10-21 03:58:21,321 - INFO: SET 2024-10-21 03:58:21,321 - INFO: SET 2024-10-21 03:58:21,321 - INFO: SET @@ -252,9 +252,9 @@ You should see logs similar to: 2024-10-21 03:58:21,388 - INFO: 2024-10-21 03:58:21,387 - INFO: Database Restore exit code: =======> 0 2024-10-21 03:58:21,388 - INFO: 2024-10-21 03:58:21,387 - INFO: Exit code: 0 2024-10-21 03:58:21,388 - INFO: 2024-10-21 03:58:21,388 - INFO: Restoring up mariadb in service mariadb -2024-10-21 03:58:21,388 - INFO: 2024-10-21 03:58:21,388 - INFO: Container NAME restore-test-mariadb-1 -2024-10-21 03:58:21,389 - INFO: 2024-10-21 03:58:21,388 - INFO: Container NAME restore-test-mariadb-1 -2024-10-21 03:58:21,389 - INFO: 2024-10-21 03:58:21,388 - INFO: Database restore_command: =======> restic -r /restic_data dump latest /backup/databases/mariadb/all_databases.sql | mysql --host=restore-test-mariadb-1 --port=3306 --user=root +2024-10-21 03:58:21,388 - INFO: 2024-10-21 03:58:21,388 - INFO: Container NAME live-two-mariadb-1 +2024-10-21 03:58:21,389 - INFO: 2024-10-21 03:58:21,388 - INFO: Container NAME live-two-mariadb-1 +2024-10-21 03:58:21,389 - INFO: 2024-10-21 03:58:21,388 - INFO: Database restore_command: =======> restic -r /restic_data dump latest /backup/databases/mariadb/all_databases.sql | mysql --host=live-two-mariadb-1 --port=3306 --user=root 2024-10-21 03:58:22,817 - INFO: 2024-10-21 03:58:22,816 - INFO: Database Restore exit code: =======> 0 2024-10-21 03:58:22,817 - INFO: 2024-10-21 03:58:22,817 - INFO: Exit code: 0 2024-10-21 03:58:22,817 - INFO: 2024-10-21 03:58:22,817 - INFO: Restoring files to ===> /restored_data/ @@ -263,7 +263,7 @@ You should see logs similar to: ``` You can now check the databases for restored data, for example -```docker exec -it restore-test-mariadb-1 bash`` +```docker exec -it live-two-mariadb-1 bash`` ```mysql -uroot -pmy-secret-pw``` @@ -285,9 +285,9 @@ You should see logs similar to: ``` ### Validate volume restore -We have two services to validate that volumes are restored correctly. see `web` and `web-2` in the `docker-compose-restore-test.yaml` file. When the `docker-compose-restore-service.yaml` file is used the `web` and `web-2` will wait for restore to complete before starting. The service then run ls on the restored mounts to confirm that the restore happened run; +We have two services to validate that volumes are restored correctly. see `web` and `web-2` in the `docker-compose-live-two.yaml` file. When the `docker-compose-restore-service.yaml` file is used the `web` and `web-2` will wait for restore to complete before starting. The service then run ls on the restored mounts to confirm that the restore happened run; -`docker logs -f restore-test-web-1` +`docker logs -f live-two-web-1` You will see the result @@ -301,13 +301,13 @@ tests.py Stop backup system -```./stop-backup-system.sh``` +```./stop-live-two.sh``` ## Clean up Remove Restored data: -```sudo rm -rf ./restored_data``` +```sudo rm -rf ./live_two_filestores``` Remove Backup Repo diff --git a/start-live-system.sh b/start-live.sh similarity index 100% rename from start-live-system.sh rename to start-live.sh diff --git a/stop-live-two.sh b/stop-live-two.sh index 84af649..258cd36 100755 --- a/stop-live-two.sh +++ b/stop-live-two.sh @@ -1 +1 @@ -docker compose -p restore-test -f docker-compose-live-two.yaml -f docker-compose-restore-service.yaml down \ No newline at end of file +docker compose -p live-two -f docker-compose-live-two.yaml -f docker-compose-restore-service.yaml down \ No newline at end of file diff --git a/stop-live.sh b/stop-live.sh index fdac04e..70a3bcf 100755 --- a/stop-live.sh +++ b/stop-live.sh @@ -1 +1 @@ -docker compose -p live-system -f docker-compose-backup-service.yaml -f docker-compose-live.yaml down \ No newline at end of file +docker compose -p live -f docker-compose-backup-service.yaml -f docker-compose-live.yaml down \ No newline at end of file