Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extend migrator to support upgrading between versioned data directories #29

Merged
2 commits merged into from
Mar 22, 2022
Merged

Extend migrator to support upgrading between versioned data directories #29

2 commits merged into from
Mar 22, 2022

Conversation

dfunckt
Copy link
Member

@dfunckt dfunckt commented Feb 15, 2022

The migrator previously would only migrate data stored directly under PGDATA (as was the case in previous versions of this project before switching to versioned directories).

This adds support for upgrading between versioned data directories as well.

@dfunckt dfunckt requested review from Page- and ab77 February 15, 2022 18:53
@dfunckt dfunckt mentioned this pull request Feb 15, 2022
Merged
@bartversluijs
Copy link
Contributor

@dfunckt as requested in balena-io/open-balena#137, I've tested the upgrade from v4.1.0 to the tag build-fix-data-upgrader, as built by the CI.

So LGTM!

Added the logs for more details

=== Upgrading data from v12 to v13
=== Installing tools for Postgres v12
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package postgresql-client-12.
(Reading database ... 12019 files and directories currently installed.)
Preparing to unpack .../postgresql-client-12_12.10-1.pgdg110+1_amd64.deb ...
Unpacking postgresql-client-12 (12.10-1.pgdg110+1) ...
Selecting previously unselected package postgresql-12.
Preparing to unpack .../postgresql-12_12.10-1.pgdg110+1_amd64.deb ...
Unpacking postgresql-12 (12.10-1.pgdg110+1) ...
Setting up postgresql-client-12 (12.10-1.pgdg110+1) ...
Setting up postgresql-12 (12.10-1.pgdg110+1) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for postgresql-common (238.pgdg110+1) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
=== Initializing new data directory /var/lib/postgresql/data/13
=== Creating data dir /var/lib/postgresql/data/13
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data/13 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data/13 -l logfile start

=== Running pg_upgrade
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok
Checking for new cluster tablespace directories             ok

*Clusters are compatible*
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Creating dump of global objects                             ok
Creating dump of database schemas
  docker
  postgres
  resin
  template1
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok
Checking for new cluster tablespace directories             ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_xact to new server                           ok
Setting oldest XID for new cluster                          ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
  template1
  docker
  postgres
  resin
                                                            ok
Copying user relation files
  /var/lib/postgresql/data/12/base/16384/2613
  /var/lib/postgresql/data/12/base/16384/2683
  /var/lib/postgresql/data/12/base/13408/2613
  /var/lib/postgresql/data/12/base/13408/2683
  /var/lib/postgresql/data/12/base/16385/2613
  /var/lib/postgresql/data/12/base/16385/2683
  /var/lib/postgresql/data/12/base/16385/16389
  /var/lib/postgresql/data/12/base/16385/16389_fsm
  /var/lib/postgresql/data/12/base/16385/16395
  /var/lib/postgresql/data/12/base/16385/16395_fsm
  /var/lib/postgresql/data/12/base/16385/16397
  /var/lib/postgresql/data/12/base/16385/16398
  /var/lib/postgresql/data/12/base/16385/16403
  /var/lib/postgresql/data/12/base/16385/16403_fsm
  /var/lib/postgresql/data/12/base/16385/16409
  /var/lib/postgresql/data/12/base/16385/16411
  /var/lib/postgresql/data/12/base/16385/16412
  /var/lib/postgresql/data/12/base/16385/16414
  /var/lib/postgresql/data/12/base/16385/16419
  /var/lib/postgresql/data/12/base/16385/16425
  /var/lib/postgresql/data/12/base/16385/16427
  /var/lib/postgresql/data/12/base/16385/16428
  /var/lib/postgresql/data/12/base/16385/16430
  /var/lib/postgresql/data/12/base/16385/16435
  /var/lib/postgresql/data/12/base/16385/16441
  /var/lib/postgresql/data/12/base/16385/16443
  /var/lib/postgresql/data/12/base/16385/16458
  /var/lib/postgresql/data/12/base/16385/16464
  /var/lib/postgresql/data/12/base/16385/16469
  /var/lib/postgresql/data/12/base/16385/16475
  /var/lib/postgresql/data/12/base/16385/16477
  /var/lib/postgresql/data/12/base/16385/16487
  /var/lib/postgresql/data/12/base/16385/16493
  /var/lib/postgresql/data/12/base/16385/16495
  /var/lib/postgresql/data/12/base/16385/16510
  /var/lib/postgresql/data/12/base/16385/16516
  /var/lib/postgresql/data/12/base/16385/16518
  /var/lib/postgresql/data/12/base/16385/16533
  /var/lib/postgresql/data/12/base/16385/16539
  /var/lib/postgresql/data/12/base/16385/16541
  /var/lib/postgresql/data/12/base/16385/16542
  /var/lib/postgresql/data/12/base/16385/16544
  /var/lib/postgresql/data/12/base/16385/16554
  /var/lib/postgresql/data/12/base/16385/16560
  /var/lib/postgresql/data/12/base/16385/16562
  /var/lib/postgresql/data/12/base/16385/16577
  /var/lib/postgresql/data/12/base/16385/16583
  /var/lib/postgresql/data/12/base/16385/16585
  /var/lib/postgresql/data/12/base/16385/16600
  /var/lib/postgresql/data/12/base/16385/16601
  /var/lib/postgresql/data/12/base/16385/16606
  /var/lib/postgresql/data/12/base/16385/16608
  /var/lib/postgresql/data/12/base/16385/16609
  /var/lib/postgresql/data/12/base/16385/16612
  /var/lib/postgresql/data/12/base/16385/16617
  /var/lib/postgresql/data/12/base/16385/16622
  /var/lib/postgresql/data/12/base/16385/16633
  /var/lib/postgresql/data/12/base/16385/16635
  /var/lib/postgresql/data/12/base/16385/16636
  /var/lib/postgresql/data/12/base/16385/16638
  /var/lib/postgresql/data/12/base/16385/16643
  /var/lib/postgresql/data/12/base/16385/16649
  /var/lib/postgresql/data/12/base/16385/16651
  /var/lib/postgresql/data/12/base/16385/16656
  /var/lib/postgresql/data/12/base/16385/16662
  /var/lib/postgresql/data/12/base/16385/16664
  /var/lib/postgresql/data/12/base/16385/16665
  /var/lib/postgresql/data/12/base/16385/16670
  /var/lib/postgresql/data/12/base/16385/16676
  /var/lib/postgresql/data/12/base/16385/16678
  /var/lib/postgresql/data/12/base/16385/16679
  /var/lib/postgresql/data/12/base/16385/16681
  /var/lib/postgresql/data/12/base/16385/16686
  /var/lib/postgresql/data/12/base/16385/16694
  /var/lib/postgresql/data/12/base/16385/16696
  /var/lib/postgresql/data/12/base/16385/16697
  /var/lib/postgresql/data/12/base/16385/16699
  /var/lib/postgresql/data/12/base/16385/16704
  /var/lib/postgresql/data/12/base/16385/16704_fsm
  /var/lib/postgresql/data/12/base/16385/16704_vm
  /var/lib/postgresql/data/12/base/16385/16710
  /var/lib/postgresql/data/12/base/16385/16712
  /var/lib/postgresql/data/12/base/16385/16713
  /var/lib/postgresql/data/12/base/16385/16718
  /var/lib/postgresql/data/12/base/16385/16727
  /var/lib/postgresql/data/12/base/16385/16729
  /var/lib/postgresql/data/12/base/16385/16730
  /var/lib/postgresql/data/12/base/16385/16735
  /var/lib/postgresql/data/12/base/16385/16742
  /var/lib/postgresql/data/12/base/16385/16744
  /var/lib/postgresql/data/12/base/16385/16745
  /var/lib/postgresql/data/12/base/16385/16747
  /var/lib/postgresql/data/12/base/16385/16757
  /var/lib/postgresql/data/12/base/16385/16763
  /var/lib/postgresql/data/12/base/16385/16765
  /var/lib/postgresql/data/12/base/16385/16780
  /var/lib/postgresql/data/12/base/16385/16786
  /var/lib/postgresql/data/12/base/16385/16788
  /var/lib/postgresql/data/12/base/16385/16789
  /var/lib/postgresql/data/12/base/16385/16791
  /var/lib/postgresql/data/12/base/16385/16801
  /var/lib/postgresql/data/12/base/16385/16807
  /var/lib/postgresql/data/12/base/16385/16809
  /var/lib/postgresql/data/12/base/16385/16810
  /var/lib/postgresql/data/12/base/16385/16812
  /var/lib/postgresql/data/12/base/16385/16822
  /var/lib/postgresql/data/12/base/16385/16822_fsm
  /var/lib/postgresql/data/12/base/16385/16828
  /var/lib/postgresql/data/12/base/16385/16828_fsm
  /var/lib/postgresql/data/12/base/16385/16830
  /var/lib/postgresql/data/12/base/16385/16831
  /var/lib/postgresql/data/12/base/16385/16833
  /var/lib/postgresql/data/12/base/16385/16848
  /var/lib/postgresql/data/12/base/16385/16860
  /var/lib/postgresql/data/12/base/16385/16862
  /var/lib/postgresql/data/12/base/16385/16863
  /var/lib/postgresql/data/12/base/16385/16865
  /var/lib/postgresql/data/12/base/16385/16867
  /var/lib/postgresql/data/12/base/16385/16902
  /var/lib/postgresql/data/12/base/16385/16908
  /var/lib/postgresql/data/12/base/16385/16910
  /var/lib/postgresql/data/12/base/16385/16911
  /var/lib/postgresql/data/12/base/16385/16913
  /var/lib/postgresql/data/12/base/16385/16923
  /var/lib/postgresql/data/12/base/16385/16929
  /var/lib/postgresql/data/12/base/16385/16931
  /var/lib/postgresql/data/12/base/16385/16932
  /var/lib/postgresql/data/12/base/16385/16934
  /var/lib/postgresql/data/12/base/16385/16944
  /var/lib/postgresql/data/12/base/16385/16950
  /var/lib/postgresql/data/12/base/16385/16952
  /var/lib/postgresql/data/12/base/16385/16962
  /var/lib/postgresql/data/12/base/16385/16968
  /var/lib/postgresql/data/12/base/16385/16970
  /var/lib/postgresql/data/12/base/16385/16971
  /var/lib/postgresql/data/12/base/16385/16973
  /var/lib/postgresql/data/12/base/16385/16983
  /var/lib/postgresql/data/12/base/16385/16989
  /var/lib/postgresql/data/12/base/16385/16991
  /var/lib/postgresql/data/12/base/16385/16992
  /var/lib/postgresql/data/12/base/16385/16994
  /var/lib/postgresql/data/12/base/16385/17004
  /var/lib/postgresql/data/12/base/16385/17011
  /var/lib/postgresql/data/12/base/16385/17013
  /var/lib/postgresql/data/12/base/16385/17014
  /var/lib/postgresql/data/12/base/16385/17016
  /var/lib/postgresql/data/12/base/16385/17026
  /var/lib/postgresql/data/12/base/16385/17036
  /var/lib/postgresql/data/12/base/16385/17038
  /var/lib/postgresql/data/12/base/16385/17039
  /var/lib/postgresql/data/12/base/16385/17041
  /var/lib/postgresql/data/12/base/16385/17091
  /var/lib/postgresql/data/12/base/16385/17097
  /var/lib/postgresql/data/12/base/16385/17099
  /var/lib/postgresql/data/12/base/16385/17100
  /var/lib/postgresql/data/12/base/16385/17102
  /var/lib/postgresql/data/12/base/16385/17112
  /var/lib/postgresql/data/12/base/16385/17118
  /var/lib/postgresql/data/12/base/16385/17120
  /var/lib/postgresql/data/12/base/16385/17121
  /var/lib/postgresql/data/12/base/16385/17123
  /var/lib/postgresql/data/12/base/16385/17133
  /var/lib/postgresql/data/12/base/16385/17139
  /var/lib/postgresql/data/12/base/16385/17141
  /var/lib/postgresql/data/12/base/16385/17156
  /var/lib/postgresql/data/12/base/16385/17162
  /var/lib/postgresql/data/12/base/16385/17164
  /var/lib/postgresql/data/12/base/16385/17165
  /var/lib/postgresql/data/12/base/16385/17167
  /var/lib/postgresql/data/12/base/16385/17177
  /var/lib/postgresql/data/12/base/16385/17184
  /var/lib/postgresql/data/12/base/16385/17186
  /var/lib/postgresql/data/12/base/16385/17187
  /var/lib/postgresql/data/12/base/16385/17189
  /var/lib/postgresql/data/12/base/16385/17199
  /var/lib/postgresql/data/12/base/16385/17199_fsm
  /var/lib/postgresql/data/12/base/16385/17206
  /var/lib/postgresql/data/12/base/16385/17208
  /var/lib/postgresql/data/12/base/16385/17209
  /var/lib/postgresql/data/12/base/16385/17211
  /var/lib/postgresql/data/12/base/16385/17221
  /var/lib/postgresql/data/12/base/16385/17227
  /var/lib/postgresql/data/12/base/16385/17229
  /var/lib/postgresql/data/12/base/16385/17249
  /var/lib/postgresql/data/12/base/16385/17255
  /var/lib/postgresql/data/12/base/16385/17257
  /var/lib/postgresql/data/12/base/16385/17272
  /var/lib/postgresql/data/12/base/16385/17278
  /var/lib/postgresql/data/12/base/16385/17280
  /var/lib/postgresql/data/12/base/16385/17295
  /var/lib/postgresql/data/12/base/16385/17301
  /var/lib/postgresql/data/12/base/16385/17303
  /var/lib/postgresql/data/12/base/16385/17304
  /var/lib/postgresql/data/12/base/16385/17306
  /var/lib/postgresql/data/12/base/16385/17316
  /var/lib/postgresql/data/12/base/16385/17322
  /var/lib/postgresql/data/12/base/16385/17324
  /var/lib/postgresql/data/12/base/16385/17325
  /var/lib/postgresql/data/12/base/16385/17327
  /var/lib/postgresql/data/12/base/16385/17417
  /var/lib/postgresql/data/12/base/16385/17419
  /var/lib/postgresql/data/12/base/16385/17423
  /var/lib/postgresql/data/12/base/16385/17424
  /var/lib/postgresql/data/12/base/16385/17425
  /var/lib/postgresql/data/12/base/16385/17426
  /var/lib/postgresql/data/12/base/16385/17427
  /var/lib/postgresql/data/12/base/16385/17428
  /var/lib/postgresql/data/12/base/16385/17429
  /var/lib/postgresql/data/12/base/16385/17430
  /var/lib/postgresql/data/12/base/16385/17431
  /var/lib/postgresql/data/12/base/16385/17432
  /var/lib/postgresql/data/12/base/16385/17433
  /var/lib/postgresql/data/12/base/16385/17434
  /var/lib/postgresql/data/12/base/16385/17435
  /var/lib/postgresql/data/12/base/16385/17436
  /var/lib/postgresql/data/12/base/16385/17437
  /var/lib/postgresql/data/12/base/16385/17438
  /var/lib/postgresql/data/12/base/16385/17439
  /var/lib/postgresql/data/12/base/16385/17440
  /var/lib/postgresql/data/12/base/16385/17441
  /var/lib/postgresql/data/12/base/16385/17442
  /var/lib/postgresql/data/12/base/16385/17443
  /var/lib/postgresql/data/12/base/16385/17444
  /var/lib/postgresql/data/12/base/16385/17445
  /var/lib/postgresql/data/12/base/16385/17446
  /var/lib/postgresql/data/12/base/16385/17447
  /var/lib/postgresql/data/12/base/16385/17448
  /var/lib/postgresql/data/12/base/16385/17449
  /var/lib/postgresql/data/12/base/16385/17450
  /var/lib/postgresql/data/12/base/16385/17451
  /var/lib/postgresql/data/12/base/16385/17452
  /var/lib/postgresql/data/12/base/16385/17453
  /var/lib/postgresql/data/12/base/16385/17454
  /var/lib/postgresql/data/12/base/16385/17455
  /var/lib/postgresql/data/12/base/16385/17456
  /var/lib/postgresql/data/12/base/16385/17457
  /var/lib/postgresql/data/12/base/16385/17458
  /var/lib/postgresql/data/12/base/16385/17459
  /var/lib/postgresql/data/12/base/16385/17460
  /var/lib/postgresql/data/12/base/16385/17461
  /var/lib/postgresql/data/12/base/16385/17462
  /var/lib/postgresql/data/12/base/16385/17463
  /var/lib/postgresql/data/12/base/1/2613
  /var/lib/postgresql/data/12/base/1/2683
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster                      ok
Creating script to delete old cluster                       ok
Checking for extension updates                              notice

Your installation contains extensions that should be updated
with the ALTER EXTENSION command.  The file
    update_extensions.sql
when executed by psql by the database superuser will update
these extensions.


Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh
=== Restoring configuration files
=== Migration succeeded; removing old data directory

PostgreSQL Database directory appears to contain a database; Skipping initialization

@dfunckt dfunckt marked this pull request as ready for review February 17, 2022 10:03
@dfunckt
Copy link
Member Author

dfunckt commented Feb 17, 2022

Thank you @bartversluijs !

@bartversluijs
Copy link
Contributor

@dfunckt any idea when this PR will be merged? I can then update balena-io/open-balena#137 with the change and upgrade our instance! 🙂

@dfunckt
Copy link
Member Author

dfunckt commented Feb 21, 2022

Just trying to put some time aside to look into this notice @bartversluijs emitted by pg_upgrade check and determine if/how I should address it:

Checking for extension updates                              notice

Your installation contains extensions that should be updated
with the ALTER EXTENSION command.  The file
    update_extensions.sql
when executed by psql by the database superuser will update
these extensions.

@bartversluijs
Copy link
Contributor

Ah, didn't notice that log!
Thanks for your response @dfunckt and let me know if I can help!

@dfunckt
Extend migrator to support upgrading between versioned data directories
a569d60 
The migrator previously would only migrate data stored directly under PGDATA (as was the case in previous versions of this project before switching to versioned directories).

This adds support for upgrading between versioned data directories as well.

Change-type: minor
@dfunckt dfunckt force-pushed the fix-data-upgrader branch from bb35884 to a569d60 Compare March 22, 2022 13:34
@dfunckt
Copy link
Member Author

dfunckt commented Mar 22, 2022

@balena-ci I self-certify!

@dfunckt
Copy link
Member Author

dfunckt commented Mar 22, 2022

@bartversluijs as soon as this is merged and the new version published it should be good to go in your versions bump PR.

@ghost ghost merged commit aa51c3f into master Mar 22, 2022
@ghost ghost deleted the fix-data-upgrader branch March 22, 2022 13:39
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Page- Page- Awaiting requested review from Page-

@ab77 ab77 Awaiting requested review from ab77

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dfunckt @bartversluijs