AiiDA team
Replies: 16 comments 3 replies
-
|
I just wanted to report that I did a "final migration" from a partially migrated profile (that was migrated as a test just before the #4345 was merged). It all went well, it took only a few seconds. I didn't record additional data though - anyway this should be a corner usecase. I'm now proceeding with testing more migrations and will report with the table all results. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Question @sphuber do we also want to ask statistics on the repo itself? E.g. old and new file count, old and new size on disk? Or we don't care as we know the answer? |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
I intentionally didn't include those since we tested the file repo migration half a year ago where we requested this data. Since this is a potentially time consuming operation, I didn't want people to get stuck waiting on that. |
Beta Was this translation helpful? Give feedback.
-
(Large) Backup of
|
| Statistic | Answer |
|---|---|
| Database backend | SqlAlchemy |
| Starting revision | 0edcdd5a30f0 |
| Node count before migration | 4025758 |
| Node count after migration | 4025758 |
| Database size before migration | 10 GB |
| Database size after migration | 16 GB |
| Time taken for the actual migration | 39h 59m 32s |
Messages printed to stdout by the migrations
$ time verdi storage migrate -f;
Report: Migrating to the head of the legacy sqlalchemy branch
Migrating file repository: shard 21 12.9%|███████████████████████▍ | 33/256Warning: Detected repository folders that were missing the required subfolder `path` or `raw_input`. The paths of those nodes repository folders have been written to a log file: /home/aiida/envs/aiida-3dd/migration-repository-missing-subfolder-3ax_6xzy.json
Warning: Migrated file repository to the new disk object store. The old repository has not been deleted out of safety and can be found at /hith/aiida-3dd/repositories/prod/repository.
Report: - 7536a82b2cc4 -> 1feaea71bd5a
Report: - 1feaea71bd5a -> 535039300e4a
Report: - 535039300e4a -> 34a831f4286d
Report: - 34a831f4286d -> 1de112340b16
Report: - 1de112340b16 -> 1de112340b17
Report: - 1de112340b17 -> 1de112340b18
Report: Migrating to the head of the main branch
Success: migration completed
real 2399m32.403s
user 254m53.758s
sys 101m56.490sOutput of `verdi storage info --statistics`
$ verdi storage info --statistics
database:
Users:
count: 2
emails:
- [email protected]
- [email protected]
Computers:
count: 3
labels:
- daint-gpu
- daint-mc
- localhost
Nodes:
count: 4025758
node_types:
- data.core.array.bands.BandsData.
- data.core.array.kpoints.KpointsData.
- data.core.array.trajectory.TrajectoryData.
- data.core.bool.Bool.
- data.core.code.Code.
- data.core.dict.Dict.
- data.core.float.Float.
- data.core.folder.FolderData.
- data.core.int.Int.
- data.core.remote.RemoteData.
- data.core.str.Str.
- data.core.structure.StructureData.
- data.core.upf.UpfData.
- process.calculation.calcfunction.CalcFunctionNode.
- process.calculation.calcjob.CalcJobNode.
- process.workflow.workchain.WorkChainNode.
process_types:
- abc.create_kpoints_from_distance
- aiida.calculations:quantumespresso.create_kpoints_from_distance
- aiida.calculations:quantumespresso.create_magnetic_allotrope
- aiida.calculations:quantumespresso.pw
- aiida.workflows:quantumespresso.pw.base
- aiida.workflows:quantumespresso.pw.relax
- aiida_quantumespresso.calculations.functions.create_kpoints_from_distance.create_kpoints_from_distance
- aiida_quantumespresso.workflows.functions.create_kpoints_from_distance.create_kpoints_from_distance
Groups:
count: 21
type_strings:
- core
- core.auto
- core.import
- core.upf
Comments:
count: 0
Logs:
count: 2738723
Links:
count: 9621976
repository:
SHA-hash algorithm: sha256
Compression algorithm: zlib+1
Packs: 839
Objects:
unpacked: 0
packed: 7041025
Size (MB):
unpacked: 0.0
packed: 3459629.9071388245
other: 1178.9375Output of final SCF query
('alembic_version', 'alembic_version_pkc', 1.0, '8192 bytes', '16 kB', 0)
('db_dbsetting', 'db_dbsetting_pkey', 4.0, '8192 bytes', '16 kB', 0)
('db_dbsetting', 'ix_pat_db_dbsetting_key', 4.0, '8192 bytes', '16 kB', 0)
('db_dbsetting', 'uq_db_dbsetting_key', 4.0, '8192 bytes', '16 kB', 0)
('db_dbuser', 'db_dbuser_pkey', 2.0, '8192 bytes', '16 kB', 0)
('db_dbuser', 'ix_pat_db_dbuser_email', 2.0, '8192 bytes', '16 kB', 0)
('db_dbuser', 'uq_db_dbuser_email', 2.0, '8192 bytes', '16 kB', 0)
('db_dbauthinfo', 'db_dbauthinfo_pkey', 3.0, '8192 bytes', '16 kB', 0)
('db_dbauthinfo', 'ix_db_dbauthinfo_aiidauser_id', 3.0, '8192 bytes', '16 kB', 0)
('db_dbauthinfo', 'ix_db_dbauthinfo_dbcomputer_id', 3.0, '8192 bytes', '16 kB', 0)
('db_dbauthinfo', 'uq_db_dbauthinfo_aiidauser_id_dbcomputer_id', 3.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'db_dbcomputer_pkey', 3.0, '8192 bytes', '16 kB', 1)
('db_dbcomputer', 'ix_pat_db_dbcomputer_label', 3.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'uq_db_dbcomputer_label', 3.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'uq_db_dbcomputer_uuid', 3.0, '8192 bytes', '16 kB', 0)
('db_dbcomment', 'db_dbcomment_pkey', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'ix_db_dbcomment_dbnode_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'ix_db_dbcomment_user_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'uq_db_dbcomment_uuid', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbgroup', 'db_dbgroup_pkey', 21.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_label', 21.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'ix_pat_db_dbgroup_label', 21.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_type_string', 21.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'ix_pat_db_dbgroup_type_string', 21.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_user_id', 21.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'uq_db_dbgroup_label_type_string', 21.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'uq_db_dbgroup_uuid', 21.0, '8192 bytes', '16 kB', 0)
('db_dbgroup_dbnodes', 'db_dbgroup_dbnodes_pkey', 203942.0, '8824 kB', '4496 kB', 0)
('db_dbgroup_dbnodes', 'ix_db_dbgroup_dbnodes_dbgroup_id', 203942.0, '8824 kB', '4504 kB', 0)
('db_dbgroup_dbnodes', 'ix_db_dbgroup_dbnodes_dbnode_id', 203942.0, '8824 kB', '4496 kB', 2)
('db_dbgroup_dbnodes', 'uq_db_dbgroup_dbnodes_dbgroup_id_dbnode_id', 203942.0, '8824 kB', '4496 kB', 0)
('db_dblink', 'db_dblink_pkey', 9621976.0, '619 MB', '206 MB', 0)
('db_dblink', 'ix_db_dblink_input_id', 9621976.0, '619 MB', '206 MB', 2)
('db_dblink', 'ix_db_dblink_label', 9621976.0, '619 MB', '307 MB', 0)
('db_dblink', 'ix_pat_db_dblink_label', 9621976.0, '619 MB', '307 MB', 0)
('db_dblink', 'ix_db_dblink_output_id', 9621976.0, '619 MB', '206 MB', 2)
('db_dblink', 'ix_db_dblink_type', 9621976.0, '619 MB', '264 MB', 0)
('db_dblink', 'ix_pat_db_dblink_type', 9621976.0, '619 MB', '264 MB', 0)
('db_dbnode', 'db_dbnode_pkey', 4025758.0, '5626 MB', '179 MB', 6)
('db_dbnode', 'ix_db_dbnode_ctime', 4025758.0, '5626 MB', '157 MB', 0)
('db_dbnode', 'ix_db_dbnode_dbcomputer_id', 4025758.0, '5626 MB', '157 MB', 1)
('db_dbnode', 'ix_db_dbnode_label', 4025758.0, '5626 MB', '162 MB', 0)
('db_dbnode', 'ix_pat_db_dbnode_label', 4025758.0, '5626 MB', '162 MB', 0)
('db_dbnode', 'ix_db_dbnode_mtime', 4025758.0, '5626 MB', '157 MB', 3)
('db_dbnode', 'ix_db_dbnode_process_type', 4025758.0, '5626 MB', '193 MB', 0)
('db_dbnode', 'ix_pat_db_dbnode_process_type', 4025758.0, '5626 MB', '193 MB', 0)
('db_dbnode', 'ix_db_dbnode_node_type', 4025758.0, '5626 MB', '330 MB', 0)
('db_dbnode', 'ix_pat_db_dbnode_node_type', 4025758.0, '5626 MB', '330 MB', 3)
('db_dbnode', 'ix_db_dbnode_user_id', 4025758.0, '5626 MB', '157 MB', 0)
('db_dbnode', 'uq_db_dbnode_uuid', 4025758.0, '5626 MB', '220 MB', 0)
('db_dblog', 'db_dblog_pkey', 2736855.0, '3069 MB', '59 MB', 3)
('db_dblog', 'ix_db_dblog_dbnode_id', 2736855.0, '3069 MB', '59 MB', 2)
('db_dblog', 'ix_db_dblog_levelname', 2736855.0, '3069 MB', '59 MB', 0)
('db_dblog', 'ix_pat_db_dblog_levelname', 2736855.0, '3069 MB', '59 MB', 0)
('db_dblog', 'ix_db_dblog_loggername', 2736855.0, '3069 MB', '201 MB', 0)
('db_dblog', 'ix_pat_db_dblog_loggername', 2736855.0, '3069 MB', '201 MB', 0)
('db_dblog', 'uq_db_dblog_uuid', 2736855.0, '3069 MB', '82 MB', 0)- ✅
verdi statusshows green checkmark forstorage. - ✅ Queries: filters, relations, projections.
- ✅ Opening file in
FolderDatawithin context manager.
Beta Was this translation helpful? Give feedback.
0 replies
-
(Small)
|
| Statistic | Answer |
|---|---|
| Database backend | Django |
| Starting revision | 0045_dbgroup_extras |
| Node count before migration | 198523 |
| Node count after migration | 198523 |
| Database size before migration | 3975 MB |
| Database size after migration | 3108 MB |
| Time taken for the actual migration | 139m37s |
Messages printed to stdout by the migrations
$ time verdi storage migrate -f
Report: Migrating to the head of the legacy django branch
Report: - django_0045 -> django_0046
Warning: Detected repository folders that were missing the required subfolder `path` or `raw_input`. The paths of those nodes repository folders have been written to a log file: /home/aiida/envs/aiida-super/code/aiida-core/migration-repository-missing-subfolder-f7v9zxab.json
Warning: Migrated file repository to the new disk object store. The old repository has not been deleted out of safety and can be found at /hith/aiida-super/repositories/dev/repository.
Report: - django_0046 -> django_0047
Report: - django_0047 -> django_0048
Report: - django_0048 -> django_0049
Report: - django_0049 -> django_0050
Report: Migrating to the head of the main branch
Success: migration completed
real 139m37.863s
user 5m8.517s
sys 2m6.587sOutput of `verdi storage info --statistics`
$ verdi storage info --statistics
database:
Users:
count: 2
emails:
- [email protected]
- [email protected]
Computers:
count: 4
labels:
- daint-gpu
- daint-mc
- eiger
- localhost
Nodes:
count: 198523
node_types:
- data.core.array.bands.BandsData.
- data.core.array.kpoints.KpointsData.
- data.core.array.projection.ProjectionData.
- data.core.array.trajectory.TrajectoryData.
- data.core.array.xy.XyData.
- data.core.bool.Bool.
- data.core.code.Code.
- data.core.dict.Dict.
- data.core.float.Float.
- data.core.folder.FolderData.
- data.core.int.Int.
- data.core.list.List.
- data.core.remote.RemoteData.
- data.core.structure.StructureData.
- data.pseudo.upf.UpfData.
- process.calculation.calcfunction.CalcFunctionNode.
- process.calculation.calcjob.CalcJobNode.
- process.workflow.workchain.WorkChainNode.
process_types:
- aiida.calculations:quantumespresso.create_kpoints_from_distance
- aiida.calculations:quantumespresso.dos
- aiida.calculations:quantumespresso.projwfc
- aiida.calculations:quantumespresso.pw
- aiida.calculations:quantumespresso.seekpath_structure_analysis
- aiida.workflows:quantumespresso.pdos
- aiida.workflows:quantumespresso.pw.bands
- aiida.workflows:quantumespresso.pw.base
- aiida.workflows:quantumespresso.pw.relax
Groups:
count: 29
type_strings:
- core
- pseudo.family.pseudo_dojo
- pseudo.family.sssp
Comments:
count: 0
Logs:
count: 25835
Links:
count: 195310
repository:
SHA-hash algorithm: sha256
Compression algorithm: zlib+1
Packs: 9
Objects:
unpacked: 0
packed: 138930
Size (MB):
unpacked: 0.0
packed: 35950.73136520386
other: 23.06640625Output of final SQL query
$ verdi devel run-sql "SELECT pt.tablename AS TableName, t.indexname AS IndexName, pc.reltuples, pg_size_pretty(pg_relation_size(quote_ident(pt.tablename)::text)), pg_size_pretty(pg_relation_size(quote_ident(t.indexrelname)::text)), t.idx_scan FROM pg_tables AS pt LEFT OUTER JOIN pg_class AS pc ON pt.tablename=pc.relname LEFT OUTER JOIN (SELECT pc.relname AS TableName, pc2.relname AS IndexName, psai.idx_scan, psai.indexrelname FROM pg_index AS pi JOIN pg_class AS pc ON pc.oid = pi.indrelid JOIN pg_class AS pc2 ON pc2.oid = pi.indexrelid JOIN pg_stat_all_indexes AS psai ON pi.indexrelid = psai.indexrelid )AS T ON pt.tablename = T.TableName WHERE pt.schemaname='public';"
('alembic_version', 'alembic_version_pkc', 0.0, '8192 bytes', '16 kB', 5)
('db_dbuser', 'db_dbuser_pkey', 2.0, '8192 bytes', '16 kB', 588114)
('db_dbuser', 'ix_pat_db_dbuser_email', 2.0, '8192 bytes', '16 kB', 0)
('db_dbuser', 'uq_db_dbuser_email', 2.0, '8192 bytes', '16 kB', 0)
('db_dbgroup_dbnodes', 'ix_db_dbgroup_dbnodes_dbgroup_id', 162776.0, '7104 kB', '3600 kB', 0)
('db_dbgroup_dbnodes', 'ix_db_dbgroup_dbnodes_dbnode_id', 162776.0, '7104 kB', '3592 kB', 4)
('db_dbgroup_dbnodes', 'db_dbgroup_dbnodes_pkey', 162776.0, '7104 kB', '3624 kB', 0)
('db_dbgroup_dbnodes', 'uq_db_dbgroup_dbnodes_dbgroup_id_dbnode_id', 162776.0, '7104 kB', '3592 kB', 7)
('db_dblink', 'db_dblink_pkey', 195310.0, '19 MB', '7960 kB', 0)
('db_dblink', 'ix_db_dblink_input_id', 195310.0, '19 MB', '4312 kB', 7)
('db_dblink', 'ix_db_dblink_label', 195310.0, '19 MB', '6640 kB', 0)
('db_dblink', 'ix_pat_db_dblink_label', 195310.0, '19 MB', '6640 kB', 0)
('db_dblink', 'ix_db_dblink_output_id', 195310.0, '19 MB', '4312 kB', 12)
('db_dblink', 'ix_db_dblink_type', 195310.0, '19 MB', '5520 kB', 0)
('db_dblink', 'ix_pat_db_dblink_type', 195310.0, '19 MB', '5520 kB', 0)
('db_dbcomment', 'ix_db_dbcomment_dbnode_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'ix_db_dbcomment_user_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'db_dbcomment_pkey', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'uq_db_dbcomment_uuid', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbauthinfo', 'ix_db_dbauthinfo_aiidauser_id', 4.0, '8192 bytes', '16 kB', 0)
('db_dbauthinfo', 'ix_db_dbauthinfo_dbcomputer_id', 4.0, '8192 bytes', '16 kB', 0)
('db_dbauthinfo', 'db_dbauthinfo_pkey', 4.0, '8192 bytes', '16 kB', 288742)
('db_dbauthinfo', 'uq_db_dbauthinfo_aiidauser_id_dbcomputer_id', 4.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'ix_pat_db_dbcomputer_label', 4.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'db_dbcomputer_pkey', 4.0, '8192 bytes', '16 kB', 1677134)
('db_dbcomputer', 'uq_db_dbcomputer_label', 4.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'uq_db_dbcomputer_uuid', 4.0, '8192 bytes', '16 kB', 0)
('db_dblog', 'db_dblog_pkey', 25835.0, '40 MB', '1128 kB', 990)
('db_dblog', 'ix_db_dblog_dbnode_id', 25835.0, '40 MB', '584 kB', 3)
('db_dblog', 'ix_db_dblog_levelname', 25835.0, '40 MB', '584 kB', 1)
('db_dblog', 'ix_pat_db_dblog_levelname', 25835.0, '40 MB', '584 kB', 0)
('db_dblog', 'ix_db_dblog_loggername', 25835.0, '40 MB', '2080 kB', 0)
('db_dblog', 'ix_pat_db_dblog_loggername', 25835.0, '40 MB', '2080 kB', 0)
('db_dblog', 'uq_db_dblog_uuid', 25835.0, '40 MB', '808 kB', 0)
('db_dbnode', 'db_dbnode_pkey', 198500.0, '530 MB', '49 MB', 29422630)
('db_dbnode', 'ix_db_dbnode_ctime', 198500.0, '530 MB', '8520 kB', 0)
('db_dbnode', 'ix_db_dbnode_dbcomputer_id', 198500.0, '530 MB', '8528 kB', 0)
('db_dbnode', 'ix_db_dbnode_label', 198500.0, '530 MB', '8624 kB', 0)
('db_dbnode', 'ix_pat_db_dbnode_label', 198500.0, '530 MB', '8624 kB', 0)
('db_dbnode', 'ix_db_dbnode_mtime', 198500.0, '530 MB', '8520 kB', 0)
('db_dbnode', 'ix_db_dbnode_process_type', 198500.0, '530 MB', '9144 kB', 0)
('db_dbnode', 'ix_pat_db_dbnode_process_type', 198500.0, '530 MB', '9144 kB', 0)
('db_dbnode', 'ix_db_dbnode_node_type', 198500.0, '530 MB', '18 MB', 0)
('db_dbnode', 'ix_pat_db_dbnode_node_type', 198500.0, '530 MB', '18 MB', 0)
('db_dbnode', 'ix_db_dbnode_user_id', 198500.0, '530 MB', '8528 kB', 0)
('db_dbnode', 'uq_db_dbnode_uuid', 198500.0, '530 MB', '12 MB', 0)
('db_dbsetting', 'db_dbsetting_pkey', 5.0, '24 kB', '120 kB', 109821)
('db_dbsetting', 'ix_pat_db_dbsetting_key', 5.0, '24 kB', '16 kB', 0)
('db_dbsetting', 'uq_db_dbsetting_key', 5.0, '24 kB', '16 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_label', 29.0, '32 kB', '16 kB', 0)
('db_dbgroup', 'ix_pat_db_dbgroup_label', 29.0, '32 kB', '16 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_type_string', 29.0, '32 kB', '16 kB', 0)
('db_dbgroup', 'ix_pat_db_dbgroup_type_string', 29.0, '32 kB', '16 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_user_id', 29.0, '32 kB', '16 kB', 0)
('db_dbgroup', 'db_dbgroup_pkey', 29.0, '32 kB', '16 kB', 1215671)
('db_dbgroup', 'uq_db_dbgroup_label_type_string', 29.0, '32 kB', '16 kB', 0)
('db_dbgroup', 'uq_db_dbgroup_uuid', 29.0, '32 kB', '16 kB', 0)Post-migration tests:
- ✅
verdi statusshows green checkmark forstorage. - ✅ Queries: filters, relations, projections.
- ✅ Opening file in
FolderDatawithin context manager.
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Results for the "Pyrene-based metal organic frameworks" (link)
Messages printed to stdout by the migrationsReport: Migrating to the head of the legacy sqlalchemy branch
Report: - 0edcdd5a30f0 -> 7536a82b2cc4
Warning: Migrated file repository to the new disk object store. The old repository has not been deleted out of safety and can be found at /root/.aiida/repository/main_profile/repository.
Report: - 7536a82b2cc4 -> 1feaea71bd5a
Report: - 1feaea71bd5a -> 535039300e4a
Report: - 535039300e4a -> 34a831f4286d
Report: - 34a831f4286d -> 1de112340b16
Report: - 1de112340b16 -> 1de112340b17
Report: - 1de112340b17 -> 1de112340b18
Report: Migrating to the head of the main branch
Success: migration completedOutput of final SQL query('alembic_version', 'alembic_version_pkc', 1.0, '8192 bytes', '16 kB', 50)
('db_dbuser', 'db_dbuser_pkey', 2.0, '8192 bytes', '16 kB', 56897)
('db_dbuser', 'ix_pat_db_dbuser_email', 2.0, '8192 bytes', '16 kB', 0)
('db_dbuser', 'uq_db_dbuser_email', 2.0, '8192 bytes', '16 kB', 0)
('db_dbauthinfo', 'db_dbauthinfo_pkey', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbauthinfo', 'ix_db_dbauthinfo_aiidauser_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbauthinfo', 'ix_db_dbauthinfo_dbcomputer_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbauthinfo', 'uq_db_dbauthinfo_aiidauser_id_dbcomputer_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomputer', 'db_dbcomputer_pkey', 3.0, '8192 bytes', '16 kB', 13628)
('db_dbcomputer', 'ix_pat_db_dbcomputer_label', 3.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'uq_db_dbcomputer_label', 3.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'uq_db_dbcomputer_uuid', 3.0, '8192 bytes', '16 kB', 0)
('db_dbgroup_dbnodes', 'db_dbgroup_dbnodes_pkey', 57973.0, '2512 kB', '1288 kB', 0)
('db_dbgroup_dbnodes', 'ix_db_dbgroup_dbnodes_dbgroup_id', 57973.0, '2512 kB', '408 kB', 0)
('db_dbgroup_dbnodes', 'ix_db_dbgroup_dbnodes_dbnode_id', 57973.0, '2512 kB', '1280 kB', 2)
('db_dbgroup_dbnodes', 'uq_db_dbgroup_dbnodes_dbgroup_id_dbnode_id', 57973.0, '2512 kB', '1288 kB', 2)
('db_dblink', 'db_dblink_pkey', 159872.0, '11 MB', '3528 kB', 0)
('db_dblink', 'ix_db_dblink_input_id', 159872.0, '11 MB', '1992 kB', 2)
('db_dblink', 'ix_db_dblink_label', 159872.0, '11 MB', '1136 kB', 0)
('db_dblink', 'ix_pat_db_dblink_label', 159872.0, '11 MB', '1136 kB', 0)
('db_dblink', 'ix_db_dblink_output_id', 159872.0, '11 MB', '2136 kB', 2)
('db_dblink', 'ix_db_dblink_type', 159872.0, '11 MB', '1112 kB', 0)
('db_dblink', 'ix_pat_db_dblink_type', 159872.0, '11 MB', '1112 kB', 0)
('db_dbsetting', 'db_dbsetting_pkey', 2.0, '8192 bytes', '16 kB', 0)
('db_dbsetting', 'ix_pat_db_dbsetting_key', 2.0, '8192 bytes', '16 kB', 0)
('db_dbsetting', 'uq_db_dbsetting_key', 2.0, '8192 bytes', '16 kB', 0)
('db_dbcomment', 'db_dbcomment_pkey', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'ix_db_dbcomment_dbnode_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'ix_db_dbcomment_user_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'uq_db_dbcomment_uuid', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbnode', 'db_dbnode_pkey', 56832.0, '61 MB', '2504 kB', 1151848)
('db_dbnode', 'ix_db_dbnode_ctime', 56832.0, '61 MB', '1264 kB', 0)
('db_dbnode', 'ix_db_dbnode_dbcomputer_id', 56832.0, '61 MB', '400 kB', 1)
('db_dbnode', 'ix_db_dbnode_label', 56832.0, '61 MB', '408 kB', 0)
('db_dbnode', 'ix_pat_db_dbnode_label', 56832.0, '61 MB', '408 kB', 0)
('db_dbnode', 'ix_db_dbnode_mtime', 56832.0, '61 MB', '1264 kB', 0)
('db_dbnode', 'ix_db_dbnode_process_type', 56832.0, '61 MB', '416 kB', 0)
('db_dbnode', 'ix_pat_db_dbnode_process_type', 56832.0, '61 MB', '416 kB', 0)
('db_dbnode', 'ix_db_dbnode_node_type', 56832.0, '61 MB', '424 kB', 0)
('db_dbnode', 'ix_pat_db_dbnode_node_type', 56832.0, '61 MB', '424 kB', 0)
('db_dbnode', 'ix_db_dbnode_user_id', 56832.0, '61 MB', '400 kB', 1)
('db_dbnode', 'uq_db_dbnode_uuid', 56832.0, '61 MB', '1776 kB', 0)
('db_dbgroup', 'db_dbgroup_pkey', 63.0, '16 kB', '16 kB', 1268)
('db_dbgroup', 'ix_db_dbgroup_label', 63.0, '16 kB', '16 kB', 0)
('db_dbgroup', 'ix_pat_db_dbgroup_label', 63.0, '16 kB', '16 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_type_string', 63.0, '16 kB', '16 kB', 0)
('db_dbgroup', 'ix_pat_db_dbgroup_type_string', 63.0, '16 kB', '16 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_user_id', 63.0, '16 kB', '16 kB', 0)
('db_dbgroup', 'uq_db_dbgroup_label_type_string', 63.0, '16 kB', '16 kB', 0)
('db_dbgroup', 'uq_db_dbgroup_uuid', 63.0, '16 kB', '16 kB', 0)
('db_dblog', 'db_dblog_pkey', 25144.0, '28 MB', '568 kB', 0)
('db_dblog', 'ix_db_dblog_dbnode_id', 25144.0, '28 MB', '360 kB', 2)
('db_dblog', 'ix_db_dblog_levelname', 25144.0, '28 MB', '192 kB', 0)
('db_dblog', 'ix_pat_db_dblog_levelname', 25144.0, '28 MB', '192 kB', 0)
('db_dblog', 'ix_db_dblog_loggername', 25144.0, '28 MB', '200 kB', 0)
('db_dblog', 'ix_pat_db_dblog_loggername', 25144.0, '28 MB', '200 kB', 0)
('db_dblog', 'uq_db_dblog_uuid', 25144.0, '28 MB', '792 kB', 0) |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
One of my local sssp verification running database.
Time taken for the actual migration and some warings: SQL queryFollowing warning shows when check one of the process node: Seems the warning message is not complete and the |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
The deprecation warning is simply triggered if anywhere the entrypoint just add those lines around lines 308 just before the |
Beta Was this translation helpful? Give feedback.
-
|
@sphuber Thanks. I check with the |
Beta Was this translation helpful? Give feedback.
-
(Gargantuan) Backup of
|
| Statistic | Answer |
|---|---|
| Database backend | SqlAlchemy |
| Starting revision | bf591f31dd12 |
| Node count before migration | 10744163 |
| Node count after migration | 10744163 |
| Database size before migration | 20 GB |
| Database size after migration | 32 GB |
| Time taken for the actual migration | 56h 5m 21s |
Messages printed to stdout by the migrations (ERROR)
$ time verdi storage migrate -f
Report: Migrating to the head of the legacy sqlalchemy branch
Report: - 0edcdd5a30f0 -> 7536a82b2cc4
Warning: Detected repository folders that were missing the required subfolder `path` or `raw_input`. The paths of those nodes repository folders have been written to a log file: /home/aiida/envs/aiida-sdb/migration-repository-missing-subf
older-hpsmolyl.json
Warning: Migrated file repository to the new disk object store. The old repository has not been deleted out of safety and can be found at /hith/aiida-sdb/repositories/prod/repository.
Report: - 7536a82b2cc4 -> 1feaea71bd5a
Report: - 1feaea71bd5a -> 535039300e4a
Report: - 535039300e4a -> 34a831f4286d
Report: - 34a831f4286d -> 1de112340b16
real 3348m16.793s
user 188m51.579s
sys 53m5.532sHere an error appeared initially (see below), that was fixed by #5402. After adapting the version number with:
verdi devel run-sql "update alembic_version set version_num='34a831f4286d' where version_num='1de112340b16'; commit; select version_num from alembic_version"The migration continued succesfully:
$ time verdi storage migrate -f
Report: Migrating to the head of the legacy sqlalchemy branch
Report: - 34a831f4286d -> 1de112340b16
Report: - 1de112340b16 -> 1de112340b17
Report: - 1de112340b17 -> 1de112340b18
Report: Migrating to the head of the main branch
Success: migration completed
real 17m4.686s
user 0m2.110s
sys 0m3.248sError trace from initial migration attempt
Traceback (most recent call last):
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
self.dialect.do_execute(
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: column "dbnode_id" contains null values
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/aiida/.virtualenvs/aiida-sdb/bin/verdi", line 8, in
sys.exit(verdi())
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/click/core.py", line 1128, in call
return self.main(*args, **kwargs)
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/aiida/envs/aiida-sdb/code/aiida-core/aiida/cmdline/commands/cmd_storage.py", line 79, in storage_migrate
storage_cls.migrate(profile)
File "/home/aiida/envs/aiida-sdb/code/aiida-core/aiida/storage/psql_dos/backend.py", line 62, in migrate
cls.migrator(profile).migrate()
File "/home/aiida/envs/aiida-sdb/code/aiida-core/aiida/storage/psql_dos/migrator.py", line 245, in migrate
self.migrate_up('sqlalchemy@head')
File "/home/aiida/envs/aiida-sdb/code/aiida-core/aiida/storage/psql_dos/migrator.py", line 263, in migrate_up
upgrade(config, version)
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/command.py", line 320, in upgrade
script.run_env()
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/script/base.py", line 563, in run_env
util.load_python_file(self.dir, "env.py")
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
module = load_module_py(module_id, path)
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "", line 848, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/aiida/envs/aiida-sdb/code/aiida-core/aiida/storage/psql_dos/migrations/env.py", line 59, in
run_migrations_online()
File "/home/aiida/envs/aiida-sdb/code/aiida-core/aiida/storage/psql_dos/migrations/env.py", line 52, in run_migrations_online
context.run_migrations() # pylint: disable=no-member
File "", line 8, in run_migrations
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
self.get_context().run_migrations(**kw)
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
step.migration_fn(**kw)
File "/home/aiida/envs/aiida-sdb/code/aiida-core/aiida/storage/psql_dos/migrations/versions/1de112340b17_django_parity_2.py", line 59, in upgrade
op.alter_column('db_dbgroup_dbnodes', 'dbnode_id', existing_type=sa.INTEGER(), nullable=False)
File "", line 8, in alter_column
File "", line 3, in alter_column
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/operations/ops.py", line 1880, in alter_column
return operations.invoke(alt)
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/operations/base.py", line 392, in invoke
return fn(self, operation)
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/operations/toimpl.py", line 50, in alter_column
operations.impl.alter_column(
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/ddl/postgresql.py", line 170, in alter_column
super(PostgresqlImpl, self).alter_column(
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/ddl/impl.py", line 227, in alter_column
self._exec(
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/alembic/ddl/impl.py", line 193, in _exec
return conn.execute(construct, multiparams)
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/sqlalchemy/future/engine.py", line 280, in execute
return self._execute_20(
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
return connection._execute_ddl(
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1381, in _execute_ddl
ret = self._execute_context(
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
self.handle_dbapi_exception(
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2026, in handle_dbapi_exception
util.raise(
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise
raise exception
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
self.dialect.do_execute(
File "/home/aiida/.virtualenvs/aiida-sdb/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) column "dbnode_id" contains null values
[SQL: ALTER TABLE db_dbgroup_dbnodes ALTER COLUMN dbnode_id SET NOT NULL]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
</details>
Beta Was this translation helpful? Give feedback.
0 replies
-
|
DB
Messages printed to stdout by the migrations: repeated multiple times apart from standard reports. In addition: I checked: they were 52, 1 was even missing (I probably deleted it?), the remaining 51 were all calcjobs in an excepted state. Output of the complex query: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
DB
Messages printed to stdout by the migrations: nothing relevant. Output of Output of the complex query: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Results for the repo with the electride tests (migration finished, workflow tests pending).
Messages printed to stdout by the migrationsReport: Migrating to the head of the legacy sqlalchemy branch
Report: - 0edcdd5a30f0 -> 7536a82b2cc4
Warning: Detected repository folders that were missing the required subfolder `path` or `raw_input`. The paths of those nodes repository folders have been written to a log file: /home/framirez/Workenvs/aiida_qdens/aiida-test/migration-repository-missing-subfolder-3pf_su31.json
Warning: Migrated file repository to the new disk object store. The old repository has not been deleted out of safety and can be found at /home/framirez/Workenvs/aiida_qdens/.aiida/repository/aiida_qdens/repository.
Report: - 7536a82b2cc4 -> 1feaea71bd5a
Report: - 1feaea71bd5a -> 535039300e4a
Report: - 535039300e4a -> 34a831f4286d
Report: - 34a831f4286d -> 1de112340b16
Report: - 1de112340b16 -> 1de112340b17
Report: - 1de112340b17 -> 1de112340b18
Report: Migrating to the head of the main branch
Success: migration completedOutput of final SQL query('alembic_version', 'alembic_version_pkc', 0.0, '8192 bytes', '16 kB', 7)
('db_dbauthinfo', 'uq_db_dbauthinfo_aiidauser_id_dbcomputer_id', 2.0, '8192 bytes', '16 kB', 0)
('db_dbauthinfo', 'ix_db_dbauthinfo_dbcomputer_id', 2.0, '8192 bytes', '16 kB', 0)
('db_dbauthinfo', 'ix_db_dbauthinfo_aiidauser_id', 2.0, '8192 bytes', '16 kB', 0)
('db_dbauthinfo', 'db_dbauthinfo_pkey', 2.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'uq_db_dbcomputer_uuid', 3.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'uq_db_dbcomputer_label', 3.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'ix_pat_db_dbcomputer_label', 3.0, '8192 bytes', '16 kB', 0)
('db_dbcomputer', 'db_dbcomputer_pkey', 3.0, '8192 bytes', '16 kB', 1)
('db_dbgroup_dbnodes', 'uq_db_dbgroup_dbnodes_dbgroup_id_dbnode_id', 66027.0, '2992 kB', '1464 kB', 1)
('db_dbgroup_dbnodes', 'ix_db_dbgroup_dbnodes_dbnode_id', 66027.0, '2992 kB', '1464 kB', 2)
('db_dbgroup_dbnodes', 'ix_db_dbgroup_dbnodes_dbgroup_id', 66027.0, '2992 kB', '1464 kB', 0)
('db_dbgroup_dbnodes', 'db_dbgroup_dbnodes_pkey', 66027.0, '2992 kB', '1528 kB', 0)
('db_dbsetting', 'uq_db_dbsetting_key', 4.0, '24 kB', '16 kB', 0)
('db_dbsetting', 'ix_pat_db_dbsetting_key', 4.0, '24 kB', '16 kB', 0)
('db_dbsetting', 'db_dbsetting_pkey', 4.0, '24 kB', '16 kB', 0)
('db_dbcomment', 'uq_db_dbcomment_uuid', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'ix_db_dbcomment_user_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'ix_db_dbcomment_dbnode_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'db_dbcomment_pkey', 0.0, '0 bytes', '8192 bytes', 0)
('db_dblink', 'ix_pat_db_dblink_type', 623804.0, '43 MB', '17 MB', 0)
('db_dblink', 'ix_db_dblink_type', 623804.0, '43 MB', '17 MB', 0)
('db_dblink', 'ix_db_dblink_output_id', 623804.0, '43 MB', '13 MB', 2)
('db_dblink', 'ix_pat_db_dblink_label', 623804.0, '43 MB', '21 MB', 0)
('db_dblink', 'ix_db_dblink_label', 623804.0, '43 MB', '21 MB', 0)
('db_dblink', 'ix_db_dblink_input_id', 623804.0, '43 MB', '13 MB', 2)
('db_dblink', 'db_dblink_pkey', 623804.0, '43 MB', '17 MB', 0)
('db_dbuser', 'uq_db_dbuser_email', 3.0, '8192 bytes', '16 kB', 0)
('db_dbuser', 'ix_pat_db_dbuser_email', 3.0, '8192 bytes', '16 kB', 0)
('db_dbuser', 'db_dbuser_pkey', 3.0, '8192 bytes', '16 kB', 150)
('db_dbgroup', 'uq_db_dbgroup_uuid', 61.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'uq_db_dbgroup_label_type_string', 61.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_user_id', 61.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'ix_pat_db_dbgroup_type_string', 61.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_type_string', 61.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'ix_pat_db_dbgroup_label', 61.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_label', 61.0, '8192 bytes', '16 kB', 0)
('db_dbgroup', 'db_dbgroup_pkey', 61.0, '8192 bytes', '16 kB', 2)
('db_dblog', 'uq_db_dblog_uuid', 63512.0, '71 MB', '1984 kB', 0)
('db_dblog', 'ix_pat_db_dblog_loggername', 63512.0, '71 MB', '5048 kB', 0)
('db_dblog', 'ix_db_dblog_loggername', 63512.0, '71 MB', '5048 kB', 0)
('db_dblog', 'ix_pat_db_dblog_levelname', 63512.0, '71 MB', '1408 kB', 0)
('db_dblog', 'ix_db_dblog_levelname', 63512.0, '71 MB', '1408 kB', 0)
('db_dblog', 'ix_db_dblog_dbnode_id', 63512.0, '71 MB', '1408 kB', 2)
('db_dblog', 'db_dblog_pkey', 63512.0, '71 MB', '1408 kB', 0)
('db_dbnode', 'uq_db_dbnode_uuid', 300565.0, '335 MB', '9280 kB', 0)
('db_dbnode', 'ix_db_dbnode_user_id', 300565.0, '335 MB', '6616 kB', 0)
('db_dbnode', 'ix_pat_db_dbnode_node_type', 300565.0, '335 MB', '15 MB', 0)
('db_dbnode', 'ix_db_dbnode_node_type', 300565.0, '335 MB', '15 MB', 0)
('db_dbnode', 'ix_pat_db_dbnode_process_type', 300565.0, '335 MB', '9480 kB', 0)
('db_dbnode', 'ix_db_dbnode_process_type', 300565.0, '335 MB', '9480 kB', 0)
('db_dbnode', 'ix_db_dbnode_mtime', 300565.0, '335 MB', '6616 kB', 0)
('db_dbnode', 'ix_pat_db_dbnode_label', 300565.0, '335 MB', '8712 kB', 0)
('db_dbnode', 'ix_db_dbnode_label', 300565.0, '335 MB', '8712 kB', 0)
('db_dbnode', 'ix_db_dbnode_dbcomputer_id', 300565.0, '335 MB', '6616 kB', 1)
('db_dbnode', 'ix_db_dbnode_ctime', 300565.0, '335 MB', '6616 kB', 0)
('db_dbnode', 'db_dbnode_pkey', 300565.0, '335 MB', '14 MB', 598439) |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Results for the repo with legacy sssp.
Messages printed to stdout by the migrations: nothing relevant. Output of verdi storage info --statistics:entities:
Users:
count: 5
emails:
- xxx
Computers:
count: 14
labels:
- Bellatrix
- Daint
- Dora
- PizDaint
- PizDaint_XC50
- Pizdaint
- aries
- bellatrix
- deneb
- deneb1
- dora
- fidis
- pizdaint
- theospc6
Nodes:
count: 468701
node_types:
- data.core.array.ArrayData.
- data.core.array.bands.BandsData.
- data.core.array.kpoints.KpointsData.
- data.core.cif.CifData.
- data.core.code.Code.
- data.core.dict.Dict.
- data.core.folder.FolderData.
- data.core.remote.RemoteData.
- data.core.singlefile.SinglefileData.
- data.core.structure.StructureData.
- data.core.upf.UpfData.
- process.calculation.calcfunction.CalcFunctionNode.
- process.calculation.calcjob.CalcJobNode.
process_types:
- aiida.calculations:quantumespresso.ph
- aiida.calculations:quantumespresso.pw
Groups:
count: 599
type_strings:
- core
- core.import
Comments:
count: 0
Logs:
count: 0
Links:
count: 645132
repository:
SHA-hash algorithm: sha256
Compression algorithm: zlib+1
Packs: 13
Objects:
unpacked: 0
packed: 14184533
Size (MB):
unpacked: 0.0
packed: 49546.82000350952
other: 2392.01953125Output of final SQL query('db_dbcomputer', 'db_dbcomputer_pkey', 14.0, '16 kB', '16 kB', 0)
('db_dbcomputer', 'ix_pat_db_dbcomputer_label', 14.0, '16 kB', '16 kB', 0)
('db_dbcomputer', 'uq_db_dbcomputer_label', 14.0, '16 kB', '16 kB', 0)
('db_dbcomputer', 'uq_db_dbcomputer_uuid', 14.0, '16 kB', '16 kB', 0)
('db_dbgroup', 'db_dbgroup_pkey', 599.0, '64 kB', '32 kB', 4)
('db_dbgroup', 'ix_db_dbgroup_label', 599.0, '64 kB', '40 kB', 0)
('db_dbgroup', 'ix_pat_db_dbgroup_label', 599.0, '64 kB', '40 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_type_string', 599.0, '64 kB', '32 kB', 0)
('db_dbgroup', 'ix_pat_db_dbgroup_type_string', 599.0, '64 kB', '32 kB', 0)
('db_dbgroup', 'ix_db_dbgroup_user_id', 599.0, '64 kB', '32 kB', 0)
('db_dbgroup', 'uq_db_dbgroup_label_type_string', 599.0, '64 kB', '40 kB', 0)
('db_dbgroup', 'uq_db_dbgroup_uuid', 599.0, '64 kB', '40 kB', 0)
('db_dblink', 'db_dblink_pkey', 645132.0, '41 MB', '14 MB', 0)
('db_dblink', 'ix_db_dblink_input_id', 645132.0, '41 MB', '14 MB', 2)
('db_dblink', 'ix_db_dblink_label', 645132.0, '41 MB', '20 MB', 0)
('db_dblink', 'ix_pat_db_dblink_label', 645132.0, '41 MB', '20 MB', 0)
('db_dblink', 'ix_db_dblink_output_id', 645132.0, '41 MB', '14 MB', 2)
('db_dblink', 'ix_db_dblink_type', 645132.0, '41 MB', '17 MB', 0)
('db_dblink', 'ix_pat_db_dblink_type', 645132.0, '41 MB', '17 MB', 0)
('db_dblog', 'db_dblog_pkey', 0.0, '0 bytes', '8192 bytes', 0)
('db_dblog', 'ix_db_dblog_dbnode_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dblog', 'ix_db_dblog_levelname', 0.0, '0 bytes', '8192 bytes', 0)
('db_dblog', 'ix_pat_db_dblog_levelname', 0.0, '0 bytes', '8192 bytes', 0)
('db_dblog', 'ix_db_dblog_loggername', 0.0, '0 bytes', '8192 bytes', 0)
('db_dblog', 'ix_pat_db_dblog_loggername', 0.0, '0 bytes', '8192 bytes', 0)
('db_dblog', 'uq_db_dblog_uuid', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbnode', 'db_dbnode_pkey', 468195.0, '487 MB', '20 MB', 6)
('db_dbnode', 'ix_db_dbnode_ctime', 468195.0, '487 MB', '10 MB', 2)
('db_dbnode', 'ix_db_dbnode_dbcomputer_id', 468195.0, '487 MB', '10 MB', 0)
('db_dbnode', 'ix_db_dbnode_label', 468195.0, '487 MB', '10 MB', 0)
('db_dbnode', 'ix_pat_db_dbnode_label', 468195.0, '487 MB', '10 MB', 0)
('db_dbnode', 'ix_db_dbnode_mtime', 468195.0, '487 MB', '10 MB', 3)
('db_dbnode', 'ix_db_dbnode_process_type', 468195.0, '487 MB', '12 MB', 0)
('db_dbnode', 'ix_pat_db_dbnode_process_type', 468195.0, '487 MB', '12 MB', 0)
('db_dbnode', 'ix_db_dbnode_node_type', 468195.0, '487 MB', '23 MB', 16)
('db_dbnode', 'ix_pat_db_dbnode_node_type', 468195.0, '487 MB', '23 MB', 5)
('db_dbnode', 'ix_db_dbnode_user_id', 468195.0, '487 MB', '10 MB', 0)
('db_dbnode', 'uq_db_dbnode_uuid', 468195.0, '487 MB', '14 MB', 0)
('db_dbgroup_dbnodes', 'db_dbgroup_dbnodes_pkey', 487404.0, '21 MB', '10 MB', 0)
('db_dbgroup_dbnodes', 'ix_db_dbgroup_dbnodes_dbgroup_id', 487404.0, '21 MB', '10 MB', 0)
('db_dbgroup_dbnodes', 'ix_db_dbgroup_dbnodes_dbnode_id', 487404.0, '21 MB', '10 MB', 2)
('db_dbgroup_dbnodes', 'uq_db_dbgroup_dbnodes_dbgroup_id_dbnode_id', 487404.0, '21 MB', '10 MB', 2)
('db_dbsetting', 'db_dbsetting_pkey', 3.0, '8192 bytes', '16 kB', 0)
('db_dbsetting', 'ix_pat_db_dbsetting_key', 3.0, '8192 bytes', '16 kB', 0)
('db_dbsetting', 'uq_db_dbsetting_key', 3.0, '8192 bytes', '16 kB', 0)
('db_dbauthinfo', 'db_dbauthinfo_pkey', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbauthinfo', 'ix_db_dbauthinfo_aiidauser_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbauthinfo', 'ix_db_dbauthinfo_dbcomputer_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbauthinfo', 'uq_db_dbauthinfo_aiidauser_id_dbcomputer_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'db_dbcomment_pkey', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'ix_db_dbcomment_dbnode_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'ix_db_dbcomment_user_id', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbcomment', 'uq_db_dbcomment_uuid', 0.0, '0 bytes', '8192 bytes', 0)
('db_dbuser', 'db_dbuser_pkey', 5.0, '8192 bytes', '16 kB', 0)
('db_dbuser', 'ix_pat_db_dbuser_email', 5.0, '8192 bytes', '16 kB', 0)
('db_dbuser', 'uq_db_dbuser_email', 5.0, '8192 bytes', '16 kB', 0)
('alembic_version', 'alembic_version_pkc', 0.0, '8192 bytes', '16 kB', 5) |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
DB Note: This DB contains few nodes, but each of them contains large arrays, totalling ~20GB of disk space)
Messages printed to stdout by the migrations: nothing relevant. Output of Output of the complex query: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
DB Note: This DB contains a very large number of (StructureData) nodes and some more trajectories and other stuff.
Messages printed to stdout by the migrations: many (~2000) repository folders that were missing the required subfolder Output of Output of the complex query: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
For what concerns me, I think I'm done with running migrations! All looks working fine, thanks a lot to all and especially @chrisjsewell and @sphuber !! Note: I didn't manage to run with it as the servers are all down. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Now that the |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
One last thing you can try if you still have access to your migrated profiles, is to run the maintenance on them. If you already backed up everything before the migration you don't need any other preparation, but if you only backed up the database you may want to now also create a backup of the repository before proceeding (if you have the space, the easiest is to just make a copy of your For this I would say:
You can report the following:
Afterwards you can also try re-running any of the "after migration" tests you may have used to check the consistency of your storage. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
This discussion is intended to collect the reports on testing the migrations of existing databases with
v2.0.0a1which is the current state ofdevelop. Instructions of setting up your environment, performing the migration and what should be reported can be found on this Wiki page. For each database that you migrate, please create a separate comment in this discussion thread using the following template:verdi storage info --statisticsIf you perform any manual data inspection after the migration is completed, please report them as well if you notice anything out of the ordinary.
Beta Was this translation helpful? Give feedback.
All reactions