Migrations: Better support for complex, nested configuration properties

[Taitava]

This is related to how new fields are added to the data.json settings file after SC version upgrades. So it's not so visible to users.

Currently, migrations work by checking if the settings file lacks some newer fields, and creates the fields if they are missing. Some old fields are also renamed and their data format is changed. So far this has been working quite well, at least I haven't gotten any reports about problematic migrations.

0.12.0 will add more properties to data.json:

• prompts (main scope)
  • fields (prompt scope)
• custom_variables (main scope)
• preactions (shell command scope)
• variable_default_values (shell command scope)

In the future, there can be different types of prompt fields, and different kinds of preactions. All of the new types will have partially different sets of configuration properties, making it not so easy for the migration process to determine what felds should be present.

#151 will provide a mechanism to import shell commands, custom variables and prompts from external sources. This importing needs to utilize the migration system. If an imported element comes from an older version of SC, it's structure needs updating. Also, to keep exports neat and tidy, they should omit all features/properties that are not used. Those properties need to be re-added at the importing phase. But this discussion can be continued in #151.

I don't have a plan for a better migration system yet, but I'll try to think about it.

A todo list of features that don't have a complete migration support

• Make sure configuration migrations can add new fields to Prompts and PromptFields.
• Make sure configuration migrations can add new fields to CustomVariables.
• Preactions in shell command scope.

I can't really do development that adds new configuration fields to any of these mentioned features, until I've got the migration system to handle them, and I don't want to postpone 0.12.0 just because of developing migrations that won't be used immediately.

[Taitava]

Write migration objects

I think the best design for migrations is to create one migration object for each version upgrade. Here is a nice, explorative article about developing a system for migratin JavaScript objects: https://levelup.gitconnected.com/an-approach-to-javascript-object-schema-migration-ae1bd9f0ce78

Some points that I'd like to do differently than in the article:

• Don't create a new schema object during each migration. Instead, modify the passed schema object directly. Schema objects should be passed to migration functions as arguments.
• Make it possible to run migrations with narrowed data, too. Usually, it's needed to migrate the complete settings object, but when Import / Export Commands #151 is implemented, the migrations might face single pieces of configuration (such as a single shell command, or just one main level setting).

Test the migrations

After a migration engine is created, and migration objects are written for each version, also test versions of settings files are needed.

Create a git repo that contains settings files named like this:

• conservative/data-0.0.0.json
• conservative/data-0.1.0.json
• ...
• conservative/data-0.17.0.json
  All the settings files would contain a valid sample configuration. Then it's easy to test migrations, can for example pick a really old version 0.0.0 and test that the migration chain will produce exactly the same 0.17.0 version of settings that defaults/data-0.17.0.json has.

There will be multiple folders. The conservative folder will contain a settings file that naturally evolves from 0.0.0 onwards without ever starting to use new features. As there will be no user-added content along the version upgrade path, the whole upgrade chain can be tested automatically from 0.0.0 to the newest version. If the settings file would have gotten e.g. a Prompt record (created by a user) in version 0.12.0, then there's no way for a migration test to come up with a Prompt record when trying to create a similar settings file from an older settings file version.

Other folders could have a similar idea, to refrain from adding new content to the settings file after the first settings file version in that folder, but instead concentrate on making sure that the particular set of content present in the first settings version of that folder is able to survive all the way to the newest version. This way there could be e.g. a folder named prompts that would start from version 0.12.0 and aim to test that all Prompts will be correctly migrated.