Core: Lazy loading experiment

[Mysteryem]

What is this fixing or adding?

Implements lazy loading by reading the game for each world from an ap_info.json (similar to the archipelago.json suggested in #4516), intercepting worlds.<world> module loading, and allowing worlds to be loaded by game name.

There is currently no intention to merge this PR, it has been created to aid with discussion surrounding the problem of lazy loading. I think there is more planning and design discussion required before lazy loading can be properly implemented into Archipelago.

• worlds.ensure_worlds_loaded(): Load worlds by game name or load all worlds
• worlds.ZipWorldLoader, worlds.SourceWorldLoader and worlds.BytecodeWorldLoader implement the module loading
  • The latter two are not strictly required because Python can load them fine on its own, but using them provides easier module execution benchmarking
• worlds.WorldFinder is an importlib.abc.MetaPathFinder, it is responsible for handling attempts to import worlds.<world> modules that have not been loaded yet, and picking the correct Loader for the world.
  • The WorldFinder functions by being prepended to sys.meta_path: sys.meta_path.insert(0, WorldFinder()). This is necessary to intercept loading of non-zipped worlds when running from source, but it can be appended to sys.meta_path instead to only handle loading of zipped worlds, which might be better for frozen builds (assuming things actually work on frozen builds)

For how module importing/loading works in Python, I would recommend reading through most of https://docs.python.org/3/library/importlib.html. I did not know much about module importing/loading before I started working on this PR and learned pretty much everything necessary to implement this PR from that documentation.

Known broken things

• meta.yaml parsing is broken
  • This happens before game yamls are parsed to determine what worlds need to be loaded. I have not looked into how to fix this, nor do I know how broken it is.
• Worlds must not be loaded while generation is ongoing on the same process
  • Loading a world can add to CollectionState.additional_copy_functions and CollectionState.additional_init_functions, so loading a world while generation is ongoing can add a copy function and then attempting to copy multiworld.state will fail because the attributes the copy function was expecting to be created by the init function do not exist.
  • Multiple generations on WebHost occurring on the same process is assumed to be broken for this reason
  • Generation (Main.main) and worlds.ensure_worlds_loaded() most likely need to use locks to prevent concurrent generations on the same process, as well as anything else on the same process that might try to load worlds. Care will be needed with implementing this to ensure that deadlocks cannot happen.

General information

• The games specified in player yamls are what load the required worlds in generation. If an unknown game is found, all worlds that did not provide an ap_info.json are attempted to be loaded to try to find a matching world.
• Launcher.py has to load every world so that the clients in the launcher get populated
• Opening clients from the launcher works because importing a world module at any point will load that module
• Opening a patch file with the launcher attempts to read the game from the archipelago.json within the patch for patches that use Patch.create_rom_file(), and then load the world for that game
• Patches that use APPatchExtension/AutoPatchExtensionRegister already provide their game, so it is easy to then ensure the world for that game is loaded
• ALTTP always gets loaded due to core Archipelago still being intertwined with it
• I don't understand a lot of the WebHost code and how it connects together with what is run on separate threads/processes. Much of WebHost has been slapped with a "stick worlds.ensure_all_worlds_loaded() at the start and pray"
• Lots of tests had to be modified to load all worlds. It's possible some tests have been missed.

Potential improvements

• Actually scanning through the world folders (worlds.scan_for_worlds_on_module_load()) should be deferred until the first attempt to load a world
• If a world does not provide an ap_info.json it is probably possible to determine the package of a loaded world class and create the ap_info.json for that world

How was this tested?

Unit tests have been run (note that if I missed updating some tests that iterate through all worlds, those tests would now be iterating no worlds, so would probably succeed and test nothing).

Seeds have been generated locally.

Solo seeds have been generated on webhost and a solo seed has been played a small amount while connected to a webhost hosted room. This solo seed also used a game with a patch file, so that was tested on the client side too.

worlds.BytecodeWorldLoader was tested by replacing the Clique source files with a copy of the Clique world from a frozen build of AP 0.5.1.

Frozen builds have not been tested at all.

[Mysteryem]

As mentioned in the PR description, for WebHost, Main.main (generation) and worlds.ensure_worlds_loaded probably need to use a lock to properly prevent world loading while a generation is ongoing on the same process.

worlds.world_loading_enabled is also currently not set back to True anywhere.