feat: Verified mods auto-downloading

[Alystrasz]

This pull request introduces automatic mods fetching.

NorthstarMods associated pull request: R2Northstar/NorthstarMods#542

---

This allows for the client to download and install mods that are required by a server (client+server-side mods, like new game mods or maps for instance). List of verified mods is local (compiled in the dll), but it can be fetched from master server in the future, to ease the addition of new verified mods.

With this feature, if users want to join a server requiring a gamemode mod or a map mod, their game can automatically download it for them, instead of forcing them to go through manual mod installation process (which can be hard for some people). While downloading a mod, progress download is displayed and live-updated on a dialog pop-up.

Verified mod entries

A verified mod entry looks like this:

{
    "Fifty.mp_frostbite": {
        "DependencyPrefix": "Fifty-Frostbite",
        "Versions": [
            { "Version": "0.0.1", "Checksum": "99fc4f0b3363e44985f20525a2bf079..." },
            { "Version": "0.0.2", "Checksum": "40780308e0e4a70f7f35f50f2ca188a..." }
        ]
    }
}

Versions are checked individually to ensure no malicious code has been added during mod updates.

Please note that it is up to the community to decide what's needed for a mod to be "verified", and how such mod is "verified".

Discussion here: https://github.com/R2Northstar/VerifiedMods

Mod installation process

1. Check if mod is verified
   a. Check if it is listed
   b. Check if version is listed
2. Download mod archive from Thunderstore
   a. Fetch zip from API
   b. Compare checksum with local checksum
3. Unzip mod archive content directly into game folder
4. Reload mods list
5. Remove downloaded archive

Please note that downloaded archive is checksumed, to verify it is the very archive that has been verified.

Changes

Northstar.Client

• OnServerSelected method becomes async, to allow waiting for mod verification;
• _OnServerSelected method now displays some dialogs:
  • One error message when one attemps to join a server requiring a mod which cannot be auto-downloaded;
  • One loading message, displaying while a verified mod is being downloaded;
  • One error message when mod downloading/extraction failed;
• If a needed mod is already present but disabled, this activates it.

Launcher

• Includes a new library, libzip
• Includes the already-included but not linked openssl lib;
• All verified-mods-related code is stored in verifiedmods.cpp
• Squirrel-exposed methods:
  • bool NSIsModVerified(string modName, string modVersion): tells if a mod is verified or not;
  • void NSDownloadMod(string modname, string modVersion): starts downloading a mod; returns nothing because actual mod downloading is done in a thread, as not to block UI;
  • bool NSIsModBeingDownloaded(string modName): tells if a mod is being downloaded; this is used by UI to check if it should display loading dialog to the user;
  • float NSGetCurrentDownloadProgress(): returns progression in percentage of current download/extraction;
  • string NSGetModExtractionResult(): returns result of mod download/extraction;

Testing

I tested this PR with two mods:

• Juicy.Zircon_Spitfire, a HUGE (1.7GB) skin mod (set as client-required for the sake of testing this);
• Fifty.mp_frostbite, a map mod.

Maps mods are a good example of mods that would benefit from auto-downloading, because they are required on both client and server sides.

Media

If mod is not verified

If something goes wrong during mod download/extraction

If mod is verified

verified_download_s2space.webm

TODOs

• Build issues (to make this work, I had to put zip.dll, bzip.dll and libzip.dll into game folder, but those are not generated sometimes, gotta find why)
  • Maybe integrate miniz instead of libzip
• Add libzip licence (will do once this is actually merged)
• Handle errors while downloading/extracting
• Verify checksums
• Test with bigger mods (we may need a progress indicator for big maps for instance)
  • Test with bigger mods
  • Add a progress indicator
• Clean downloaded files
• Mod managers support
  • Save manifest.json and put it in mod directory
  • Put a "AutoDownloaded": true entry into mod.json

[ASpoonPlaysGames]

failing build D:

[Alystrasz]

Might make sense to set the server to a mod where being able to join implies that mod download worked. I think the LTS community have a bunch that are required on client and server. Maybe https://northstar.thunderstore.io/package/Dinorush/DinorushBrute4/ ?

I added this mod both to the client list of verified mods and to the list of required mods on my server, so it can be auto-downloaded.

Flipside wouldn't be included because its only a server side mod. In this case it doesn't matter since its server side.
but if I where to make my mods depend on client side utils mods, that would produce compile errors

This will definitely be an issue. Should we force including mods' dependencies into required mods server lists as well?

[EladNLG]

IMO, a simple solution for this would be "for your mod to get verified, it's dependencies also have to be verified."

[GeckoEidechse]

IMO, a simple solution for this would be "for your mod to get verified, it's dependencies also have to be verified."

Agreed. We should check both when verifying the mod as well as in launcher when it pulls in the mod dependencies in case we had a slip up.

[Alystrasz]

Agreed too!
I'll add this rule to verified mods repo.

[Alystrasz]

This is now up-to-date with main branch.
Here is an example of an auto-downloaded (single-player reused) map.

verified_download_s2space.webm

[GeckoEidechse]

Basically all that is missing for getting this PR merged is statically linking the necessary dlls into northstar.dll, right?

[Alystrasz]

Basically all that is missing for getting this PR merged is statically linking the necessary dlls into northstar.dll, right?

Yep!
@pg9182 mentioned using miniz instead of libzip to address this, so I'd like to try.

[Alystrasz]

Ehm, actually, miniz does not expose the same API as libzip at all...
Would including libzip's .c and .h files solve this issue?

[Alystrasz]

Waiting for #500 to be merged before solving conflicts.

I plan to push the parkour gamemode through the verification process, and use it as the final example mod for this feature.

[GeckoEidechse]

For testing ones own mod it might be useful to have like a command line arg to bypass verification and allow downloading any mod from Thunderstore if the server requires it. I'm fine with having a separate PR for this though afterwards. Just something that came to mind rn ^^

[Alystrasz]

For testing ones own mod it might be useful to have like a command line arg to bypass verification and allow downloading any mod from Thunderstore if the server requires it.

I agree on the principle, however we still need to link mod names to related Thunderstore packages, which can only be done manually...

[GeckoEidechse]

For testing ones own mod it might be useful to have like a command line arg to bypass verification and allow downloading any mod from Thunderstore if the server requires it.

I agree on the principle, however we still need to link mod names to related Thunderstore packages, which can only be done manually...

Outside of the scope of this PR but with packages we can start parsing Thunderstore stuff natively in Northstar which in turn we could use to expose the required Thunderstore package next to the required mod on server 👀

[Alystrasz]

Closing this in favour of a new branch.

[Alystrasz]

New version: #545