"Addon" archive support

[iainlane]

For things like PPAs, (eventually) bikesheds, ISVs and so on - we should support archives which are "on top" of suites we already know about.

I think the problem here is mainly about stock icons?

The workflow should make it easy for both the central instance and random external instances to operate in this mode - the former for PPAs and the latter for ISVs. Maybe this is the same though - acquire the preprocessed data either by downloading it or having it on the filesystem, generate a configuration, and run.

[ximion]

Yes, this should be a solvable issue - the only thing that we would need from the main archive would be the icon tarball.
Since each instance of asgen has its own database (one asgen per repository) and doesn't communicate with instances for other repos, this is a tricky issue to solve.
I wonder if it should be resolved in a way that we just tell the new repo "pull iconset with this name from repo X" instead of overlaying suites like we do when stuff is in the same repository.

[iainlane]

We had some small thoughts

• Process these in a central asgen instance which knows about the main archive already - could be like another suite
• PPA-specific: PPAs can depend on other PPAs
• Launchpad would prefer to use webhooks to notify the central asgen of archives that need processing; would need a small webapp to receive these
• There could be thousands of archives; scaling issue so make asgen more cloudy to distribute it (would imply losing lmdb in favour of mongo or something)
• Not sure yet how to deliver the results back

[ximion]

Sounds good, PPA dependencies will likely be the trickiest thing.

[z3ntu]

I believe this is also what's missing for postmarketOS (based on Alpine Linux). We currently have this config:

Expand

{
"ProjectName": "postmarketOS",
"ArchiveRoot": "http://mirror.postmarketos.org/postmarketos/",
"Backend": "alpinelinux",
"Features":
  {
    "validateMetainfo": false,
    "createScreenshotsStore": false,
    "noDownloads": false
  },
"Suites":
  {
    "v20.05":
      {
        "sections": ["postmarketos"],
        "architectures": ["aarch64", "x86_64"]
      }
  }
}

Our repos are used in conjuction with the Alpine Linux repos so for example we don't have the breeze icons in our repo but the package from Alpine gets used for that. Unfortunately appstream-generator can't handle that so a lot of 'applications' have no icons in the metadata. @Cogitri had the idea that the baseSuite parameter could be used for that but we'd need to somehow specify the path to an Alpine mirror for that in the suite there.

[ximion]

@z3ntu You could inject the icon themes via the extradata fake package: https://github.com/ximion/appstream-generator/blob/master/docs/asgen-config.md#injecting-extra-metainfo--removing-components - you would just need to keep that up to date.
You definitely want a baseSuite but asgen currently assumes all suites are on the same mirror for that to work.

"Features":
  {
    "validateMetainfo": false,

You really should leave the validation on to get notified of issues and deprecations early - it's also a great debugging aid. The only reason to ever disable this is when the data already gets generated elsewhere and you want a tiny bit of extra performance (validation is really quick now though).

    "createScreenshotsStore": false,
    "noDownloads": false
  },

You could remove the noDownloads line, downloads are always enabled unless explicitly disabled.

[Cogitri]

At least on Alpine validating the metainfo doesn't work due to #83.

[z3ntu]

@ximion If that's supposed to work, I need an example on how to do that. I don't understand the documentation enough to figure that out myself.

[ximion]

@ximion If that's supposed to work, I need an example on how to do that. I don't understand the documentation enough to figure that out myself.

Create a directory structure extra-metainfo/<suite>/<section>/icons/ in your workspace directory (e.g. extra-metainfo/bullseye/main/icons/ - symlink shenanigans are allowed if that fake package should be prresent with the same contents in multiple suites) and then drop your additional icons into that directory - make sure to follow the directory structure of the hicolor icon theme (so <size>/<section> subdirectories, like scalable/apps/myicon.svgz or 64x64/apps/myapp.png).
I haven't tested this but this should result in icons being displayed (it is a quite a hack though ;-) )

[z3ntu]

@ximion I don't think the icon from e.g. extra-metainfo/v20.05/postmarketos/icons/scalable/apps/camera-photo.svg gets picked up though. Did I understand you correctly? I do see the log line INFO: Loading additional metainfo from local directory for v20.05/postmarketos/aarch64 so the first couple of folder names should be correct.

[ximion]

Yes, that's correct. Most likely in order for this to work, you would require an actual icontheme to be present and loaded, which is not supported. I'll have a look at how hard it would be to implement this.