.desktop ID divergence between manual data and generated data

[hsitter]

For quite a while in neon we've been observing inconsistencies between desktop ids the same entity (e.g. org.kde.kdenlive) would sometimes be available as ID: org.kde.kdenlive and other times as ID: org.kde.kdenlive.desktop and as far as neon is concerned it would always have a superset of both.

The reason appears to be that when asgen autogenerates data from the desktop file it picks ID: org.kde.kdenlive whereas the actual upstream information is declaring it ID: org.kde.kdenlive.desktop client-side these two are not treated as equal resulting in duplicated entries.

Either the client should treat them as the same entity or the autogenerator ought to append .desktop I suppose.

$ appstreamcli search org.kde.kdenlive        
Identifier: org.kde.kdenlive [desktop-application]
Name: Kdenlive
Summary: Nonlinear video editor by KDE
Package: kdenlive
Icon: kdenlive_kdenlive.png
---
Identifier: org.kde.kdenlive.desktop [desktop-application]
Name: Kdenlive
Summary: Nonlinear video editor for KDE
Package: kdenlive
Icon: kdenlive_kdenlive.png

[ximion]

Does Kdenlive have a metainfo file?
The rule is:

• If there is a metainfo file, always use the ID from the <id/> tag, no exceptions
• If there is no metainfo file, set the .desktop file name without the .desktop suffix as ID, and add a <launchable/> tag with the full desktop-id, in order to be able to launch the application.

If these rules are followed, this is not a bug. If the rules aren't followed, it is an issue.

[hsitter]

Mh, perhaps this ought to be tracked as an appstream issue then.

Considering it is possible that both options are meant to be generatable by the generator (org.kde.kdenlive and org.kde.kdenlive.desktop), albeit in different conditions, the client side of appstream ought to know that they are the same and treat them as one and the same.

e.g. consider a composite metadata set with part of the data coming from the distribution and the other part from an extension repo.
The base could contain org.kde.kdenlive, the extension repo could contain a newer version of the app which now has proper data and identifies as org.kde.kdenlive.desktop. From a client side perspective the fact that they have different ids should not matter. There is only one org.kde.kdenlive.desktop, one just happens to have been autogenerated. The output I pasted above should only contain one entry, the most suitable one.

So, there are two options as I see it: the client should deal with this by "merging" the two options into one and picking the highest priority one, or the generator should deal with this and not assign different identifiers based on where the data comes from.

[ximion]

So, there are two options as I see it: the client should deal with this by "merging" the two options into one and picking the highest priority one

This is something that should already happen, given that data is also collected from the applications directory directly.
The absolutely easiest way to prevent this issue is to add a metainfo file to projects, because then the ID is fixed and will never change.

I will look into AppStream itself later and see if there are things that can be improved. You can I think also prevent additional heuristics for component-ID naming based on .desktop files by having ascli enforce an older AppStream spec version (there is a flag for that in its configuration)

[ximion]

@apachelogger Since this is neon, where is the second Kdenlive coming from? The Ubuntu data (in which case this one isn't a bug) or the data installed on the system (in which case this is a bug in AppStream that should be fixed as soon as possible)?

[hsitter]

Ubuntu I think.

With sources > 2 there's always potential for the same "thing" having been autogenerated in one while being properly curated in the other resulting in the name divergence as asgen generates "foo" whereas the curated version would be "foo.desktop" as seen in the output I pasted in the initial report. That's not specific to Neon or it's relationship to Ubuntu.

The bug isn't that there are two of them on a raw data level, but rather that libappstream does not know that they are the same. Hence why I am now thinking at this bug may, in fact, need dealing with there rather than asgen.

e.g.
User Annie uses Ubuntu with KDE Plasma. She finds my application 'foobar' and installs it. Ubuntu is packaging foobar, so she gets Ubuntu's version. Unfortunately, that version had no appstream data so it got the auto generated by asgen based on the desktop file. It is available in the appstream cache as id: foobar.
So far so good.
Now Ubuntu being a frozen release distribution at some point Annie may find that foobar is having defects which are fixed in a newer version. Since I am a pro-active appstream she has no trouble finding a newer deb provided by me. She installs it. As is common practice that also injects a deb repo for my application so it now gets automatically updated through the regular channels (e.g. spotify and chrome do that).
At this point foobar has a second appstream data source!
The new version from my repo has a manually curated metadata file and as such my repo supplies a id: foobar.desktop rather than the autogenerated id from Ubuntu.
When Annie now searches for 'foobar' in discover or gnomesoftware she will get two entries for the actually same app. Only difference being in the raw data set one has an id with .desktop and the other does not.

[ximion]

@apachelogger I understand that, the problem is though that I would need quite a bunch of deduplication queries for this to work in all cases (component-id, component-id+".desktop" suffix, desktop-entry launchable) and I am worrying a bit about whether the rarity of such events warrants sacrificing memory and/or CPU cycles every time we load the data (it's less bad due to the cache we create after the first load, but still).

Instead, I am thinking addressing issue #29 is maybe time better spent, so Neon (and Ubuntu itself) can override the data of immutable suites easily. Doing that would nullify this issue, and also fix the problem of wanting to update data after a repository was frozen.

[hsitter]

Ah yes fixing #29 would solve it nicely as well. Closing in favor of that.

FTR though: if asgen actually called it 'foobar.desktop' the issue would also be non-existant. Although I expect there is a reason for it dropping the .desktop.

[ximion]

Asgen will generate an AppStream component internally, and make up a new component-id for that. So, if it has enough data to make a good component-id it will prefer that. Which means that if your .desktop file starts with <tld>.<domain>., like in this case, asgen will think the most likely component-id an app author would choose for this is that name minus the .desktop suffix.
If it doesn't manage to find a good ID, the desktop-filename is used.
If a metainfo file is present, the value of the <id/> tag will always be used, no matter what.
So, you could actually also "fix" this by giving Kdenlive a metainfo file.

[hsitter]

Oh ah, I actually thought the preferred form was the desktop file id 'org.kde.kdenlive.desktop' rather than basename. The other way around it makes more sense that asgen drops the suffix of course. Not sure how I got that impression.

[ximion]

@apachelogger In older version of AppStream, the <id/> tag for desktop-applications had to contain the .desktop file ID. Today, the ID just has to be a reverse-DNS string, and desktop-apps should have a <launchable/> tag instead to indicate if and how they can be launched (libappstream creates one automatically though if the ID ends with ".desktop").

[iainlane]

On Mon, Oct 02, 2017 at 01:36:26PM +0000, Matthias Klumpp wrote: Instead, I am thinking addressing issue #29 is maybe time better spent, so Neon (and Ubuntu itself) can override the data of immutable suites easily. Doing that would nullify this issue, and also fix the problem of wanting to update data after a repository was frozen.

I don't understand how #29 will fully fix this issue - if the new asgen still generates a different ID to the one the external repo supplied, it'll be a duplicate then too? If you fix #29 by masking out the overridden suite completely instead of merging then it fixes instances of this where the distro and external repository always generate the same IDs (e.g. when they use close enough versions of asgen), but I don't think it fixes all of them. If that's right, then probably client libraries or frontends need fixing for this too. A similar problem already exists when there's a limba/appimage/snap/flatpak of an application as well as a deb, I think, and they should handle that in one way or another for a good user experience. (Although those are probably more likely to have the same ID everywhere, unless the metainfo turned up after the distro was released.) Cheers,

…

-- Iain Lane [ [email protected] ] Debian Developer [ [email protected] ] Ubuntu Developer [ [email protected] ]

[ximion]

Yeah - there is a reason why hughsie disabled extracting data from .desktop files alltogether, it makes things simpler.
Since the way IDs are generated is very predictable though, I don't think we will see this issue much except for the cases presented in this bug report.
The more severe issue is actually apps changing .desktop filenames and having no metainfo file - that messes up things a lot, and there is no way we can fix that.