[Bug]: AP_ID with 'http://changeme.invalid/' + 24 chars

[freamon]

Requirements

• Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
• Did you check to see if this issue already exists?
• Is this only a single bug? Do not put multiple bugs in one issue.
• Do you agree to follow the rules in our Code of Conduct?
• Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

I have been receiving activities for which my server hasn't been able to fetch the relevant object, and have finally tracked one down.

The activity was:

{
  "@context": [
    "https://join-lemmy.org/context.json",
    "https://www.w3.org/ns/activitystreams"
  ],
  "actor": "https://lemmy.world/c/nostupidquestions",
  "cc": [
    "https://lemmy.world/c/nostupidquestions/followers"
  ],
  "id": "https://lemmy.world/activities/announce/like/c1122f5d-b322-4322-a4f1-1c7fca9e935b",
  "object": {
    "actor": "https://lemmy.world/u/5too",
    "audience": "https://lemmy.world/c/nostupidquestions",
    "id": "https://lemmy.world/activities/like/f9b8126f-c8ee-40ae-86e2-cd29cba9f22b",
    "object": "http://changeme.invalid/4bcffd164a915f319fc412df",
    "type": "Like"
  },
  "to": [
    "https://www.w3.org/ns/activitystreams#Public"
  ],
  "type": "Announce"
}

I searched for that object on LW, and it returned a comment: https://lemmy.world/search?q=http%3A%2F%2Fchangeme.invalid%2F4bcffd164a915f319fc412df&type=All&listingType=All&page=1&sort=TopAll

Requesting that comment via the API returned that it had that URL for its ap_id:
curl https://lemmy.world/api/v3/comment?id=15207268 | jq .comment_view.comment

{
  "id": 15207268,
  "creator_id": 2509869,
  "post_id": 25814241,
  "content": "Wild Tigger \n![](https://lemmy.world/pictrs/image/39e9d683-b28c-47a6-9171-1967cf31b798.gif)",
  "removed": false,
  "published": "2025-02-20T13:05:25.148941Z",
  "deleted": false,
  "ap_id": "http://changeme.invalid/4bcffd164a915f319fc412df",
  "local": true,
  "path": "0.15206320.15207268",
  "distinguished": false,
  "language_id": 0
}

Steps to Reproduce

1. curl https://lemmy.world/api/v3/comment?id=15207268 | jq .comment_view.comment

(EDIT: I don't have the activities for these, but have also discovered):
2. curl https://lemmy.world/api/v3/comment?id=15208607 | jq .comment_view.comment
3. curl https://lemmy.world/api/v3/comment?id=15186242 | jq .comment_view.comment

Technical Details

The ones I've found so far are from LW accounts, and they're a bit funny about staying up to date, so hopefully this is a bug that you already know about and is fixed (I couldn't find any other relevant Issues though).

Version

BE: 0.19.3-7-g527ab90b7

Lemmy Instance URL

lemmy.world

[dessalines]

0.19.3 was released in january of last year.

But also I can see that in crates/routes/src/utils/code_migrations.rs , we're updating incorrect changeme% ap_ids since 2020. So unless those scheduled jobs are getting skipped, I don't know what could be causing it.

[Nutomic]

That changeme url is the default value for the database field, which gets updated by a trigger to the correct id. Seems like the trigger was not executed for some reason.

cc @MrKaplan-lw

[MrKaplan-lw]

That's odd.
We haven't disabled any code migrations and as far as I know we haven't touched triggers in the DB.

It also doesn't seem to be incorrect anymore at this point, select count(*) from comment where ap_id like 'http://changeme.invalid/%' returns zero matches and the examples above are no longer returning changeme.invalid. The search also doesn't have any results anymore.

I haven't checked the code yet, were there cases in 0.19.3 where this was changed outside of the insert transaction? Since triggers are synchronous within the DB transaction those IDs should then never be exposed to another query. If this was the case in 0.19.3, what about 0.19.9? Does 0.19.9 have logic where these IDs would exist temporarily outside of the insert transaction?

[freamon]

I got another one, as it was Announced this time, rather than my server trying to fetch something that was Liked or replied to.

It has the correct AP_ID now - it's at https://lemmy.world/comment/15251256, but when it was Announced, the IDs were:

ap_id          | http://changeme.invalid/6cf49300a3579814ab9b46d3
ap_create_id   | https://lemmy.world/activities/update/c9fe1cc5-dcc1-4b0d-9448-f91340a15e2a
ap_announce_id | https://lemmy.world/activities/announce/update/adbacd65-5ead-4a45-a3d8-4b7eb19fd781

My guess would be that other Lemmy servers initially rejected it, but it had internally fixed itself by the time other people replied to it, so they were then able to fetch it (it looks like it was fetched, because the score is 0, but no-one has downvoted it)