Ability to specify uuid when capturing events

[evgeek]

Hello!

We really miss the ability to specify the uuid of events manually when capturing events.
The thing is that we cannot guarantee that events come to our system only once. In the current implementation, in such cases, events in Posthog will be duplicated.
In the SDK in other languages, we got around this by manually generating the uuid, calculated based on the immutable fields of the event - however, in the Golang SDK, unfortunately, there is no such option yet.
Please consider the PR adding this functionality.

P.S. As far as I can see, the current implementation does not use the posthog.Config.uid() function, so I removed it.

[evgeek]

@fuziontech @dmarticus Hi guys!
Sorry to bother you, but maybe you can help me with this? :)

I checked the functionality on our project - everything works perfectly well!

[dmarticus]

A few comments about versioning and the changelog, but the implementation itself looks solid. Thanks for the contribution!

Before I approve this change though, I'd like to understand a bit more about the motivation and use case. Right now, PostHog auto-generates UUIDs for given events when we ingest them, and there shouldn't be any risk of event duplication, since we also dedupe events as part of our ingestion pipeline. Generally, PostHog clients aren't responsible for assigning UUIDs to events themselves – as far as I know, we don't support specifically passing in UUIDs in any of our other SDKs (Python example.

What's the use case here that you're solving for that can't be handled by just passing the UUID in through the properties payload? Ideally, you shouldn't be managing event uniqueness at all on your side when sending things through PostHog.

[dmarticus]

@evgeek this file is auto-generated, you don't need to make edits to it.

[evgeek]

+

[dmarticus]

same note here; we auto-bump the version when we deploy.

[evgeek]

+

[evgeek]

We have bunch of projects that send events. They save them in a DB, and then send them to different services via AMQP (https://github.com/umbrellio/table_sync), where they are also saved in a DB. These messages can be sent multiple times for different reasons: from updating a row in a DB to forcing sync of some events. In our services, we handle retries using message versions (old messages are ignored, new ones update previously written data by primary key). But sending to Posthog is different from saving to a DB because of updating unavailable, and we have to ensure that a retried AMQP message does not cause new events. We can save the id of the sent events in a DB, but this will cause unnecessary overhead: creating and managing a new DB, slowing down the system due to IO, resolving race conditions when multiple k8s pods try to access the same row, etc. Or we can just calculate a reproductible UUID - in this case, Posthog will simply skip the re-sent event. Very simple and elegant, I belive :)

About deduplication on ingestion: This is a great feature, but since our events are stored in a database and can be updated, it may not always recognize duplicates. And I'm not sure how dedupe will work if the same event is sent on different days.

About other SDKs - I haven't tried them all, but PHP definitely allows it, we used the same approach in the previous version of the event service.

[dmarticus]

@evgeek thanks for your thorough reply! I understand your situation better now. I think your perspective makes sense, and frankly it makes me think that all of our SDKs should let consumers supply optional UUIDs. Let's get this change merged, thanks for doing it!

[evgeek]

Excellent! Thanks for your responsiveness and awesome product! :)

[migueldv90]

I think this pull request broke posthog.Capture. I am currently getting an error in v1.2.19.
posthog 2024/08/24 23:59:13 INFO: response 400 400 Bad Request – failed to parse request: data did not match any variant of untagged enum RawRequest

I had to downgrade to v1.2.18 and everything is working as expected.

[dmarticus]

I think this pull request broke posthog.Capture. I am currently getting an error in v1.2.19.

posthog 2024/08/24 23:59:13 INFO: response 400 400 Bad Request – failed to parse request: data did not match any variant of untagged enum RawRequest

I had to downgrade to v1.2.18 and everything is working as expected.

Ooof; thanks for the report! Sorry to cause breakage; I'm away from my computer this weekend but I'll look into a fix when I'm back!

[evgeek]

@migueldv90 Sorry for the inconvenience!

@dmarticus It seem I tested the feature on the /capture endpoind, but the SDK sends via /batch. My bad :(

Difference is that /capture can ingest uuid with an empty string, while /batch cannot.
Good news: /capture can handle batch events. Example:

curl --location 'https://eu.i.posthog.com/capture/' \
--header 'Content-Type: application/json' \
--header 'User-Agent: posthog-go (version: 1.2.19)' \
--data '{
  "api_key" : "phc_...",
  "batch" : [ {
    "type" : "capture",
    "uuid" : "",
    "library" : "posthog-go",
    "library_version" : "1.2.19",
    "timestamp" : "2024-08-25T11:50:20.154164+00:00",
    "distinct_id" : "1234",
    "event" : "event1",
    "properties" : {
      "$lib" : "posthog-go",
      "$lib_version" : "1.2.19",
      "project" : "test3"
    },
    "send_feature_flags" : false
  },
  {
    "type" : "capture",
    "uuid" : "",
    "library" : "posthog-go",
    "library_version" : "1.2.19",
    "timestamp" : "2024-08-25T11:50:20.154164+00:00",
    "distinct_id" : "1234",
    "event" : "event2",
    "properties" : {
      "$lib" : "posthog-go",
      "$lib_version" : "1.2.19",
      "project" : "test2"
    },
    "send_feature_flags" : false
  },
  {
    "type" : "capture",
    "uuid" : "",
    "library" : "posthog-go",
    "library_version" : "1.2.19",
    "timestamp" : "2024-08-25T11:50:20.154164+00:00",
    "distinct_id" : "1234",
    "event" : "event3",
    "properties" : {
      "$lib" : "posthog-go",
      "$lib_version" : "1.2.19",
      "project" : "test1"
    },
    "send_feature_flags" : false
  } ]
}'

As a quick fix I made this PR - #70. It works, but using an endpoint not designed for batch processing is obviously is not very good decision. It seems that adjusting /batch endpoint to works with empty uuid is better solution.

[dmarticus]

I'm going to revert this change so that folks get the fix ASAP. @evgeek has the start of a fix here: #70, but I want to clarify a few things first, and this will stop the pain for folks pulling from latest.

[dmarticus]

Hi @evgeek

Did some digging into our ingestion endpoint and found that, yes – due to the way we serialize batch requests, we can handle missing uuid entries or valid uuids, but not empty strings. I think it makes the most sense to fix our ingestion endpoint to handle empty strings, rather than work around it within the SDK. I've made a PR for this here: PostHog/posthog#24586. Once it ships, you should be able to re-implement your original changes and the batch endpoint should handle empty UUIDs.

[evgeek]

Excellent, thank you, I'll be looking forward to it!

[evgeek]

@dmarticus Hi!

I see your PR about /batch is already in production.

I made a new PR with reimplementation of uuid specification feature - #72. I tested it, now it works with empty uuid. Please review it when you can :)

[dmarticus]

Hi @evgeek

Thanks for reaching out! I was actually going to DM you today; I merged the /batch PR but for that service, we do manual releases (it's a new service, we're treating it with a bit more caution), so it's only live in our EU datacenter for now (which is why your testing worked – your PostHog account is in the EU). I'm going to release this to US today and then we can merge your PR so that everyone gets UUID goodness :)

[evgeek]

Got it!

[dmarticus]

Hi @evgeek !

Our /batch changes have been deployed everywhere! I'm going to merge #72 now :)

[evgeek]

@dmarticus We did it! Peachy! Thanks :)