You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The kafka output stores offsets whenever an event is stored.
However, writing into kafka is asynchroneous and there is no feed-back in Logprep if writing was actually successful.
Thus, it can happen that an offset is set before an event was really written into kafka.
It would be desireable if offsets are only set for events that were successfully sent.
Otherwise there is a possibility for data loss.
The kafka output stores offsets whenever an event is stored.
However, writing into kafka is asynchroneous and there is no feed-back in Logprep if writing was actually successful.
Thus, it can happen that an offset is set before an event was really written into kafka.
It would be desireable if offsets are only set for events that were successfully sent.
Otherwise there is a possibility for data loss.
A possible solution could involve using a callback in the produce method:
https://docs.confluent.io/kafka-clients/python/current/overview.html#asynchronous-writes
The text was updated successfully, but these errors were encountered: