Retention Policy For OPC Write

[panManfredini]

In the default topic config of kafka, a broker will retain data for that topic for a certain period.

Problem Case:
A consumer dies for long period of time, and a user sends a lot of OPC writes in that period. Kafka will retain those write request and will send the non-consumed data to the consumer as soon is available again.

Expected behavior:
The user does not expect that the actions he requested will be written unexpectedly as soon as the consumer is up again. This can be very dangerous.

Solution:
The WRITE topic must have a quite short life time, if messages are not delivered within a few second they will be lost. This can be achieved by changing the topic configuration.
Use the IAdminClient to:

• Assess config of a topic, if topic exist
• Create write topic if does not exists
• Change configuration of topic, in particular set the property: "retention.ms" to a certain number of seconds, something like 5 or 10s.

To Do:
Need to add this call to the AdminClient to the constructor of the kafkaConsumer.

[panManfredini]

Nope... The topic retention policy doesn't need to be modified.
It is enough to set AutoOffsetReset = AutoOffsetReset.Latest in the consumerConfig.

[panManfredini]

Au contraire...
AutoOffsetReset = AutoOffsetReset.Latest in the consumerConfig is not enough, it will always start from the latest not committed record, if any is committed.