Improvements to the HTTP Sink (#163)
* Improvements to the HTTP Sink
This PR introduces the following enhancements to the HTTP Sink:
1. **Queue Limiting**: We've set a limit on the queue size per topic to reduce the chances of an Out-of-Memory (OOM) issue. Previously the queue was unbounded and in a scenario where the http calls are slow and the sink gets more records than it clears, it would eventually lead to OOM.
2. **Offering Timeout**: The offering to the queue now includes a timeout. If there are records to be offered, but the timeout is exceeded, a retriable exception is thrown. Depending on the connector's retry settings, the operation will be attempted again. This helps avoid situations where the sink gets stuck processing a slow or unresponsive batch.
3. **Duplicate Record Handling**: To prevent the same records from being added to the queue multiple times, we've introduced a `Map[TopicPartition, Offset]` to track the last processed offset for each topic-partition. This ensures that the sink does not attempt to process the same records repeatedly.
4. **Batch Failure Handling**: The changes also address a situation where an HTTP call fails due to a specific input, but the batch is not removed from the queue. This could have led to the batch being retried indefinitely, which is now prevented.
In the near future, there will be a new PR to further reduce the code complexity around the batching approach and the boilerplate code.
* fix the unit test
* Rename variable
* Removes the invalid functional tests. a failed batch request is not retried anymore.
* Remove unused functions
---------
Co-authored-by: stheppi <[email protected]>
