[STALE] Replace Kafka Streams with Confluent Parallel Consumer

[nscuro]

Note

This PR is superseded by #552, #553, #554, and later PRs. It's still kept open in order to extract more of it.

Description

Kafka Streams has served us well for the initial stage of the project, and helped us get our feet wet in how Kafka clients operate. However, it has several drawbacks that do not make it a good fit for what we currently use it for. Refer to DependencyTrack/hyades#907 for more details.

This PR replaces Kafka Streams with Confluent's Parallel Consumer. From a developer perspective, processor logic remains pretty much the same.

Functionality-wise, this migration enables:

• Head-of-line blocking semantics
  • Desirable sometimes, e.g. to halt consumption when database is down
  • This is optional and can be disabled by using a processing.order other than partition, see https://github.com/confluentinc/parallel-consumer#head-of-line-blocking
• Parallel consumption exceeding the number of partitions
  • When using processing.order of either key or unordered, see https://github.com/confluentinc/parallel-consumer#ordered-by-key
• Retries for transient errors
  • See https://github.com/confluentinc/parallel-consumer#retries
• Batch consumption
  • Although somewhat limited, see Batching not working as expected confluentinc/parallel-consumer#551 (comment) (still better than what Kafka Streams offers, which is no batching at all)
• More granular configuration
  • Kafka Streams is very limiting when it comes to work allocation, i.e. it's not possible to assign more threads to certain sub-topologies (https://medium.com/@andy.bryant/kafka-streams-work-allocation-4f31c24753cc)

Addressed Issue

• Investigate usage of Confluent's parallel consumer hyades#346
• Implement wrapper around KafkaConsumer to support batch consumption with head of line blocking hyades#907

Additional Details

Checklist

• I have read and understand the contributing guidelines
• This PR fixes a defect, and I have provided tests to verify that the fix is effective
• This PR implements an enhancement, and I have provided tests to verify that it works as intended
• This PR introduces changes to the database model, and I have added corresponding update logic
• This PR introduces new or alters existing behavior, and I have updated the documentation accordingly