Inconsistent consumer instances | "Consumer instance not found" error

[abdelrahman-a-ibrahim]

Dear Team,

I'm currently deploying a REST API as a sidecar container on Kubernetes, alongside my primary application container within the same pod. The application successfully creates topics and produces data via the Kafka REST sidecar container using a localhost connection.

However, I've encountered an issue while attempting to create and subscribe to Kafka consumers using the following REST API:

kubectl exec -it curl-pod -- curl -s -w "%{http_code}" -X POST \ --url http://my-consumer-k8s-service:8082/consumers/my-consumer-group-name/ \ --header "Content-Type: application/vnd.kafka.json.v2+json" \ --data '{"name": "consumer-1", "format": "json", "auto.offset.reset": "earliest", "auto.commit.enable": "false"}'

Upon using this API endpoint to create the consumer instance, I receive the correct response as per the documentation:

{ "instance_id": "consumer-1", "base_uri": "http://some_uri" }

However, when attempting to subscribe to the created consumer using the "base_uri", I intermittently receive the response "Consumer instance not found". If I keep trying to subscribe (keep sending the subscription request), it might work and I get a response for a successful subscription after so many trials.

After a hardly successful subscription, this inconsistency persists even when fetching data from the topic using the created consumer instance (sometimes it works and fetches the data, sometimes I get the "Consumer instance not found" response).

After a hardly successful data fetching, when monitoring the created consumer-group and consumer instance using my Kafka UI, they appear only momentarily while fetching data, then disappear from the consumers list.

I have two questions:

a) How can I ensure consistent creation of consumers and subscriptions?
b) How can I prevent the created consumer-group and consumer instance from disappearing prematurely?

Note: I have tested with the following confluentinc/cp-kafka-rest versions:

• latest (at the time of this inquiry)
• 7.6.0

Your assistance and support in this matter are greatly appreciated.

[abdelrahman-a-ibrahim]

I've identified the issue. The inconsistency arises from my attempt to deploy three pods of the Kafka REST Proxy. It appears that if a particular pod creates the consumer instance, the other pods are unaware of it. This behavior is perplexing to me because I expected the consumer instances to be saved in Kafka, allowing each Kafka REST pod to interact with them.