Consumer duplicates in consumer report output

[IgorDeinechin]

Hi!

I'm experiencing a weird behavior with our cluster consisting of 3 nats-server instances.
We have JetStream enabled there with one stream with replicas set to 3 (to match number of instances on the cluster).
The regular NATS is also processed by those same 3 nats-server instances.

We faced some kind of failure recently with one of these three instances having restarted several times nats container within the pod.
After that failure I notices an increasing number of messages sitting in the stream. We have our stream configured with the Interest policy so the processed messages should go away.

However, though the messages indeed were processed, they kept staying in the stream.

I ran the nats consumer report command and noticed duplicates in the output.

│ Consumer │ Mode │ Ack Policy │ Ack Wait │ Ack Pending │ Redelivered │ Unprocessed │ Ack Floor │ Cluster │
│ consumer_CandidateStatusUpdated │ Pull │ Explicit │ 30.00s │ 0 │ 0 │ 0 │ 652,382 │ nats-0*, nats-1, nats-2! │
│ consumer_CandidateStatusUpdated │ Pull │ Explicit │ 30.00s │ 0 │ 0 │ 0 │ 652,382 │ nats-0!, nats-1, nats-2* │

Also messages pulled from the consumer_CandidateStatusUpdated consumer started coming twice.

Looks like the nats consumer cluster step-down command for that consumer fixed the issue and the report no longer shows duplicates for this consumer.

However, would be great if someone could help me to figure out the reason of such behavior or potential steps that can be completed to prevent such situation in the future.

May this be related somehow to routes setting in cluster configuration?
I noticed that examples only enumerate two of three node there and our config specifies all three nodes in routes for each server in our cluster.

Server version is 2.9.11
NATS Client (for .NET) version is 1.0.3