when i rollout restart ingester, distributor occur error log,at least 2 live replicas required, could only find 1 - unhealthy

[kwenzh]

when i rollout restart ingester, distributor occur error log,

level=warn ts=2022-11-29T12:20:03.883116232Z caller=logging.go:86 traceID=1d6b8feac433899f msg="POST /api/v1/push (500) 567.742µs Response: \"at least 2 live replicas required, could only find 1 - unhealthy instances: 10.244.158.7:9095\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 6220; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.26.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: anonymous; "

but no one pod ip is 10.244.158.7

some ingster starting log, ingester-14

level=info ts=2022-11-29T11:35:09.294367653Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5756 maxSegment=5769
level=info ts=2022-11-29T11:35:27.507504781Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5757 maxSegment=5769
level=info ts=2022-11-29T11:35:45.712193413Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5758 maxSegment=5769
level=info ts=2022-11-29T11:36:05.734217654Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5759 maxSegment=5769
level=info ts=2022-11-29T11:36:23.939091357Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5760 maxSegment=5769
level=info ts=2022-11-29T11:36:42.347255239Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5761 maxSegment=5769
level=info ts=2022-11-29T11:36:59.963055819Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5762 maxSegment=5769
level=info ts=2022-11-29T11:37:10.689662183Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5763 maxSegment=5769
level=info ts=2022-11-29T11:37:28.727453163Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5764 maxSegment=5769
level=info ts=2022-11-29T11:37:46.966350211Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5765 maxSegment=5769
level=info ts=2022-11-29T11:38:07.670661416Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5766 maxSegment=5769
level=info ts=2022-11-29T11:38:26.115827781Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5767 maxSegment=5769
level=info ts=2022-11-29T11:38:41.912058985Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5768 maxSegment=5769
level=info ts=2022-11-29T11:38:41.930956482Z caller=head.go:702 user=anonymous msg="WAL segment loaded" segment=5769 maxSegment=5769
level=info ts=2022-11-29T11:38:41.931025672Z caller=head.go:739 user=anonymous msg="WAL replay completed" checkpoint_replay_duration=29.379200278s wal_replay_duration=4m42.760356707s wbl_replay_duration=914ns total_replay_duration=5m13.235067745s
level=info ts=2022-11-29T11:38:58.624778613Z caller=db.go:1704 user=anonymous msg="Compactions disabled"
level=info ts=2022-11-29T11:38:58.624857004Z caller=ingester.go:1608 user=anonymous msg="Running compaction after WAL replay"
level=info ts=2022-11-29T11:38:58.67270908Z caller=ingester.go:1784 msg="successfully opened existing TSDBs"
level=info ts=2022-11-29T11:38:58.673004412Z caller=mimir.go:762 msg="Application started"
level=info ts=2022-11-29T11:38:58.712021471Z caller=lifecycler.go:612 msg="existing entry found in ring" state=ACTIVE tokens=512 ring=ingester

how distributor find the alive ingeser? and keep Consistent state

[kwenzh]

https://grafana.com/docs/mimir/latest/operators-guide/architecture/memberlist-and-the-gossip-protocol/

[kwenzh]

use default memberlist

[pracucci]

Sorry for the super late reply. This discussion slipped and I haven't noticed it. Sorry, again.

Could you share your Mimir configuration, please?

[pracucci]

The issue is caused by replication_factor: 2. Currently, replication factor 2 is not fully supported and shouldn't be used. See #3844 (comment) for more information.

[kwenzh]

The issue is caused by replication_factor: 2. Currently, replication factor 2 is not fully supported and shouldn't be used. See #3844 (comment) for more information.

default value is 3

[kwenzh]

The issue is caused by replication_factor: 2. Currently, replication factor 2 is not fully supported and shouldn't be used. See #3844 (comment) for more information.

but unregister_on_shutdown is false

[kwenzh]

modif config :

    ingester:
      ring:
        replication_factor: 3
        final_sleep: 0s
        num_tokens: 512
        tokens_file_path: /data/tokens
        unregister_on_shutdown: true

[pracucci]

Ok, thanks for having changed the replication factor 3. Let me know if everything works as expected now.