[Bug]: verify-segment --fix insert_log not work

[Niceies]

Current Behavior

I found that the file is missing in minio, but the metadata recorded in etcd contains the file, causing a querynode that loads this segment to fail to start. fileID：455863820064330546

[2025/02/08 09:13:55.901 +00:00] [WARN] [segments/segment_loader.go:669] ["release new segment created due to load failure"] [traceID=caed4261a9e44ff888b81cd1245dbfec] [collectionID=455863820065742927] [segmentType=Sealed] [requestSegments="[455863820065742996]"] [preparedSegments="[455863820065742996]"] [segmentID=455863820065742996] [error="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=milvus-bucket, object=file/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330546]"] [errorVerbose="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=milvus-bucket, object=file/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330546]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:734\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) At LoadSegment\nWraps: (3) Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=milvus-bucket, object=file/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330546]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/02/08 09:13:55.902 +00:00] [INFO] [segments/segment.go:1618] ["delete segment from memory"] [collectionID=455863820065742927] [partitionID=455863820065742928] [segmentID=455863820065742996] [segmentType=Sealed] [insertCount=10000]
[2025/02/08 09:13:55.908 +00:00] [INFO] [segments/collection.go:239] ["collection ref decrement"] [nodeID=325] [collectionID=455863820065742927] [refCount=1]
[2025/02/08 09:13:55.908 +00:00] [WARN] [delegator/delegator_data.go:445] ["worker failed to load segments"] [traceID=caed4261a9e44ff888b81cd1245dbfec] [collectionID=455863820065742927] [channel=milvus-cluster-rootcoord-dml_14_455863820065742927v0] [replicaID=455863820172656651] [workID=325] [segments="[455863820065742996]"] [error="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=milvus-bucket, object=file/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330546]"]
[2025/02/08 09:13:55.908 +00:00] [WARN] [querynodev2/services.go:470] ["delegator failed to load segments"] [traceID=caed4261a9e44ff888b81cd1245dbfec] [collectionID=455863820065742927] [partitionID=455863820065742928] [shard=milvus-cluster-rootcoord-dml_14_455863820065742927v0] [segmentID=455863820065742996] [level=L1] [currentNodeID=325] [dstNodeID=325] [error="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=milvus-bucket, object=file/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330546]"]

I used the birdwatcher tool to verify the collection and found that the file was missing. Then I used the --fix option to repair it but it did not take effect. From the code view, it was to delete the etcd key, but in fact the logID was recorded in the values ​​​​of the binlog key. The deletion function was wrong.

Start to verify segment(455863820065742996) --- (0/1)
file/insert_log/455863820065742927/455863820065742928/455863820065742996/0/455863820064330528
file/insert_log/455863820065742927/455863820065742928/455863820065742996/0/455863820064330536
file/insert_log/455863820065742927/455863820065742928/455863820065742996/0/455863820064330545
file/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330529
file/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330537
file/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330546
failed to check  file/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330546 The specified key does not exist. NoSuchKey
path file/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330546 true
remove milvus-cluster/meta/datacoord-meta/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330546
audit delete milvus-cluster/meta/datacoord-meta/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330546
failed to delete insert binlogs from etcd for segment 455863820065742996, err: key not found: milvus-cluster/meta/datacoord-meta/insert_log/455863820065742927/455863820065742928/455863820065742996/1/455863820064330546
failed to remove segment insert path
file/insert_log/455863820065742927/455863820065742928/455863820065742996/100/455863820064330522
file/insert_log/455863820065742927/455863820065742928/455863820065742996/100/455863820064330538
file/insert_log/455863820065742927/455863820065742928/455863820065742996/100/455863820064330547
file/insert_log/455863820065742927/455863820065742928/455863820065742996/101/455863820064330523
file/insert_log/455863820065742927/455863820065742928/455863820065742996/101/455863820064330531
file/insert_log/455863820065742927/455863820065742928/455863820065742996/101/455863820064330540
file/insert_log/455863820065742927/455863820065742928/455863820065742996/102/455863820064330524
file/insert_log/455863820065742927/455863820065742928/455863820065742996/102/455863820064330532
file/insert_log/455863820065742927/455863820065742928/455863820065742996/102/455863820064330541
file/insert_log/455863820065742927/455863820065742928/455863820065742996/103/455863820064330525
file/insert_log/455863820065742927/455863820065742928/455863820065742996/103/455863820064330533
file/insert_log/455863820065742927/455863820065742928/455863820065742996/103/455863820064330542
file/insert_log/455863820065742927/455863820065742928/455863820065742996/104/455863820064330526
file/insert_log/455863820065742927/455863820065742928/455863820065742996/104/455863820064330534
file/insert_log/455863820065742927/455863820065742928/455863820065742996/104/455863820064330543
file/insert_log/455863820065742927/455863820065742928/455863820065742996/105/455863820064330527
file/insert_log/455863820065742927/455863820065742928/455863820065742996/105/455863820064330535
file/insert_log/455863820065742927/455863820065742928/455863820065742996/105/455863820064330544
Segment(455863820065742996) binlog done
file/stats_log/455863820065742927/455863820065742928/455863820065742996/100/455863820064330530
file/stats_log/455863820065742927/455863820065742928/455863820065742996/100/455863820064330539
file/stats_log/455863820065742927/455863820065742928/455863820065742996/100/455863820064330548
file/stats_log/455863820065742927/455863820065742928/455863820065742996/100/1

Where the file id 455863820064330546 is actually recorded in key value

fieldID:1 binlogs:<entries_num:4000 timestamp_from:455864520399912964 timestamp_to:455864528172482561 log_size:32000 logID:455863820064330529 7:32000 > binlogs:<entries_num:4000 timestamp_from:455864530675957761 timestamp_to:455864539549532162 log_size:32000 logID:455863820064330537 7:32000 > binlogs:<entries_num:2000 timestamp_from:455864542878760962 timestamp_to:455864546155560962 log_size:16000 logID:455863820064330546 >

Expected Behavior

verify-segment --fix insert_log can work

Steps To Reproduce

Environment

Anything else?

No response