fix simultaneously creating a snapshot and updating the repository can potentially trigger an infinite loop #17532

kkewwei · 2025-03-06T10:45:41Z

Description

Related Issues

Resolves #17531

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-03-06T11:04:31Z

❌ Gradle check result for 2a17f65: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-07T14:47:16Z

❌ Gradle check result for d766eb3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-08T05:57:42Z

❌ Gradle check result for 2757776: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-08T07:47:27Z

❌ Gradle check result for f234048: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-10T04:58:38Z

❕ Gradle check result for 7957cdb: UNSTABLE

TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2025-03-10T04:59:05Z

Codecov Report

Attention: Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 72.41%. Comparing base (6d53f9d) to head (0ad9642).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...ch/repositories/blobstore/BlobStoreRepository.java	25.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17532      +/-   ##
============================================
+ Coverage     72.40%   72.41%   +0.01%     
+ Complexity    65828    65777      -51     
============================================
  Files          5316     5316              
  Lines        305294   305339      +45     
  Branches      44289    44294       +5     
============================================
+ Hits         221033   221100      +67     
+ Misses        66187    66111      -76     
- Partials      18074    18128      +54

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

kkewwei · 2025-03-10T05:14:40Z

UNSTABLE

❕ Gradle check result for 7957cdb: UNSTABLE

TEST FAILURES:
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation ##14293

bugmakerrrrrr · 2025-03-13T11:00:10Z

server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

    @Override
    public void executeConsistentStateUpdate(
        Function<RepositoryData, ClusterStateUpdateTask> createUpdateTask,
        String source,
+        Supplier<Repository> currentRepositeSupplier,


I don't want to change the API, I'm wondering if we can retry in SnapshotsService, that is, re-call the createSnapshot method when startRepository != currentRepository

It seems that there is no elegant way to exit unless the executeConsistentStateUpdate is modified.

I think you can use the onFailure to do retry in SnapshotService. When this repo is closed, not to re-call the executeConsistentStateUpdate and invoke the onFailure.

github-actions · 2025-03-14T07:34:51Z

❌ Gradle check result for b054499: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kkewwei · 2025-03-14T08:01:05Z

❌ Gradle check result for b054499: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

S3RemoteStoreIT.testPeerRecoveryWithLowActivityTimeout #16145

github-actions · 2025-03-19T07:00:06Z

❌ Gradle check result for b3e8d32: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-19T10:37:44Z

❌ Gradle check result for d0ae308: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-19T12:12:09Z

❕ Gradle check result for 27de151: UNSTABLE

TEST FAILURES:

      2 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2025-03-20T05:29:42Z

❌ Gradle check result for 53a2e8c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

bugmakerrrrrr · 2025-03-24T12:57:19Z

server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

@@ -643,6 +646,10 @@ public void executeConsistentStateUpdate(
        String source,
        Consumer<Exception> onFailure
    ) {
+        if (this.isOpen() == false) {
+            onFailure.accept(new RepositoryException(metadata.name(), "the repository is closed, maybe updated or deleted, please retry"));


I think it's ok to throw exception here, we just need to notify the user that the repo have changed and let the user decide whether to retry or not.

bugmakerrrrrr · 2025-03-24T12:58:16Z

server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

@@ -4690,6 +4697,11 @@ private void checkAborted() {
        }
    }

+    @Override
+    public boolean isOpen() {


I don't think we need this method, we can track the state within this class

Signed-off-by: kkewwei <[email protected]> Signed-off-by: kkewwei <[email protected]>

github-actions · 2025-03-24T15:37:57Z

❕ Gradle check result for 0ad9642: UNSTABLE

TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

kkewwei · 2025-03-25T02:48:35Z

testSnapshotWithStuckNode

❕ Gradle check result for 0ad9642: UNSTABLE

TEST FAILURES:
      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode #15806

kkewwei force-pushed the fix_loop_in_updateRep branch from 3185a72 to 2a17f65 Compare March 6, 2025 10:47

github-actions bot added bug Something isn't working Storage:Snapshots labels Mar 6, 2025

kkewwei force-pushed the fix_loop_in_updateRep branch 2 times, most recently from 0a9da2d to 42e3000 Compare March 6, 2025 16:08

kkewwei force-pushed the fix_loop_in_updateRep branch from d766eb3 to 2757776 Compare March 8, 2025 05:08

kkewwei force-pushed the fix_loop_in_updateRep branch from 2757776 to f234048 Compare March 8, 2025 07:23

kkewwei force-pushed the fix_loop_in_updateRep branch from f234048 to 7957cdb Compare March 10, 2025 04:07

kkewwei marked this pull request as ready for review March 10, 2025 04:08

bugmakerrrrrr reviewed Mar 13, 2025

View reviewed changes

opensearch-ci-bot mentioned this pull request Feb 28, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ResourceAwareTasksTests #14293

Open

kkewwei force-pushed the fix_loop_in_updateRep branch from 7957cdb to b054499 Compare March 14, 2025 06:39

kkewwei force-pushed the fix_loop_in_updateRep branch from b054499 to b3e8d32 Compare March 19, 2025 06:47

kkewwei force-pushed the fix_loop_in_updateRep branch from b3e8d32 to d0ae308 Compare March 19, 2025 10:27

kkewwei force-pushed the fix_loop_in_updateRep branch from d0ae308 to 27de151 Compare March 19, 2025 11:05

opensearch-ci-bot mentioned this pull request Mar 19, 2025

[AUTOCUT] Gradle Check Flaky Test Report for DedicatedClusterSnapshotRestoreIT #15806

Open

kkewwei force-pushed the fix_loop_in_updateRep branch from 27de151 to 53a2e8c Compare March 20, 2025 04:31

bugmakerrrrrr reviewed Mar 24, 2025

View reviewed changes

fix loop when updating reposity and creating snapshot

0ad9642

Signed-off-by: kkewwei <[email protected]> Signed-off-by: kkewwei <[email protected]>

kkewwei force-pushed the fix_loop_in_updateRep branch from 53a2e8c to 0ad9642 Compare March 24, 2025 14:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix simultaneously creating a snapshot and updating the repository can potentially trigger an infinite loop #17532

fix simultaneously creating a snapshot and updating the repository can potentially trigger an infinite loop #17532

kkewwei commented Mar 6, 2025 •

edited

Loading

github-actions bot commented Mar 6, 2025

github-actions bot commented Mar 7, 2025

github-actions bot commented Mar 8, 2025

github-actions bot commented Mar 8, 2025

github-actions bot commented Mar 10, 2025

codecov bot commented Mar 10, 2025 •

edited

Loading

kkewwei commented Mar 10, 2025

bugmakerrrrrr Mar 13, 2025

kkewwei Mar 14, 2025

bugmakerrrrrr Mar 14, 2025

github-actions bot commented Mar 14, 2025

kkewwei commented Mar 14, 2025

github-actions bot commented Mar 19, 2025

github-actions bot commented Mar 19, 2025

github-actions bot commented Mar 19, 2025

github-actions bot commented Mar 20, 2025

bugmakerrrrrr Mar 24, 2025

bugmakerrrrrr Mar 24, 2025

github-actions bot commented Mar 24, 2025

kkewwei commented Mar 25, 2025

fix simultaneously creating a snapshot and updating the repository can potentially trigger an infinite loop #17532

Are you sure you want to change the base?

fix simultaneously creating a snapshot and updating the repository can potentially trigger an infinite loop #17532

Conversation

kkewwei commented Mar 6, 2025 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Mar 6, 2025

github-actions bot commented Mar 7, 2025

github-actions bot commented Mar 8, 2025

github-actions bot commented Mar 8, 2025

github-actions bot commented Mar 10, 2025

codecov bot commented Mar 10, 2025 • edited Loading

Codecov Report

kkewwei commented Mar 10, 2025

bugmakerrrrrr Mar 13, 2025

Choose a reason for hiding this comment

kkewwei Mar 14, 2025

Choose a reason for hiding this comment

bugmakerrrrrr Mar 14, 2025

Choose a reason for hiding this comment

github-actions bot commented Mar 14, 2025

kkewwei commented Mar 14, 2025

github-actions bot commented Mar 19, 2025

github-actions bot commented Mar 19, 2025

github-actions bot commented Mar 19, 2025

github-actions bot commented Mar 20, 2025

bugmakerrrrrr Mar 24, 2025

Choose a reason for hiding this comment

bugmakerrrrrr Mar 24, 2025

Choose a reason for hiding this comment

github-actions bot commented Mar 24, 2025

kkewwei commented Mar 25, 2025

kkewwei commented Mar 6, 2025 •

edited

Loading

codecov bot commented Mar 10, 2025 •

edited

Loading