Add Convert-Index-To-Remote Action for issue #808 #1302

wntmddus · 2024-11-02T01:14:55Z

Description

Currently Searchable snapshot features are available from Opensearch 2.6 version. But this feature itself would not make the fully automated searchable snapshot possible without convert_index_to_remote action within ism actions. Currently ISM plugin does not provide any form of restore action as part of ism actions.
Solution.
Adding convert_index_to_remote action that performs remote restore on snapshot that was created in previous action or stages and that into search nodes as REMOTE_SNAPSHOT. Only thing that need to be passed in is repository name where customer need to assign their repository first then use that to take snapshot and restore.
"convert_index_to_remote": { "properties": { "repository": { "type": "keyword" } } },
Since actions in policy are initially going through schema validation through schema in opendistro-ism-config.json file, this need to be added in here so that we can submit policy with new action.
Then Added new AttemptRestoreStep and WaitForRestoreStep to perform restore.
Then Single ConvertIndexToRemoteAction will perform AttemptRestoreStep and WaitForRestoreStep to perform restore.
Adding this functionality to the user would benefit them to have fully automated searchable snapshot feature on all of their clusters.

Documentation PR: opensearch-project/documentation-website#8638

Related Issues

Related Issue #808

Check List

[V] New functionality includes testing.
[V] New functionality has been documented.
[v] API changes companion pull request created. not required
[V] Commits are signed per the DCO using --signoff.
[V] Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Seung Yeon Joo <[email protected]> Convert-Index-To-Remote action IT added Signed-off-by: Seung Yeon Joo <[email protected]> Fixing IT Signed-off-by: Seung Yeon Joo <[email protected]> Fixing IT Signed-off-by: Seung Yeon Joo <[email protected]>

bowenlan-amzn · 2025-01-07T00:31:35Z

Tried to look into this error, but cannot make sense of it

org.opensearch.indexmanagement.indexstatemanagement.action.ConvertIndexToRemoteActionIT > test basic conversion to remote index FAILED
    org.junit.ComparisonFailure: expected:<[Successfully started restore for [index=convertindextoremoteactionit_index_basic]]> but was:<[Failed to start restore for [index=convertindextoremoteactionit_index_basic], cause: no permissions for [] and associated roles [own_index, all_access]]>
        at __randomizedtesting.SeedInfo.seed([82D2EDFDB4C2F9AC:D379ECB[62](https://github.com/opensearch-project/index-management/actions/runs/12626094047/job/35212093229?pr=1302#step:6:63)C9A9419]:0)
        at org.junit.Assert.assertEquals(Assert.java:117)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.opensearch.indexmanagement.indexstatemanagement.action.ConvertIndexToRemoteActionIT$test basic conversion to remote index$2.invoke(ConvertIndexToRemoteActionIT.kt:[64](https://github.com/opensearch-project/index-management/actions/runs/12626094047/job/35212093229?pr=1302#step:6:65))
        at org.opensearch.indexmanagement.indexstatemanagement.action.ConvertIndexToRemoteActionIT$test basic conversion to remote index$2.invoke(ConvertIndexToRemoteActionIT.kt:62)
        at org.opensearch.indexmanagement.TestHelpersKt.waitFor(TestHelpers.kt:126)
        at org.opensearch.indexmanagement.TestHelpersKt.waitFor$default(TestHelpers.kt:113)
        at org.opensearch.indexmanagement.indexstatemanagement.action.ConvertIndexToRemoteActionIT.test basic conversion to remote index(ConvertIndexToRemoteActionIT.kt:62)
REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.indexmanagement.indexstatemanagement.action.ConvertIndexToRemoteActionIT.test basic conversion to remote index" -Dtests.seed=82D2EDFDB4C2F9AC -Dtests.security.manager=false -Dtests.locale=mzn-IR -Dtests.timezone=America/Resolute -Druntime.java=21

It's thrown from here
https://github.com/opensearch-project/security/blob/cd1dcbde45a6c2523505360f2fa15483b170e7f5/src/main/java/org/opensearch/security/filter/SecurityFilter.java#L462-L464

But it doesn't tell what permission is missing, and all_access is shown in roles which I suppose should be able to perform this restore action.

wntmddus · 2025-01-07T00:37:26Z

Can I at least have permission to run workflow? it gets very hard to debug this IT error as I need permission to run workflow @bowenlan-amzn

bowenlan-amzn · 2025-01-07T00:42:02Z

src/main/resources/mappings/opendistro-ism-config.json

@@ -216,6 +216,13 @@
                    }
                  }
                },
+                "convert_index_to_remote": {


I prefer this to be a "restore" action which clearly indicate which API it calls behind the scene. With storage_type as a param here, this suits the searchable snapshot use case.

repository would be the required parameter as you have here

snapshot name, rename pattern.., can be optional params with default values

indices should be hard-coded to the managed index itself

https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/searchable_snapshot/#create-a-searchable-snapshot-index
https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/searchable_snapshot/#create-a-searchable-snapshot-index

Reason why I chose this name is that restore can mean different things here. as Restore could just mean regular restore. Since this action is specifically designed for searchable snapshot's remote restore, I named it like this.

Understand, in ISM I think we follow the naming around the wrapped API pretty well.
And a normal way to inform users is by mentioning what to do using ISM in the searchable snapshot documentation page so it won't be hard to find out.

cwperks · 2025-01-07T00:58:17Z

An error message like the one in #1302 (comment) usually indicates that a regular user is trying to perform an admin operation on a system index. I'm not familiar with this PR, but I can see errors in the logs pertaining to the security index.

? WARN ][o.o.s.p.SnapshotRestoreEvaluator] [integTest-0] cluster:admin/snapshot/restore for '.opendistro_security' as source index is not allowed
? ERROR][o.o.i.i.s.r.AttemptRestoreStep] [integTest-0] Failed to start restore for [index=convertindextoremoteactionit_index_basic], cause: no permissions for [] and associated roles [own_index, all_access]
?  org.opensearch.OpenSearchSecurityException: no permissions for [] and associated roles [own_index, all_access]

bowenlan-amzn · 2025-01-07T00:43:42Z

...otlin/org/opensearch/indexmanagement/indexstatemanagement/step/restore/AttemptRestoreStep.kt

+
+            // Proceed with the restore operation
+            val restoreSnapshotRequest = RestoreSnapshotRequest(repository, snapshotName)
+                .indices("*")


May not harm to hard code this to be $indexName

bowenlan-amzn · 2025-01-07T00:46:44Z

...otlin/org/opensearch/indexmanagement/indexstatemanagement/step/restore/AttemptRestoreStep.kt

+            // List snapshots matching the pattern
+            val getSnapshotsRequest = GetSnapshotsRequest()
+                .repository(repository)
+                .snapshots(arrayOf("$indexName*"))


Seems this will retrieve snapshots with name started with index name, what if the snapshot is for many indexes (User has snapshot management to take a snapshot of group of indexes) and the name would not start with specific index.

I think $indexName* can be a good default value, but we better provide a param for user to specify other snapshot prefix.

bowenlan-amzn · 2025-01-07T01:13:50Z

...otlin/org/opensearch/indexmanagement/indexstatemanagement/step/restore/AttemptRestoreStep.kt

+                .storageType(RestoreSnapshotRequest.StorageType.REMOTE_SNAPSHOT)
+                .renamePattern("^(.*)\$")
+                .renameReplacement("$1_remote")
+                .waitForCompletion(false)


Not an expert on searchable snapshot, I can see that it won't download the data but only some cluster state, I feel safe to just wait for completion true here as it probably won't take long. And this would simplify the workflow by about half 😜

@kotwanikunal to have a second opinion on this.

I was debating alot on this on my mind whether to wait for it so making the thread blocking and simplify or just make it false and wait for it to complete.

bowenlan-amzn · 2025-01-07T01:18:11Z

@wntmddus commented on Jan 6, 2025, 4:37 PM PST:

Can I at least have permission to run workflow? it gets very hard to debug this IT error as I need permission to run workflow @bowenlan-amzn

Originally posted by @wntmddus in #1302 (comment)

I suppose only maintainer can re-run, please feel free to ping me on slack

And seems the first contribution even needs maintainer to manually approve to run checks

bowenlan-amzn · 2025-01-07T01:20:12Z

@cwperks commented on Jan 6, 2025, 4:58 PM PST:

An error message like the one in #1302 (comment) usually indicates that a regular user is trying to perform an admin operation on a system index. I'm not familiar with this PR, but I can see errors in the logs pertaining to the security index.
? WARN ][o.o.s.p.SnapshotRestoreEvaluator] [integTest-0] cluster:admin/snapshot/restore for '.opendistro_security' as source index is not allowed
? ERROR][o.o.i.i.s.r.AttemptRestoreStep] [integTest-0] Failed to start restore for [index=convertindextoremoteactionit_index_basic], cause: no permissions for [] and associated roles [own_index, all_access]
?  org.opensearch.OpenSearchSecurityException: no permissions for [] and associated roles [own_index, all_access]
Originally posted by @cwperks in #1302 (comment)

Ahh it seems the snapshot includes security system index, @wntmddus would you try create snapshot only including the test index and see.

wntmddus · 2025-01-07T01:23:27Z

@cwperks I see whats happening. I have taken snapshot of the whole cluster instead of using snapshotAction to wait for it. thats why its trying to restore that system index. let me try to fix that part on IT

wntmddus · 2025-01-07T01:34:08Z

@bowenlan-amzn where Can I join the slack community?

shiv0408 · 2025-01-07T06:01:20Z

Find the public slack link from: https://opensearch.org/slack.html

Signed-off-by: Seung Yeon Joo <[email protected]>

wntmddus requested review from bowenlan-amzn, Hailong-am and vikasvb90 as code owners November 2, 2024 01:14

wntmddus force-pushed the main branch from e1abee8 to 4063eca Compare December 17, 2024 04:57

wntmddus force-pushed the main branch from 1af9daa to 309321e Compare December 17, 2024 21:08

bowenlan-amzn requested review from tandonks, shiv0408 and soosinha as code owners January 6, 2025 03:40

bowenlan-amzn reviewed Jan 7, 2025

View reviewed changes

Fix ConvertIndexToRemoteActionIT

174cd04

Signed-off-by: Seung Yeon Joo <[email protected]>

wntmddus force-pushed the main branch from ff70d76 to 174cd04 Compare January 7, 2025 06:02

Merge branch 'main' into main

da3053b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Convert-Index-To-Remote Action for issue #808 #1302

Add Convert-Index-To-Remote Action for issue #808 #1302

wntmddus commented Nov 2, 2024 •

edited by shiv0408

Loading

bowenlan-amzn commented Jan 7, 2025

wntmddus commented Jan 7, 2025 •

edited

Loading

bowenlan-amzn Jan 7, 2025

wntmddus Jan 7, 2025

bowenlan-amzn Jan 7, 2025

cwperks commented Jan 7, 2025

bowenlan-amzn Jan 7, 2025

wntmddus Jan 7, 2025

bowenlan-amzn Jan 7, 2025

bowenlan-amzn Jan 7, 2025

wntmddus Jan 7, 2025

bowenlan-amzn commented Jan 7, 2025 •

edited

Loading

bowenlan-amzn commented Jan 7, 2025

wntmddus commented Jan 7, 2025

wntmddus commented Jan 7, 2025

shiv0408 commented Jan 7, 2025

Add Convert-Index-To-Remote Action for issue #808 #1302

Are you sure you want to change the base?

Add Convert-Index-To-Remote Action for issue #808 #1302

Conversation

wntmddus commented Nov 2, 2024 • edited by shiv0408 Loading

Description

Related Issues

Check List

bowenlan-amzn commented Jan 7, 2025

wntmddus commented Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwperks commented Jan 7, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bowenlan-amzn commented Jan 7, 2025 • edited Loading

bowenlan-amzn commented Jan 7, 2025

wntmddus commented Jan 7, 2025

wntmddus commented Jan 7, 2025

shiv0408 commented Jan 7, 2025

wntmddus commented Nov 2, 2024 •

edited by shiv0408

Loading

wntmddus commented Jan 7, 2025 •

edited

Loading

bowenlan-amzn commented Jan 7, 2025 •

edited

Loading