Well, this is expected.
SM restores schema by applying the output of DESC SCHEMA WITH INTERNALS queried during the backup.
The problem is that schema contains topology related information - dcs in which the keyspace is replicated.
So in order to use SM restore schema task, the restore destination cluster needs to consist of the same dcs as the backed up cluster.

A workaround is to take the schema file from backup location, modify it to fit your needs, and apply it manually.

@yarongilor Is there anything you suggest to change in Scylla Manager ? As per #4041 (comment) this is expected behavior of the manager.

It looks that there is no datacenter of the "us-east" name in the destination cluster.

# cassandra-rackdc.properties
#
# The lines may include white spaces at the beginning and the end.
# The rack and data center names may also include white spaces.
# All trailing and leading white spaces will be trimmed.
#
dc=thedatacentername
rack=therackname
# prefer_local=<false | true>
# dc_suffix=<Data Center name suffix, used by EC2SnitchXXX snitches>

@roydahan , @fruch , is there any known resolution for this issue?
The test ran on eu-west-1 region (with Datacenter: eu-west) and failed restoring backup to us-east datacenter. Is it a matter of wrong selected region to test? or it require an SCT fix?

It's not a new issue, mostly a usability issue.
@mikliapko I think the original issue is assigned to you, are you planning to change SCT so it will change the DC name while trying to restore?

Issue about restoring schema into a differenct DC setting: #4049.

Issue about restoring schema into a differenct DC setting: #4049.

So currently the user is supposed to do the schema restore manually

@mikliapko so I'll say we should at least skip the nemesis if the region of the snapshots doesn't match.

At least until it would be implemented on the test end or manager end.

It's not a new issue, mostly a usability issue. @mikliapko I think the original issue is assigned to you, are you planning to change SCT so it will change the DC name while trying to restore?

I don't remember we have an issue for that.
Created the new one for me to correctly handle the case when the region of the snapshots doesn't match.
#4052

@mikliapko - IMO we can plan for a workaround, depends when this issue will be fixed from Manager side.
@karol-kokoszka , @Michal-Leszczynski - please discuss this in the next Manager refinement meeting. If it's not going to be handled soon, @mikliapko will create a WA in the test for it.

It's not a new issue, mostly a usability issue. @mikliapko I think the original issue is assigned to you, are you planning to change SCT so it will change the DC name while trying to restore?

I don't remember we have an issue for that. Created the new one for me to correctly handle the case when the region of the snapshots doesn't match. #4052

there was an issue about this, long long ago:
https://github.com/scylladb/qa-tasks/issues/1477

I don't know if anything was done to try to apply any workaround.

The issue reproduced at https://jenkins.scylladb.com/job/scylla-master/job/tier1/job/longevity-mv-si-4days-streaming-test/7/
plus latest run

@mikliapko we want to have at least a workaround for this issue until it's being fixed in manager.

@mikliapko we want to have at least a workaround for this issue until it's being fixed in manager.

Sorry, lost track of this issue for a while. I’ll try to come up with a solution no later than next week.

If there is no fix planned in manager or an easy workaround on the manager side, you can have a workaround in SCT by uploading backups to several regions (each with correct region in schema).
Then, change the nemesis to pull the backup from the correct region.

A workaround is to take the schema file from backup location, modify it to fit your needs, and apply it manually.

@Michal-Leszczynski
I'd like to workaround this issue.
Where can I find a list of CQL statements I need to apply after manual modification of schema file?

Where can I find a list of CQL statements I need to apply after manual modification of schema file?

It's in the backup location under /backup/schema/cluster/<clusterID>/task_<taskID>_tag_<snapshotTag>_schema_with_internals.json.gz.
This file contains json array of schema statements returned from DESCRIBE SCHEMA WITH INTERNALS.

Uncompressed file can look like that:

[
  {
    "keyspace": "restoretest_full",
    "type": "keyspace",
    "name": "restoretest_full",
    "cql_stmt": "CREATE KEYSPACE restoretest_full WITH replication = {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'dc1': '2'} AND durable_writes = true;"
  },
  {
    "keyspace": "restoretest_full",
    "type": "table",
    "name": "big_table",
    "cql_stmt": "CREATE TABLE restoretest_full.big_table (\n    id int,\n    data blob,\n    PRIMARY KEY (id)\n) WITH ID = d3607a70-a8bf-11ef-851d-38d002d034f1\nAND bloom_filter_fp_chance = 0.01\n    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}\n    AND comment = ''\n    AND compaction = {'class': 'NullCompactionStrategy', 'enabled': 'false'}\n    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}\n    AND crc_check_chance = 1\n    AND default_time_to_live = 0\n    AND gc_grace_seconds = 864000\n    AND max_index_interval = 2048\n    AND memtable_flush_period_in_ms = 0\n    AND min_index_interval = 128\n    AND speculative_retry = '99.0PERCENTILE'\n    AND paxos_grace_seconds = 864000\n    AND tombstone_gc = {'mode': 'repair', 'propagation_delay_in_seconds': '3600'};\n"
  },
  {
    "keyspace": "restoretest_full",
    "type": "index",
    "name": "bydata_index",
    "cql_stmt": "CREATE INDEX bydata ON restoretest_full.big_table(data);\n"
  },
  {
    "keyspace": "restoretest_full",
    "type": "view",
    "name": "testmv",
    "cql_stmt": "CREATE MATERIALIZED VIEW restoretest_full.testmv AS\n    SELECT id, data\n    FROM restoretest_full.big_table\n    WHERE data IS NOT null\n    PRIMARY KEY (id, data)\n    WITH ID = d38ae5d0-a8bf-11ef-8cb1-45ed76572674\nAND CLUSTERING ORDER BY (data ASC)\n    AND bloom_filter_fp_chance = 0.01\n    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}\n    AND comment = ''\n    AND compaction = {'class': 'SizeTieredCompactionStrategy'}\n    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}\n    AND crc_check_chance = 1\n    AND default_time_to_live = 0\n    AND gc_grace_seconds = 864000\n    AND max_index_interval = 2048\n    AND memtable_flush_period_in_ms = 0\n    AND min_index_interval = 128\n    AND speculative_retry = '99.0PERCENTILE'\n    AND paxos_grace_seconds = 864000\n    AND tombstone_gc = {'mode': 'repair', 'propagation_delay_in_seconds': '3600'};\n"
  }
]

reproduced there:

Packages

Scylla version: 6.3.0~dev-20241122.e2e6f4f441be with build-id 2493a7aae1f855d3df502197f757822b6afc1033

Kernel Version: 6.8.0-1019-aws

Installation details

Cluster size: 5 nodes (i4i.8xlarge)

Scylla Nodes used in this run:

• longevity-mv-si-4d-master-db-node-299884c7-8 (3.250.175.37 | 10.4.11.76) (shards: 30)
• longevity-mv-si-4d-master-db-node-299884c7-7 (3.255.213.87 | 10.4.11.152) (shards: 30)
• longevity-mv-si-4d-master-db-node-299884c7-6 (54.75.36.72 | 10.4.10.102) (shards: 30)
• longevity-mv-si-4d-master-db-node-299884c7-5 (34.255.195.233 | 10.4.9.117) (shards: 30)
• longevity-mv-si-4d-master-db-node-299884c7-4 (18.202.252.170 | 10.4.9.20) (shards: 30)
• longevity-mv-si-4d-master-db-node-299884c7-3 (3.255.99.166 | 10.4.10.136) (shards: 30)
• longevity-mv-si-4d-master-db-node-299884c7-2 (52.50.139.63 | 10.4.11.189) (shards: 30)
• longevity-mv-si-4d-master-db-node-299884c7-1 (46.137.67.238 | 10.4.8.195) (shards: 30)

OS / Image: ami-001a2091244fdbdf3 (aws: undefined_region)

Test: longevity-mv-si-4days-streaming-test
Test id: 299884c7-f5ee-4e0d-8e21-a27a3509b0a6
Test name: scylla-master/tier1/longevity-mv-si-4days-streaming-test
Test method: longevity_test.LongevityTest.test_custom_time
Test config file(s):

• longevity-mv-si-4days.yaml

Was trying to workaround it via manual restore of the schema (with putting the right region into the CQL statements for keyspace creation) - failed because of kms:decryption issue (see details here) - the backup was encrypted with the key of the region it belongs to and can't be decryption with key from replaced region.

Looks like backup uploading to several regions is the only way left so far.

Was trying to workaround it via manual restore of the schema (with putting the right region into the CQL statements for keyspace creation) - failed because of kms:decryption issue (see details here) - the backup was encrypted with the key of the region it belongs to and can't be decryption with key from replaced region.

Looks like backup uploading to several regions is the only way left so far.

those backup were create with KMS keys which are long gone, regardless of region.
or I'm missing something from how the flow of resoute from KMS EaR encrypted sstables should work ?

those backup were create with KMS keys which are long gone, regardless of region.

I recreated those backups a few months ago.
They must be encrypted with the relevant key.

those backup were create with KMS keys which are long gone, regardless of region.

I recreated those backups a few months ago. They must be encrypted with the relevant key.

I take it back, we clear the aliases and not the keys.

Hm, we have an issue related to backup_bucket_region parameter.

In test_defaults.yaml this parameter is empty string:

backup_bucket_region: ''  # use the same region as a cluster

Then it gets rewritten by aws_config.yaml:

backup_bucket_region: 'us-east-1'

This parameter is used to configure manager agent:

        node.update_manager_agent_backup_config(
            region=self.params.get("backup_bucket_region"),
            general_config=agent_backup_general_config,
        )

As a result, if the region_name in test differs from us-east-1, we have a misconfiguration between actual region and region configured in scylla-agent.yaml.

I'm thinking about adding a validation rule for backup_bucket_region parameter in sdcm.sct_config.py. Something like this to forbid such kind of situations:

if self.get("backup_bucket_region") != self.get("region_name"):
    self["backup_bucket_region"] = self.get("region_name")

@fruch May this change lead to any unexpected consequences? How do you think?
Or perhaps you have some better ideas how to fix it?

Hm, we have an issue related to backup_bucket_region parameter.

In test_defaults.yaml this parameter is empty string:

backup_bucket_region: ''  # use the same region as a cluster

Then it gets rewritten by aws_config.yaml:

backup_bucket_region: 'us-east-1'

This parameter is used to configure manager agent:

    node.update_manager_agent_backup_config(
        region=self.params.get("backup_bucket_region"),
        general_config=agent_backup_general_config,
    )

As a result, if the region_name in test differs from us-east-1, we have a misconfiguration between actual region and region configured in scylla-agent.yaml.

I'm thinking about adding a validation rule for backup_bucket_region parameter in sdcm.sct_config.py. Something like this to forbid such kind of situations:

if self.get("backup_bucket_region") != self.get("region_name"):
self["backup_bucket_region"] = self.get("region_name")
@fruch May this change lead to any unexpected consequences? How do you think? Or perhaps you have some better ideas how to fix it?

as long as you have buckets in all regions for doing the backup nemesis (including of in gce / azure)
I doesn't think it's would be a problem.

I would recommend removing backup_bucket_region if it's not gonna be really usable, can be a followup

I would recommend removing backup_bucket_region if it's not gonna be really usable, can be a followup

Yes, as for now we can't set different region_name and backup_bucket_region, I suppose it make sense to get rid of it and use just region_name instead. Okay, I'll remove it in scope of this ticket.

After duplicating all snapshots and fixing backup location issues, disrupt_mgmt_restore is still failing with :

Traceback (most recent call last):
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 5426, in wrapper
    result = method(*args[1:], **kwargs)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 187, in wrapper
    return func(*args, **kwargs)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 3092, in disrupt_mgmt_restore
    self.tester.set_ks_strategy_to_network_and_rf_according_to_cluster(
  File "/home/ubuntu/scylla-cluster-tests/sdcm/tester.py", line 1111, in set_ks_strategy_to_network_and_rf_according_to_cluster
    NetworkTopologyReplicationStrategy(**datacenters).apply(node, keyspace)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/replication_strategy_utils.py", line 47, in apply
    session.execute(cql)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/common.py", line 1318, in execute_verbose
    return execute_orig(*args, **kwargs)
  File "cassandra/cluster.py", line 2729, in cassandra.cluster.Session.execute
  File "cassandra/cluster.py", line 5120, in cassandra.cluster.ResponseFuture.result
cassandra.InvalidRequest: Error from server: code=2200 [Invalid query] message="Only one DC's RF can be changed at a time and not by more than 1"

The test alters keyspace with RF=3 to 5 what is prohibited.
https://argus.scylladb.com/tests/scylla-cluster-tests/0089ac32-5cc0-4168-852a-73718ab10242

I suppose to fix it, we need to implement the procedure described here:
https://opensource.docs.scylladb.com/stable/kb/rf-increase.html#example

I wonder how this test was introduces at first time?
With this limitation disrupt_mgmt_restore test couldn't pass.

reproduced there:
https://argus.scylladb.com/tests/scylla-cluster-tests/1b3e80d1-e6cb-46c0-a07b-0ca1c1b8974d
Backend: aws
Region: eu-west-1, eu-west-2, eu-north-1
Image id: ami-0c7b4b0835c9342f7 ami-039f35b0f1e04947e ami-03a78f37d7eaf9c88
SCT commit sha: 1d4cbaa1ed74fd3d4748a4636c3f6d57805bc24b
SCT repository: [email protected]:scylladb/scylla-cluster-tests.git
SCT branch name: origin/master
Kernel version: 6.8.0-1019-aws
Scylla version: 6.3.0~dev-20241206.7e2875d6489d
Build id: 5227dd2a3fce4d2beb83ec6c17d47ad2e8ba6f5c
Instance type: i4i.4xlarge
Node amount: 8