Restore multiple node cluster : Error with table ***, found X directories

[blatinier]

Hello,

I posted an issue in the upstream nuvo#17 because it had issues already but I post it here too seems it looks like the cain that is up to date.

I'm using cain to backup & restore a cassandra cluster on kubernetes and I have the following error when I restore my cassandra :

2019/09/12 14:54:01 Error with table my_table, found 2 directories

The 2 directories match with my two nodes. The tree of my cain generated in S3 backup looks like that:

bucket
-- default
-- -- Cassandra
-- -- -- keyspace_name
-- -- -- -- schema_id
-- -- -- -- -- tag_name
-- -- -- -- -- -- cassandra-0
-- -- -- -- -- -- -- my_table
-- -- -- -- -- -- cassandra-1
-- -- -- -- -- -- -- my_table

cassandra-0 and cassandra-1 are my pod names
The restore command I used:

./cain restore -n default -l name=cassandra -k keyspace_name --src=s3://bucket/default/Cassandra -t "tag_name" --schema schema_id

The full log:

2019/09/12 14:54:01 Getting current schema
2019/09/12 14:54:01 Found schema: schema_id
2019/09/12 14:54:01 Calculating paths. This may take a while...
2019/09/12 14:54:01 Error with table my_table, found 2 directories

I tried to play a lot with the --src the tag_name with no luck.
Is there something I am missing regarding the cassandra restore into a multi node cluster?

[maorfr]

Thanks for bringing this up!

It means that on the cassandra container, under the data dir, there are 2 directories named my_table-*. Cain does not know which one to use.

Did you try to restore to a fresh cluster? The 2 directories feel like leftovers from manual actions...

[maorfr]

I don't think this is a problem with restoring to multiple nodes. As long as you have 2 nodes or over (when restoring a 2 node backup), you should be fine.

The error does not say which pod is the problematic one though. Not a problem in a 2 node cluster, but thinking of n, ouch. I'll create an issue for it.

[blatinier]

You are right! \o/
I wipe the folder and the keyspace and it went great after that !
I would improve the error message like this I think:

Error with table my_table, found 2 directories in /var/lib/whatever-the-path-is on pod XXX, expecting only one

Or something like that. I really didn't understood the error. But I'm pretty new to Cassandra so maybe it's clear for someone with more experience.