Restoring data in case of Elasticsearch corruption

[tvelovraf]

Hi!

I tried to upgrade from v1.15.1 to v2.0.0 and unfortunately missed release notes that I need to upgrade to v1.17 first. Thinking this information shoud be also updated on the website.

I also made a great mistake: I didn't backup the database before upgrading. Of course, server didn't run correctly but it seemed to corrupt my database as well. Only after that I did a backup :(
Finally, I tried to re-run v1.15.1 server with this database and got problem with redis: redis container constantly restarting. docker logs clearml-redis throws this:

...
1:M 15 Jan 2025 18:00:41.710 # Can't handle RDB format version 12
1:M 15 Jan 2025 18:00:41.710 # Fatal error loading the DB: Invalid argument. Exiting.

If I delete /opt/clearml/data/dump.rdb, redis stops throwing an error but there comes problem with elasticsearch container restarting.
In that case docker logs clearml-elastic throws this:

...
java.lang.IllegalStateException: data path [/usr/share/elasticsearch/data] is not compatible with Elasticsearch v7.17.18, perhaps it has already been upgraded to a later version
Likely root cause: java.lang.IllegalStateException: [/usr/share/elasticsearch/data/nodes] is a file which contains [written by Elasticsearch 8.17.0 to prevent a downgrade to a version prior to v8.0.0 which would result in data loss]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:286)
	at org.elasticsearch.node.Node.<init>(Node.java:429)
	at org.elasticsearch.node.Node.<init>(Node.java:309)
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
	at org.elasticsearch.cli.Command.main(Command.java:77)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/clearml.log

Are there any ways to restore data or it's a bad idea?

[oren-clearml]

It seems that skipping the intermediate ClearML versions caused Elastic and Redis to upgrade successfully, resulting in file formats that are now incompatible with ClearML 1.15.1.
However, MongoDB couldn't complete its upgrade, since a major version of MongoDB was skipped in the process.
I would suggest trying to manually upgrade the MongoDB data:
First, start MongoDB version 5.
Launch mongo using the docker-compose of version 1.17:

docker-compose -f /opt/clearml/docker-compose.yml up -d mongo

Once the container is running, access the MongoDB shell:

sudo docker exec -it clearml-mongo mongo

And in the MongoDB shell, execute the following commands to update the feature compatibility version:

use auth
db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } )
use backend
db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } )

After these commands complete successfully, try to start the newest version of ClearML.

Let me know if this worked