Just wanting to Know whether we can use plugin to Backup/Restore Elastic Search 5.2 or newer version #3

codervarun · 2018-02-13T04:41:07Z

Please comment if anybody has idea. I can see that in commit somebody has written ES 5.2.
If plugin is made compatible for 5.x versions, then please write down the steps to download the plugin and install it.

synhershko · 2018-02-14T19:19:26Z

Hi, we are in the process of refactoring the codebase. We have compiled it successfully against ES 5.2 but haven't yet released it officially because of said refactor. Watch this space :)

codervarun · 2018-02-15T04:21:51Z

@synhershko Cool but It would be great if you can share the time when you are planning to release it as this would help us taking decision whether we can use swift to backup/restore ES indexes ?

uded · 2018-02-28T22:12:14Z

Don't like to be a bearer of a bad news, but current Elastics search is more like 6.2 :-)
Just saying, no pressure...

synhershko · 2018-03-01T19:13:12Z

We are well aware :) currently busy with shipping other products but this will happen very soon :)

uded · 2018-03-02T12:25:51Z

Create tickets, happy to help! Now I do not know what your plan is, therefore hard to pitch in...

drazzib · 2018-08-07T07:30:44Z

Hi @synhershko !

Is there any news from you side regarding Elastic >= 5.x support ?
Could you please push your modifications to a specific branch ? so that anyone can start reviewing and helping.

Regards,

shamimgeek · 2018-09-10T20:43:37Z

Hi @synhershko : Any update for new release which supports 5.x and 6.x ?

synhershko · 2018-09-28T10:43:14Z

Hi all,

A quick status update.

Since 5.x versions are too old at this point, we don't have plans to support them. Only 6.x.

I have just pushed some work + tests to fix some issues the plugin had, and make it compatible with 6.x. Currently it compiles for 6.0.0 but making it work with any 6.x version is rather trivial at this point.

We are still working on testing the plugin and making sure it works flawlessly - it previously had no tests and during the process of adding tests we indeed found various issues with it and fixed what we found.

If any of you can give it a try we will be more than happy to support your tests and fix whatever you might find. Compiling it at this point should be trivial but we can provide binaries for any version > 6.x in case you need it.

Thank you for your understanding.

shamimgeek · 2018-09-28T12:32:23Z

@synhershko : I can test it in my openstack environment. Could you please tell me how to install test version of this plugin ?

synhershko · 2018-09-28T14:05:52Z

git clone -> change the ES version in build.gradle -> gradle build -> install the plugin file from file path (in /build folder)

shamimgeek · 2018-09-28T16:22:51Z

Build is failing

changed as below:
elasticsearch = '6.0.0' -->> elasticsearch = '5.4.3'

./gradlew build.gradle 
Picked up _JAVA_OPTIONS: -Dhttps.proxyHost=xxxxxxx -Dhttps.proxyPort=8080

> Configure project :
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
  Gradle Version        : 4.10.1
  OS Info               : Linux 4.4.0-134-generic (amd64)
  JDK Version           : Oracle Corporation 1.8.0_131 [Java HotSpot(TM) 64-Bit Server VM 25.131-b11]
  JAVA_HOME             : /usr/lib/jvm/java-8-oracle
  Random Testing Seed   : F3804EC030B83C32

FAILURE: Build failed with an exception.

* What went wrong:
Task 'build.gradle' not found in root project 'elasticsearch-repository-swift'.

* Try:
Run gradlew tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 0s

synhershko · 2018-09-29T16:49:32Z

Like I said, we are not going to support ES 5.x, only ES 6.x. Tho the error you pasted seems to be related to running gradle build in the wrong folder.

shamimgeek · 2018-09-29T17:30:38Z

ok. I did not change version of elasticsearch this time. I followed below steps. however, i am getting same error. From which directory should i run gradle build ?

sakhtar@sakhtar:~/SRE/elasticsearch_backup$ git clone https://github.com/BigDataBoutique/elasticsearch-repository-swift.git
Cloning into 'elasticsearch-repository-swift'...
remote: Enumerating objects: 26, done.
remote: Counting objects: 100% (26/26), done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 761 (delta 3), reused 17 (delta 1), pack-reused 735
Receiving objects: 100% (761/761), 279.75 KiB | 415.00 KiB/s, done.
Resolving deltas: 100% (312/312), done.
Checking connectivity... done.
sakhtar@sakhtar:~/SRE/elasticsearch_backup$ cd elasticsearch-repository-swift/
sakhtar@sakhtar:~/SRE/elasticsearch_backup/elasticsearch-repository-swift$ ls
build.gradle  gradle  gradlew  gradlew.bat  LICENSE  NOTICE  README.md  settings.gradle  src
sakhtar@sakhtar:~/SRE/elasticsearch_backup/elasticsearch-repository-swift$ ./gradlew build.gradle 
Picked up _JAVA_OPTIONS: -Dhttps.proxyHost=x.x.x.x -Dhttps.proxyPort=8080

> Configure project :
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
  Gradle Version        : 4.10.1
  OS Info               : Linux 4.4.0-134-generic (amd64)
  JDK Version           : Oracle Corporation 1.8.0_131 [Java HotSpot(TM) 64-Bit Server VM 25.131-b11]
  JAVA_HOME             : /usr/lib/jvm/java-8-oracle
  Random Testing Seed   : 885E5C8B865712B5

FAILURE: Build failed with an exception.

* What went wrong:
Task 'build.gradle' not found in root project 'elasticsearch-repository-swift'.

* Try:
Run gradlew tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 0s



``

synhershko · 2018-09-29T17:37:16Z

You should run `./gradlew build` -- Itamar Syn-Hershko CTO, Founder BigData Boutique <http://bigdataboutique.com/> Elasticsearch Consulting Partner Microsoft MVP | Lucene.NET PMC http://code972.com | @synhershko <https://twitter.com/synhershko>

…

On Sat, Sep 29, 2018 at 8:30 PM Shamim Akhtar ***@***.***> wrote: ok. I did not change version of elasticsearch this time. I followed below steps. however, i am getting same error. From which directory should i run gradle build ? ***@***.***:~/SRE/elasticsearch_backup$ git clone https://github.com/BigDataBoutique/elasticsearch-repository-swift.git Cloning into 'elasticsearch-repository-swift'... remote: Enumerating objects: 26, done. remote: Counting objects: 100% (26/26), done. remote: Compressing objects: 100% (17/17), done. remote: Total 761 (delta 3), reused 17 (delta 1), pack-reused 735 Receiving objects: 100% (761/761), 279.75 KiB | 415.00 KiB/s, done. Resolving deltas: 100% (312/312), done. Checking connectivity... done. ***@***.***:~/SRE/elasticsearch_backup$ cd elasticsearch-repository-swift/ ***@***.***:~/SRE/elasticsearch_backup/elasticsearch-repository-swift$ ls build.gradle gradle gradlew gradlew.bat LICENSE NOTICE README.md settings.gradle src ***@***.***:~/SRE/elasticsearch_backup/elasticsearch-repository-swift$ ./gradlew build.gradle Picked up _JAVA_OPTIONS: -Dhttps.proxyHost=x.x.x.x -Dhttps.proxyPort=8080 > Configure project : ======================================= Elasticsearch Build Hamster says Hello! ======================================= Gradle Version : 4.10.1 OS Info : Linux 4.4.0-134-generic (amd64) JDK Version : Oracle Corporation 1.8.0_131 [Java HotSpot(TM) 64-Bit Server VM 25.131-b11] JAVA_HOME : /usr/lib/jvm/java-8-oracle Random Testing Seed : 885E5C8B865712B5 FAILURE: Build failed with an exception. * What went wrong: Task 'build.gradle' not found in root project 'elasticsearch-repository-swift'. * Try: Run gradlew tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/4.10.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 0s `` — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAM9HAjod0iCqxMfJhn3bb2-olcnQk6Iks5uf64_gaJpZM4SDNZO> .

shamimgeek · 2018-09-29T20:11:45Z

now i get below errors

sakhtar@sakhtar:~/SRE/elasticsearch_backup/elasticsearch-repository-swift$ ./gradlew build

> Configure project :
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
  Gradle Version        : 4.10.1
  OS Info               : Linux 4.4.0-134-generic (amd64)
  JDK Version           : Oracle Corporation 1.8.0_131 [Java HotSpot(TM) 64-Bit Server VM 25.131-b11]
  JAVA_HOME             : /usr/lib/jvm/java-8-oracle
  Random Testing Seed   : 8ED6A944C4F9FB53

> Task :integTestCluster#installElasticsearchRepositorySwiftPlugin FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integTestCluster#installElasticsearchRepositorySwiftPlugin'.
> A problem occurred starting process 'command '/home/sakhtar/SRE/elasticsearch_backup/elasticsearch-repository-swift/build/cluster/integTestCluster node0/elasticsearch-6.0.0/bin/elasticsearch-plugin''

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 6s
23 actionable tasks: 23 executed
sakhtar@sakhtar:~/SRE/elasticsearch_backup/elasticsearch-repository-swift$

synhershko · 2018-09-29T20:23:31Z

Can you try again? it failed for me on first run and succeeds in subsequent runs

shamimgeek · 2018-09-29T20:30:32Z

In subsequent runs it is successful. Thanks for your support
Where can i find artifacts and how can i install this plugin, i would like to keep these artifacts in artifactory.


> Configure project :
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
  Gradle Version        : 4.10.1
  OS Info               : Linux 4.4.0-134-generic (amd64)
  JDK Version           : Oracle Corporation 1.8.0_131 [Java HotSpot(TM) 64-Bit Server VM 25.131-b11]
  JAVA_HOME             : /usr/lib/jvm/java-8-oracle
  Random Testing Seed   : 6FD239D6E76C650F

> Task :test
==> Test Info: seed=1DFDA67A3C7EE2FC; jvms=3; suites=3
==> Test Summary: 3 suites, 6 tests

> Task :integTestRunner
==> Test Info: seed=A18247DEC8C8546A; jvm=1; suite=1
==> Test Summary: 1 suite, 1 test

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 25s
34 actionable tasks: 19 executed, 15 up-to-date

synhershko · 2018-09-29T20:35:55Z

build/distributions/repository-swift-3.0.0-es6.0.0.zip

shamimgeek · 2018-09-29T20:41:14Z

Installation will be like below ?

elasticsearch-plugin install/remove <artifactoryurl>/repository-swift-3.0.0-es6.0.0.zip

synhershko · 2018-09-30T07:15:50Z

yes, please follow the docs in https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/plugin-management-custom-url.html

shamimgeek · 2018-10-02T12:42:09Z

@synhershko
Thanks alot for your support.

After installation of this plugin in all 3 nodes, I restarted elasticsearch service. Then when I try to crate repository with curl command provided. my cluster crash.

do I need to also configure path.repo in elasticsearch.yml file ?

Swift plugin is not (and will not) be available on ES 5.X versions (see: BigDataBoutique/elasticsearch-repository-swift#3 (comment))

qherreros · 2019-01-22T10:03:54Z

Hello,

I've tried to build it againt elasticsearch 6.2.4 without success (using openJDK 9.0). The test task always fails. Am I missing something ?

Suite:org.wikimedia.elasticsearch.swift.SwiftBlobStoreTests
1> [2019-01-22T16:46:25,544][INFO ][o.w.e.s.SwiftBlobStoreTests] [testContainerCreationAndDeletion]: before test
1> [2019-01-22T16:46:25,918][INFO ][o.w.e.s.SwiftBlobStoreTests] [testContainerCreationAndDeletion]: after test
2> REPRODUCE WITH: gradle :test -Dtests.seed=FCD7975541B64D0C -Dtests.class=org.wikimedia.elasticsearch.swift.SwiftBlobStoreTests -Dtests.method="testContainerCreationAndDeletion" -Dtests.security.manager=true -Dtests.locale=mzn -Dtests.timezone=Asia/Novosibirsk
ERROR 0.49s J1 | SwiftBlobStoreTests.testContainerCreationAndDeletion <<< FAILURES!
Throwable #1: java.lang.NoClassDefFoundError: javax/activation/MimetypesFileTypeMap
at __randomizedtesting.SeedInfo.seed([FCD7975541B64D0C:EF9D89AAC4AE4329]:0)
at org.javaswift.joss.swift.SwiftStoredObject.uploadObject(SwiftStoredObject.java:86)
at org.javaswift.joss.swift.Swift.uploadObject(Swift.java:330)
at >
org.javaswift.joss.command.mock.object.UploadObjectCommandMock.callSwift(UploadObjectCommandMock.java:28)
at org.javaswift.joss.command.mock.core.CommandMock.call(CommandMock.java:62)
at org.javaswift.joss.client.core.AbstractStoredObject.directlyUploadObject(AbstractStoredObject.java:204)
at org.javaswift.joss.client.core.AbstractStoredObject.uploadObject(AbstractStoredObject.java:147)
at org.javaswift.joss.client.core.AbstractStoredObject.uploadObject(AbstractStoredObject.java:208)
at org.wikimedia.elasticsearch.swift.repositories.blobstore.SwiftBlobContainer.lambda$writeBlob$4(SwiftBlobContainer.java:172)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at org.wikimedia.elasticsearch.swift.SwiftPerms.exec(SwiftPerms.java:33)
at org.wikimedia.elasticsearch.swift.repositories.blobstore.SwiftBlobContainer.writeBlob(SwiftBlobContainer.java:171)
at org.elasticsearch.repositories.ESBlobStoreTestCase.writeBlob(ESBlobStoreTestCase.java:83)
at org.elasticsearch.repositories.ESBlobStoreTestCase.testContainerCreationAndDeletion(ESBlobStoreTestCase.java:44)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.lang.ClassNotFoundException: javax.activation.MimetypesFileTypeMap
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
... 49 more
2> NOTE: leaving temporary files on disk at: /home/quentin/git/elasticsearch-repository-swift/build/testrun/test/J1/temp/org.wikimedia.elasticsearch.swift.SwiftBlobStoreTests_FCD7975541B64D0C-001
2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1146, maxMBSortInHeap=6.041239543663229, sim=RandomSimilarity(queryNorm=false): {}, locale=mzn, timezone=Asia/Novosibirsk
2> NOTE: Linux 4.20.2-107.current amd64/Oracle Corporation 9.0.4 (64-bit)/cpus=8,threads=1,free=496285304,total=536870912
2> NOTE: All tests run in this JVM: [SwiftBlobStoreTests]
Completed [1/3] on J1 in 2.20s, 1 test, 1 error <<< FAILURES!

Thanks for your work.

Verify etag and retry

johnsudaar pushed a commit to johnsudaar/elasticsearch that referenced this issue Oct 19, 2018

Remove Swift plugin on 5.X banches

f81b7a6

Swift plugin is not (and will not) be available on ES 5.X versions (see: BigDataBoutique/elasticsearch-repository-swift#3 (comment))

This was referenced Oct 19, 2018

Remove Swift plugin on 5.X banches johnsudaar/elasticsearch#1

Closed

Remove Swift plugin on 5.X banches elastic/elasticsearch#34633

Merged

johnsudaar pushed a commit to johnsudaar/elasticsearch that referenced this issue Oct 19, 2018

Remove Swift plugin on 5.X banches

c078b18

Swift plugin is not (and will not) be available on ES 5.X versions (see: BigDataBoutique/elasticsearch-repository-swift#3 (comment))

cbuescher pushed a commit to elastic/elasticsearch that referenced this issue Oct 26, 2018

[Docs] Remove Swift plugin on 5.X banches (#34633)

b51ac8a

Swift plugin is not (and will not) be available on ES 5.X versions (see: BigDataBoutique/elasticsearch-repository-swift#3 (comment))

cbuescher pushed a commit to elastic/elasticsearch that referenced this issue Oct 26, 2018

[Docs] Remove Swift plugin on 5.X banches (#34633)

a60ac01

Swift plugin is not (and will not) be available on ES 5.X versions (see: BigDataBoutique/elasticsearch-repository-swift#3 (comment))

synhershko pushed a commit that referenced this issue Dec 8, 2020

Merge pull request #3 from UltimateSoftware/VerifyEtagAndRetry

ffb0537

Verify etag and retry

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Just wanting to Know whether we can use plugin to Backup/Restore Elastic Search 5.2 or newer version #3

Just wanting to Know whether we can use plugin to Backup/Restore Elastic Search 5.2 or newer version #3

codervarun commented Feb 13, 2018 •

edited

Loading

synhershko commented Feb 14, 2018

codervarun commented Feb 15, 2018 •

edited

Loading

uded commented Feb 28, 2018

synhershko commented Mar 1, 2018

uded commented Mar 2, 2018

drazzib commented Aug 7, 2018

shamimgeek commented Sep 10, 2018

synhershko commented Sep 28, 2018

shamimgeek commented Sep 28, 2018

synhershko commented Sep 28, 2018

shamimgeek commented Sep 28, 2018

synhershko commented Sep 29, 2018

shamimgeek commented Sep 29, 2018

synhershko commented Sep 29, 2018 via email

shamimgeek commented Sep 29, 2018

synhershko commented Sep 29, 2018

shamimgeek commented Sep 29, 2018

synhershko commented Sep 29, 2018

shamimgeek commented Sep 29, 2018

synhershko commented Sep 30, 2018

shamimgeek commented Oct 2, 2018

qherreros commented Jan 22, 2019 •

edited

Loading

Just wanting to Know whether we can use plugin to Backup/Restore Elastic Search 5.2 or newer version #3

Just wanting to Know whether we can use plugin to Backup/Restore Elastic Search 5.2 or newer version #3

Comments

codervarun commented Feb 13, 2018 • edited Loading

synhershko commented Feb 14, 2018

codervarun commented Feb 15, 2018 • edited Loading

uded commented Feb 28, 2018

synhershko commented Mar 1, 2018

uded commented Mar 2, 2018

drazzib commented Aug 7, 2018

shamimgeek commented Sep 10, 2018

synhershko commented Sep 28, 2018

shamimgeek commented Sep 28, 2018

synhershko commented Sep 28, 2018

shamimgeek commented Sep 28, 2018

synhershko commented Sep 29, 2018

shamimgeek commented Sep 29, 2018

synhershko commented Sep 29, 2018 via email

shamimgeek commented Sep 29, 2018

synhershko commented Sep 29, 2018

shamimgeek commented Sep 29, 2018

synhershko commented Sep 29, 2018

shamimgeek commented Sep 29, 2018

synhershko commented Sep 30, 2018

shamimgeek commented Oct 2, 2018

qherreros commented Jan 22, 2019 • edited Loading

codervarun commented Feb 13, 2018 •

edited

Loading

codervarun commented Feb 15, 2018 •

edited

Loading

qherreros commented Jan 22, 2019 •

edited

Loading