Update Elasticsearch

[Kathrin-Huber]

Description

Elasticsearch is used in an old version. There are two options:

Option 1

Update Elasticsearch to a newer Version.
Version 6 of the Elasticsearch index uses a new format. The so-called mapping types have been removed. This requires changing the way Elasticsearch is addressed so that we can use a more recent version of Elasticsearch.

Option 2

Use Elasticsearch Integration of Hibernate, this resolves a few issues like

correct deleting of data in Hibernate and ES
writing similar queries instead of two different (HQL for Hibernate and ES-dialect for ES)
not adjusting ourself the changes for new ES versions (less maintenance for Kitodo.Production)

TODO

• evaluate Options and decide
• implement choosen Option

Estimated Costs and Complexity

this is a high-ranged project with 15-20 PT

Related Issues

• Update ElasticSearch version #3153
• Document the search index implementation #3158
• Column indexAction not updated #3719
• Indexing does not work #3999
• Indexing is pausing on long index run #3713
• Indexing issue: illegal character in path #3694
• White page if ElasticSearch cannot be reached #3496
• Limitation of processes in result lists #4331

[solth]

Would there be any downside to the second option? It sounds like the certain winner!

[stweil]

Do we expect problems with the recent license change for Elasticsearch?

[stefanCCS]

As general comment: I would appreciate very much, if we can get rid off old elastic search version 5 :-) ,
as this old version can be considered as some kind of security risk.

I cannot evaluate, if option 1 or 2 (or any other option) is better, but still I would like you to please consider the following:

• As far as I know already elastic search version 7 is existing - so, why not migrate directly to 7 ?

• Please consider migration efforts, which are created out of this change. In my opinion it is worth to do a more complex software develop in KITODO, if the migration effort for all instances is lower.
  e.g. we have setup an own elasticsearch server, so it is somehow easy for us to setup a new one, with version 6 or 7, and assuming that the a new KITODO-Software version will simply support this version, the migration is somehow simple (as the KITODO-Software can already today re-create the whole index, there is no need to migrate the elasticsearch database itself).

[stefanCCS]

Do we expect problems with the recent license change for Elasticsearch?

According to https://en.wikipedia.org/wiki/Server_Side_Public_License (used by ElasticSearch from version 7.11 on), this new license is similar to AGPLv3, and therefore, also of "copyleft" type (in comparison to Apache 2.0, which is NOT of copyleft type).

In my opinion, this is ok for just running an ElasticSearch Server (as this is just running binaries). What I do not know, to which extend the KITODO-Software uses the Elastic Search SourceCode. If this is the case, for sure, a license consideration needs to done (maybe with the result, NOT to use version 7.11(and newer).

[matthias-ronge]

See also #3153 (comment)