Can't change the state of index from INSTALLED to ENABLED or DISABLED

[fatihdogmus]

Hello everyone,

We are using cassandra and ElasticSearch for our mixed storage back-ends. We want to disable an old index that will be replaced with a new index. However, whatever I did, the status of the index doesn't change from INSTALLED to DISABLED. In the docs, it says that the index state can go from INSTALLED to DISABLED, but I couldn't manage it. Here is the code that I'm using to disable the index:

JanusGraphManagement m = graph.openManagement();
JanusGraphIndex oldIndex = m.getGraphIndex(oldIndexName);
m.updateIndex(oldIndex, SchemaAction.DISABLE_INDEX).get();
m.commit();
graph.tx().commit();
GraphIndexStatusReport call = awaitGraphIndexStatus(graph, oldIndexName).status(SchemaStatus.DISABLED).call();
System.out.println(call);

It waits of awaitGraphIndexStatus, but it is not successful. It returns the following output:

https://gist.github.com/fatihdogmus/fc0eae8f2eae7d9c973b4ff76f219064

(Since it is too long, I pasted it into a gist)

I don't know what the problem is. I followed the documentation to the minute, but the index can't be DISABLED or ENABLED and the output doesn't give any information about why.

Would appreciate a lot if anyone can help.

[porunov]

@fatihdogmus await waits for all current nodes to receive an index updates. Sometimes there could be ghost nodes (previous JanusGraph nodes which are now dead). If that happens you need to force clean those nodes. Moreover, you need to ensure that your management transaction is the only opened management transaction for the current JanusGraph instance.
We should probably add that into documentation but for now you can check out this post: https://developer.ibm.com/articles/janusgraph-tips-and-tricks-pt-2/

In short, what you need to do is:

mgmt = graph.openManagement()
mgmt.getOpenInstances().forEach(instance -> {
   // You need to force close only ghost instances. Do not close active normal instances.
   // you need to develop your own isGhostInstance method implementation 
   if(isGhostInstance(instance)){
      mgmt.forceCloseInstance(instance);
   }
});
mgmt.commit();

Most likely you have ghost instances which you need to cleanup.
If you are unsure that some other threads may open management transactions as well then check them and chose necessary transactions:

graph.getOpenTransactions().foreach(tx -> {
   // you need to close only stale management transactions. Normal active transactions are not needed to be rolled back.
   // you need to develop your own isStaleManagementTransaction method implementation 
  if(isStaleManagementTransaction(tx)){
     tx.rollback();
  }
});

In case you are deploying JanusGraph nodes in a system where their hostnames may change from time to time (due to re-deploys. Let's say you configured Kubernetes which may assign JanusGraph to a new pod during any redeployment), then it's better to ensure that your instance id doesn't change (use stateful set or some fixed identifiers). Otherwise you may end up with new ghost instances after such redeployments.
To set instance id to a JanusGraph node and be sure it is reused after restarts you may use the next properties:

graph.unique-instance-id=${YOUR_UNIQUE_INSTANCE_ID}
graph.replace-instance-if-exists=true

When you are sure you don't have ghost instances and ghost management transactions, you can try to disable your index again.
Hopefully it will help to resolve your index issue.

[fatihdogmus]

Hello @porunov Thank you very much for your response! It was open management transactions. When I commit all of them, it can change the index state, but in a broken way, since right now I can't detect which are stale management transactions. Can you give me some pointers what I should do to detect if a transaction is stale? I looked through the documentation and the class code and properties, but there doesn't seem be a way to detect if it is still. Would love some pointers. Just some place to start or a documentation would be more than enough to get me started.

[porunov]

The problem is that there is no such status as stale transaction. It is specific to application logic. A single transaction can have a lifetime of the whole application lifetime and not be considered as stale due to application specific logic. In other case a transaction which is opened for 1 second can be considered as stale, but JanusGraph can't decide that for users.
In case your business logic opens management transactions which are never closed then you may end up waiting for index status to be changed until all the opened management transactions are closed (which is infinite in this case).
This is rarely happens that someone needs to open a management transaction for the whole life-cycle of the application because the use-cases for that are unknown (at least for me). Usually management transaction is opened when there is a need to interact with the JanusGraph schema (like CRUD operations over schema elements). Such transactions are usually short lived.
In case you found out that in your instance more then one management transaction is opened and a second transaction is stuck there forever then it most likely means that there is a bug in the code (like leaked transaction most likely or a dead lock). If that happens it's better to detect such bugs and fix them.
Basically, to fix the issues you just need to always close any management transaction after logic which interacts with the schema is finished. Moreover, you must ensure that there are no leaked transactions.
I think a good way to do that would be using try / finally like below:

JanusGraphManagement management = graph.openManagement();
try{
    // interact with management
    management.commit();
} finally {
    if(management.isOpen()){
        management.rollback();
    }
}

[nikhilbonte21]

Hello there,
We are also facing similar issue while Disabling an Enabled index.

Based on @porunov's response on this thread we tried below steps:

1. Made sure that all management objects opened in application are committed
2. Close all open transactions (since not sure how to check valid Active txn, closing all of them for now)
3. We saw so many open Instances in graph, so we tried following

Connected to Janusgraph via gremlin shell
Closed all open Instances except current one
Verified no other transaction is open for graph
Ran command to Disable index
mgmt.updateIndex(ind_24, SchemaAction.DISABLE_INDEX).get();

This actually moved index to INSTALLED state instead of DISABLED (from ENABLED)

Please note that I am working on a Mixed Index

Here is bit about our overall goal:
We have a Mixed index with few properties & want to update an existing property
- Updates can be like change Mapping of prop key OR add/remove Analyzer

We are currently trying to create new Mixed index & Disable the existing one
But, is there any way to just update an existing property inside a Mixed index? Also tried to search in Janusgraph docs but did not find any way to do this.

[porunov]

Notice in JanusGraph 1.0.0 the index management is changed and it is possible to remove a mixed index (not released yet but you can use onc of the latest builds if you want to test it like this). See new index lifecycle for the future JanusGraph 1.0.0 release.

It's possible to add new properties to an existing mixed index using addIndexKey (see doc here). Notice that you will need to perform reindex operation to enable the newly added key in the index (or directly enable it risking of non-indexing previous elements).

In case your question is about changing the definition of the property which is already used in the mixed index (i.e. changing it's data type or cardinality) then it's not possible as for now because it will require providing JanusGraph with the conversion logic which is usually user's business logic. I.e. it's not possible to convert ClassA to ClassB without having specified logic about the conversion from ClassA to ClassB. That said, it's possible to change element names if that's something you are looking to but you be aware of using such properties in the mixed index. See documentation about changing schema elements here: https://docs.janusgraph.org/schema/#changing-schema-elements