Error while crawling /Volumes: 30.000 milliseconds timeout on connection http-outgoing-138 [ACTIVE]

[cadm-frank]

I've noticed recently that on two of my systems fscrawler barely manages to index new files before fscrawler exits.
At some point it just seems to stop working and then times out 30 seconds later.

Okt 21 14:08:04 lxfilcgnprd01 fscrawler[5291]: Looking for removed files in [/Volumes/XXX]...
Okt 21 14:08:04 lxfilcgnprd01 fscrawler[5291]: computeVirtualPathName(/Volumes, /Volumes/XXX) = /XXX
Okt 21 14:08:04 lxfilcgnprd01 fscrawler[5291]: directory = [false], filename = [/XXX], includes = [null], excludes = [null]
Okt 21 14:08:04 lxfilcgnprd01 fscrawler[5291]: filename = [/XXX], excludes = [null]
Okt 21 14:08:04 lxfilcgnprd01 fscrawler[5291]: filename = [/XXX], includes = [null]
Okt 21 14:08:04 lxfilcgnprd01 fscrawler[5291]: Looking for removed directories in [/Volumes/XXX]...
Okt 21 14:08:34 lxfilcgnprd01 fscrawler[5291]: Error while crawling /Volumes: 30.000 milliseconds timeout on connection http-outgoing-138 [ACTIVE]
Okt 21 14:08:34 lxfilcgnprd01 fscrawler[5291]: Full stacktrace
Okt 21 14:08:34 myserver fscrawler[5291]: java.net.SocketTimeoutException: 30.000 milliseconds timeout on connection http-outgoing-138 [ACTIVE]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:864) ~[elasticsearch-rest-client-7.10.1.jar:7.10.1]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283) ~[elasticsearch-rest-client-7.10.1.jar:7.10.1]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) ~[elasticsearch-rest-client-7.10.1.jar:7.10.1]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1602) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1572) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1088) ~[elasticsearch-rest-high-level-client-7.10.1.jar:7.10.1]
Okt 21 14:08:34 myserver fscrawler[5291]:         at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.search(ElasticsearchClientV7.java:499) ~[fscrawler-elasticsearch-client-v7-2.7-SNAPSHOT.jar:?]
Okt 21 14:08:34 myserver fscrawler[5291]:         at fr.pilato.elasticsearch.crawler.fs.FsParserAbstract.getFolderDirectory(FsParserAbstract.java:406) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
Okt 21 14:08:34 myserver fscrawler[5291]:         at fr.pilato.elasticsearch.crawler.fs.FsParserAbstract.addFilesRecursively(FsParserAbstract.java:338) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
Okt 21 14:08:34 myserver fscrawler[5291]:         at fr.pilato.elasticsearch.crawler.fs.FsParserAbstract.addFilesRecursively(FsParserAbstract.java:304) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
Okt 21 14:08:34 myserver fscrawler[5291]:         at fr.pilato.elasticsearch.crawler.fs.FsParserAbstract.addFilesRecursively(FsParserAbstract.java:304) ~[fscrawler-core-2.7-SNAPSHOT.jar:?]
Okt 21 14:08:34 myserver fscrawler[5291]:         at fr.pilato.elasticsearch.crawler.fs.FsParserAbstract.run(FsParserAbstract.java:160) [fscrawler-core-2.7-SNAPSHOT.jar:?]
Okt 21 14:08:34 myserver fscrawler[5291]:         at java.lang.Thread.run(Thread.java:832) [?:?]
Okt 21 14:08:34 myserver fscrawler[5291]: Caused by: java.net.SocketTimeoutException: 30.000 milliseconds timeout on connection http-outgoing-138 [ACTIVE]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.14.jar:4.4.14]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar:4.1.4]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.14.jar:4.4.14]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261) ~[httpcore-nio-4.4.14.jar:4.4.14]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502) ~[httpcore-nio-4.4.14.jar:4.4.14]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211) ~[httpcore-nio-4.4.14.jar:4.4.14]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.14.jar:4.4.14]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.14.jar:4.4.14]
Okt 21 14:08:34 myserver fscrawler[5291]:         at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.14.jar:4.4.14]
Okt 21 14:08:34 myserver fscrawler[5291]:         ... 1 more
Okt 21 14:08:34 myserver fscrawler[5291]: FS crawler is stopping after 1 run
Okt 21 14:08:34 myserver fscrawler[5291]: Closing FS crawler [myserver]
Okt 21 14:08:34 myserver fscrawler[5291]: FS crawler thread is now stopped
Okt 21 14:08:34 myserver fscrawler[5291]: Closing Elasticsearch client manager
Okt 21 14:08:34 myserver fscrawler[5291]: ES Client Manager stopped
Okt 21 14:08:34 myserver fscrawler[5291]: FS crawler [myserver] stopped
Okt 21 14:08:34 myserver systemd[1]: fscrawler.service: Succeeded.

[dadoonet]

This is the typical error I'd like to address with #1264.

[dadoonet]

I don't think I have a good answer for now.
The only thing I can see would be to start multiple instances of FSCrawler (one per subdir). That way, if you have to restart, you won't have to scan again the whole content.

[cadm-frank]

Ok, thanks. Maybe I'll have to trick fscrawler then.

[cadm-frank]

This patch is pretty blunt, but it solved the issue for us.

diff -ruN fscrawler.old/elasticsearch-client/elasticsearch-client-v7/src/main/java/fr/pilato/elasticsearch/crawler/fs/client/v7/ElasticsearchClientV7.java fscrawler.src/elasticsearch-client/elasticsearch-client-v7/src/main/java/fr/pilato/elasticsearch/crawler/fs/client/v7/ElasticsearchClientV7.java
--- fscrawler.old/elasticsearch-client/elasticsearch-client-v7/src/main/java/fr/pilato/elasticsearch/crawler/fs/client/v7/ElasticsearchClientV7.java	2021-10-22 10:02:04.426548109 +0200
+++ fscrawler.src/elasticsearch-client/elasticsearch-client-v7/src/main/java/fr/pilato/elasticsearch/crawler/fs/client/v7/ElasticsearchClientV7.java	2021-10-22 10:53:03.352896499 +0200
@@ -411,6 +411,13 @@
 
         RestClientBuilder builder = RestClient.builder(hosts.toArray(new HttpHost[0]));
 
+        builder.setRequestConfigCallback(requestConfigBuilder -> 
+                requestConfigBuilder
+                        .setConnectTimeout(10000)
+                        .setSocketTimeout(60000)
+                        .setConnectionRequestTimeout(0)
+        );
+
         if (settings.getPathPrefix() != null) {
             builder.setPathPrefix(settings.getPathPrefix());
         }

[dadoonet]

It would be great if you could send a PR for this.
I can merge it as is or we could iterate on it to make it configurable.

[cadm-frank]

A PR is probably not going to work without additional work, as we're using an older elasticsearch version.