Got 500 Internal Server Error sometime #531

InfoDevkota · 2021-01-25T10:22:48Z

InfoDevkota
Jan 25, 2021

I have photon on 4 GB RAM, 2 vCPUs, 80 GB SSD Ubuntu server. Then I have created a systemctl service to run the photon-*.jar (photon-0.3.4.jar).

Then I have Nginx reverse proxy to serve the photon as a domain.

But the server returns 500 Internal Server Error sometime. and after few seconds it works as expected.
<html><body><h2>500 Internal Server Error</h2></body></html>

Some time the nginx default 500 errors.

Even when no requests the Metrics Graphs (AWS Lightsail) shows (6-7 % CPU uses)

Is there any better way that I should have configured? Do I need some more powerful server?

I am using it for map data of Nepal, .pbf file size: 262 MB.

If its using the single core, any suggestion on using available cors?

other than that worked like a charm.

Thank you!

Answered by karussell

Jan 25, 2021

How do you start photon? E.g. which Xmx setting are you using?

View full answer

karussell · 2021-01-25T10:29:08Z

karussell
Jan 25, 2021
Maintainer

How do you start photon? E.g. which Xmx setting are you using?

13 replies

lonvia Jan 26, 2021
Maintainer

Rerouting output works via:

[Service]
StandardOutput=file:/var/log/photon.log
StandardError=file:/var/log/photon.log

We use Type=simple in our systemd config, not forking.

InfoDevkota Jan 27, 2021
Author

I updated the Type=simple and add the logs.

This error is being constantly added to log:

2021-01-27 15:19:48,284 [elasticsearch[y9nEotM][generic][T#3]] WARN  org.elasticsearch.cluster.action.shard.ShardStateAction - [photon][2] received shard failed for shard 
id [[photon][2]], allocation id [_prLxHuTSvqw4XPyUeoV4g], primary term [0], message [failed recovery], failure [RecoveryFailedException[[photon][2]: Recovery failed on {y9
nEotM}{y9nEotM6QtmL1y0VdM5Z2g}{_H3DslORRq2vkhU1EQSb2A}{127.0.0.1}{127.0.0.1:9300}]; nested: IndexShardRecoveryException[failed to recover from gateway]; nested: EngineCrea
tionFailureException[failed to create engine]; nested: FileSystemException[/home/ubuntu/photon/photon_data/elasticsearch/data/nodes/0/indices/aMfwwxIpS82qlsyTQtAw_Q/2/tran
slog/translog-507.ckp: Too many open files]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [photon][2]: Recovery failed on {y9nEotM}{y9nEotM6QtmL1y0VdM5Z2g}{_H3DslORRq2vkhU1EQSb2A}{127.0.0.1}{127.0.0.1:
9300}

at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1491) ~[photon-0.3.4.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [photon-0.3.4.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:365) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:90) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:88) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1239) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1487) ~[photon-0.3.4.jar:?]
... 4 more
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: failed to create engine
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:162) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1605) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1587) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1030) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:990) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:360) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:90) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:88) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1239) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1487) ~[photon-0.3.4.jar:?]
... 4 more

Caused by: java.nio.file.FileSystemException: /home/ubuntu/photon/photon_data/elasticsearch/data/nodes/0/indices/aMfwwxIpS82qlsyTQtAw_Q/2/translog/translog-507.ckp: Too ma
ny open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:370) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:421) ~[?:?]
at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.translog.Checkpoint.read(Checkpoint.java:92) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:230) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.translog.Translog.<init>(Translog.java:177) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:271) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:159) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1605) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1587) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:1030) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:990) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:360) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:90) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:88) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1239) ~[photon-0.3.4.jar:?]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1487) ~[photon-0.3.4.jar:?]
... 4 more

InfoDevkota Jan 27, 2021
Author

Using the Type=simple or rerouting the output or because of both, Drops the CPU uses,

lonvia Jan 27, 2021
Maintainer

There is your problem: Too many open files. This is an issue with your operating system. Not something we can help you with here. Try googling the error, there are quite a few explanations on the web what it means.

InfoDevkota Jan 27, 2021
Author

Thank you!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Got 500 Internal Server Error sometime #531

{{title}}

Replies: 1 comment 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Got 500 Internal Server Error sometime #531

InfoDevkota Jan 25, 2021

Replies: 1 comment · 13 replies

karussell Jan 25, 2021 Maintainer

lonvia Jan 26, 2021 Maintainer

InfoDevkota Jan 27, 2021 Author

InfoDevkota Jan 27, 2021 Author

lonvia Jan 27, 2021 Maintainer

InfoDevkota Jan 27, 2021 Author

InfoDevkota
Jan 25, 2021

Replies: 1 comment 13 replies

karussell
Jan 25, 2021
Maintainer

lonvia Jan 26, 2021
Maintainer

InfoDevkota Jan 27, 2021
Author

InfoDevkota Jan 27, 2021
Author

lonvia Jan 27, 2021
Maintainer

InfoDevkota Jan 27, 2021
Author