Polyglot Crashing

[jimboca]

I've been having an issue where Polyglot just stops, all the nodeservers stay "Connected" but Polyglot is not responding, and all nodeserver heartbeats stop and nothing showed up in the logs. So I ran it in the forground with:
~/polyglot/polyglot-v2-linux-armv7 > out.log 2>&1 &
and then when I came back after I noticed the problem (I was traveling so couldn't check it right away) the terminal showed:
[1]+ Aborted ~/polyglot/polyglot-v2-linux-armv7 > out.log 2>&1
And the last of the log showed:

<--- Last few GCs --->

[29860:0x2c3c748] 321641335 ms: Mark-sweep 227.7 (231.4) -> 227.7 (231.4) MB, 868.9 / 0.0 ms  (average mu = 0.877, current mu = 0.003) last resort GC in old space requested
[29860:0x2c3c748] 321642404 ms: Mark-sweep 227.7 (231.4) -> 227.6 (231.4) MB, 1069.7 / 0.0 ms  (average mu = 0.790, current mu = 0.000) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x14b15cc]
    1: StubFrame [pc: 0x140b164]
Security context: 0x2808dbd5 <JSObject>
    2: command [0x4d3dc6d9] [/snapshot/polyglot-v2/node_modules/mongodb-core/lib/wireprotocol/2_4_support.js:~149] [pc=0x47a8850c](this=0x4cfc6931 <WireProtocol map = 0x5a12aaf5>,0x70a5f375 <BSON map = 0x4d53ae7d>,0x4cfc9269 <String[20]: polyglot.nodeservers>,0x24b3b929 <Object map = 0x5a11ad85>,0x24b3ba95 <Object map = 0x5a11b325>,0x...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

This was happening with 2.2.3 (although could have been a different error) and still happening with 2.2.4.

1 This problem needs to be figured out
2 We need a way to debug this without having to run Polyglot in the forground
3 The nodeservers need to be started in a way in which they will die with the parent Polyglot also dies

Let me know if you want the entire out.log to debug.

Another user has reported a similar issue.

[jimboca]

Not sure if this applies? Seems to say that it's an issue in node 10 and can be fixed by passing memory options?
https://stackoverflow.com/questions/26094420/fatal-error-call-and-retry-last-allocation-failed-process-out-of-memory

[Einstein42]

export NODE_OPTIONS=--max_old_space_size=4096

Then run polyglot. See if it still crashes.

[jimboca]

Thanks, will try that. Also, just noticed node dumped a json file and it says:

      "old_space": {
        "memorySize": 238096384,
        "committedMemory": 237676024,
        "capacity": 233911020,
        "used": 233911020,
        "available": 0
      },

[jimboca]

Using 4096 is to much for my RPi 3... This is all I get

<--- Last few GCs --->


<--- JS stacktrace --->


#
# Fatal process OOM in insufficient memory to create an Isolate
#

[jimboca]

And cutting in half fails with no info

pi@rpi3-2:~/.polyglot/log $ export NODE_OPTIONS=--max_old_space_size=2048
pi@rpi3-2:~/.polyglot/log $ ~/polyglot/polyglot-v2-linux-armv7
11/8/2019, 09:28:14 [polyglot] debug: Created PID file: /home/pi/.polyglot/polyglot.pid
pi@rpi3-2:~/.polyglot/log $ ps -ef | grep poly
pi       11705  1019  0 09:28 pts/0    00:00:00 grep --color=auto poly

[Einstein42]

modified armv7 build to use node10 vs latest, @jimboca currently testing

[jimboca]

Was testing this, but nodeservers were still eventually failing to respond so moved back to 2.2.4 but that is showing the same problem so have no idea what is going on. @Einstein42 and I tried to debug but currently no idea what is happening.

[jimboca]

Went back to last build mentioned 2 comments up, and running from command line, will see what happens.