IGUANA cannot handle queries larger than ~2GB

[liss-h]

Java arrays and probably some other things are for some reason limited to roughtly Integer.MAX_VALUE (= 2^32 / 2) bytes which is just over 2GB. Which is problematic when the query file is larger than that (which can happen in DELETE DATA or INSERT DATA queries) as IGUANA will just crash with an out of memory error.

I don't know if that is fixable

[bigerl]

• Larger queries should always be streamed from disk and not reside at RAM. Configurable threshold.
• Supporting larger queries requires Array with long size type (or streams?)
• two modes:
  • chunked HTTP without content-length
  • regular HTTP request

[bigerl]

This is almost solved. The only remaining issue is:

• Right now, we can only hash queries up to 2GB because of the hashing scheme used. We should use a streaming XXH3 instead