Evaluate using awkwardarray for variable length arrays

[maxnoe]

We should evaluate if we can use awkward to speed things up where we currently use list of lists or arrays of arrays where we have variable length data.

https://awkward-array.org/quickstart.html

[kosack]

I tried some experiments using it in ctapipe a year ago or so, but decided in the end it would require too much redesign. Although it depends where and how heavily we would use it. It has some implications (and benefits) also for data format (e.g. ability to write these var-length arrays to parquet).

It's interesting technology though, and could help solve the problem of event-wise vs bunch-of-event processing. Ideally we would like to support the latter and do away with all explicit loops over events for efficiency purposes, but the current design makes that difficult. My main issue with awkward was just that it was not very stable, but now that there is a 1.x release, that is encouraging.

Places where it could be interesting to explore using it would be:

• internal representation of merged telescope event data (avoid loops over telescopes, perhaps)
• storing sparse waveforms and images (data volume reduced)
• replace event structure entirely, allowing algorithms potentially to be easily applied at to bunches of events (no event loop)
• instrument model

[maxnoe]

Here I was mainly talking about the places where simtel array uses variable length data and that have a quite large performance impact when reading eventio.

These are at the moment:

• pixel sector information eventio/simtel/parsing.pyx, which we currently parse into a list[array.array('h')]
• Zero suppressed adc samples (currently no files are produced like this)

[kosack]

Perhaps a similar issue should be opened for pyeventio, since there it's clearly useful.

[maxnoe]

Sorry, I misclicked. I intended this to be an eventio issue.