Adding option to SimTelEventSource to include MC non triggered events

[bryankim96]

Following on the discussion from #592, we can now use eventio.SimTelFile.iter_mc_events() to loop over non-triggered MC events in simtel files. Are there plans to expose this functionality through to ctapipe's SimTelEventSource (possibly through some sort of optional flag) so it's not necessary to always use eventio directly when working with non-triggered MC data?

[nietootein]

This feature would be very useful to have! We are about to resume the reduction of the current reference dataset (prod3b nsb1x) to DL1DH format and having this feature available will enable us to store the non-triggered MC events into our events tables, tremendously facilitating the generation of figures that require access to this information (e.g. effective areas, trigger efficiencies, etc.), directly from the reduced h5 files.

[vuillaut]

This feature would be very useful to have! We are about to resume the reduction of the current reference dataset (prod3b nsb1x) to DL1DH format and having this feature available will enable us to store the non-triggered MC events into our events tables, tremendously facilitating the generation of figures that require access to this information (e.g. effective areas, trigger efficiencies, etc.), directly from the reduced h5 files.

@nietootein Though the option would be nice, it's not a show stopper for the reduction of the reference dataset (prod3b nsb1x) to DL1DH format as for now we are only dealing with power-law simulated distributions and the power-law parameters are now passed along in the metadata.
The reduction should be resumed without waiting for additional options imho.

[maxnoe]

There are also tue histograms at the end of the files summarizing produced and triggered events.

Can get it using source._file.histograms after all events have been read.

[bryankim96]

@vuillaut @nietootein For the time being I'm going to finish implementing the temporary workaround using eventio.SimTelFile.iter_mc_events(), as I've already started on it. I'll finish it up and test it later today with some other minor changes. With that said, doing it this way makes the code messier, adds a new explicit dependency on eventio, and breaks the design assumption of the writer (that it should depend only on ctapipe event sources and containers). So if it's not strictly required and we can avoid the requirement to read additional information from eventio, I think that would be preferred.

[maxnoe]

Storing an addtional table with these histograms might be a good idea

[bryankim96]

@maxnoe I just tried looking at source.file_.histograms for one of the sample simtel files we're using, but it's empty (None). This is after exhausting the EventSource iterator. I also went through all events in the underlying SimTelFile separately w/:

for event in event_source.file_:
    #do something

Is the histogram not always included in every SimTel file?

[maxnoe]

Could be, you can check with eventio_print_object_information.

You can also plot any histograms if there are any in the using eventio_plot_histograms

Are you trying with a test file of ctapipe? Many of those are truncated, so the histograms will be missing as they are the very last objects to be written.

[bryankim96]

@maxnoe Yeah, after using those scripts it looks like there are no histograms in this particular file I have (I guess it was truncated?). I don't think it is a ctapipe test file though, it's a file from Prod3. I don't know the details about its origins, but @nietootein will know. The filename is:

gamma_20deg_0deg_run100___cta-prod3-demo_desert-2150m-Paranal-baseline-sample_cone10.simtel.gz

[nietootein]

Hi @bryankim96. You are right, that file is truncated. I think it's one of the files we prepared for the hands-on session in Montpellier. I'll point you to a regular file (PM).

[maxnoe]

I am working on providing this option in pyeventio here: cta-observatory/pyeventio#214