hddm_merge_files utility doesn't work anymore

[jonzarling]

I'm trying to use the hddm_merge_files program to merge smeared and danarest files, as a way to reduce the large number of files that MCWrapper produces. It looks like it doesn't work though. Both smeared files and REST files seem to loop endlessly, writing out way more events than were supplied. Maybe something in the while loop in Process_r.cc and Process_s.cc is busted?

One can verify this behavior on the farm with the following steps:

• cp -r /w/halld-scshelf2101/home/jzarling/github_issues/hddm_merge_REST .
• hddm_merge_files -osmeared_out_test.hddm genEtaRegge_*_smeared.hddm
• or hddm_merge_files -odana_merged_test.hddm dana_rest*.hddm

The original two files contain 20,000 and 319 events respectively. You can see the merged output file quickly exceeds this number.

[aaust]

You can try python instead:
https://halldweb.jlab.org/wiki/index.php/Guide_to_roll-your-own_python_hddm_transforms#example_1:_merging_files
We use this to merge the hddm files during the post-processing.

[rjones30]

Hello Jon, What are you trying to do in "merge smeared and danarest files". This seems like an ill-conceived operation. A REST file is the output from a reconstruction pass, written by hd_ana or hd_root after tracking and clustering, dirc reconstruction, etc. are all done. On the other hand, a smeared file is a pre-reconstruction event format, containing hit objects not tracks and clusters. I don't know of any way these could be merged and have the output be meaningful to anyone who would want to read it.

…

-Richard Jones

On Wed, Aug 24, 2022 at 11:41 AM Jonathan Zarling ***@***.***> wrote: I'm trying to use the hddm_merge_files program to merge smeared and danarest files, as a way to reduce the large number of files that MCWrapper produces. It looks like it doesn't work though. Both smeared files and REST files seem to loop endlessly, writing out way more events than were supplied. Maybe something in the while loop in Process_r.cc <https://github.com/JeffersonLab/halld_recon/blob/master/src/programs/Utilities/hddm_merge_files/Process_r.cc> and Process_s.cc <https://github.com/JeffersonLab/halld_recon/blob/master/src/programs/Utilities/hddm_merge_files/Process_s.cc> is busted? One can verify this behavior on the farm with the following steps: - cp -r /w/halld-scshelf2101/home/jzarling/github_issues/hddm_merge_REST . - hddm_merge_files -osmeared_out_test.hddm genEtaRegge_*_smeared.hddm - or hddm_merge_files -odana_merged_test.hddm dana_rest*.hddm The original two files contain 20,000 and 319 events respectively. You can see the merged output file quickly exceeds this number. — Reply to this email directly, view it on GitHub <#687>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3YKWDWAXNOKWJFYSRA65DV2Y7CHANCNFSM57P2US7Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[rjones30]

Jon, looking at your examples further down in your email, I see that you were trying to merge sets of smeared files into a single smeared file, or sets of rest files into a single rest file, not merge smeared and rest files together. That's how this utility is supposed to work. I tested it and verified that the present build of this utility does not terminate correctly when it hits the EOF at the end of a hddm input stream. I have applied a fix to Process_s.cc and Process_r.cc in the Utilities/hddm_merge_files directory. Can you have a look at the PR I just posted to halld_recon for this, and apply it if you find it works? -Richard Jones

…

On Wed, Aug 31, 2022 at 2:33 PM Richard Jones ***@***.***> wrote: Hello Jon, What are you trying to do in "merge smeared and danarest files". This seems like an ill-conceived operation. A REST file is the output from a reconstruction pass, written by hd_ana or hd_root after tracking and clustering, dirc reconstruction, etc. are all done. On the other hand, a smeared file is a pre-reconstruction event format, containing hit objects not tracks and clusters. I don't know of any way these could be merged and have the output be meaningful to anyone who would want to read it. -Richard Jones On Wed, Aug 24, 2022 at 11:41 AM Jonathan Zarling < ***@***.***> wrote: > I'm trying to use the hddm_merge_files program to merge smeared and > danarest files, as a way to reduce the large number of files that MCWrapper > produces. It looks like it doesn't work though. Both smeared files and REST > files seem to loop endlessly, writing out way more events than were > supplied. Maybe something in the while loop in Process_r.cc > <https://github.com/JeffersonLab/halld_recon/blob/master/src/programs/Utilities/hddm_merge_files/Process_r.cc> > and Process_s.cc > <https://github.com/JeffersonLab/halld_recon/blob/master/src/programs/Utilities/hddm_merge_files/Process_s.cc> > is busted? > > One can verify this behavior on the farm with the following steps: > > - cp -r > /w/halld-scshelf2101/home/jzarling/github_issues/hddm_merge_REST . > - hddm_merge_files -osmeared_out_test.hddm genEtaRegge_*_smeared.hddm > - or hddm_merge_files -odana_merged_test.hddm dana_rest*.hddm > > The original two files contain 20,000 and 319 events respectively. You > can see the merged output file quickly exceeds this number. > > — > Reply to this email directly, view it on GitHub > <#687>, or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AB3YKWDWAXNOKWJFYSRA65DV2Y7CHANCNFSM57P2US7Q> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> >

[rjones30]

Jon, this fix was introduced in response to your problem report. Could you have a look and act on the PR if it fixes the problem? -Richard

…

On Wed, Aug 31, 2022 at 4:38 PM Richard Jones ***@***.***> wrote: Jon, looking at your examples further down in your email, I see that you were trying to merge sets of smeared files into a single smeared file, or sets of rest files into a single rest file, not merge smeared and rest files together. That's how this utility is supposed to work. I tested it and verified that the present build of this utility does not terminate correctly when it hits the EOF at the end of a hddm input stream. I have applied a fix to Process_s.cc and Process_r.cc in the Utilities/hddm_merge_files directory. Can you have a look at the PR I just posted to halld_recon for this, and apply it if you find it works? -Richard Jones On Wed, Aug 31, 2022 at 2:33 PM Richard Jones ***@***.***> wrote: > Hello Jon, > > What are you trying to do in "merge smeared and danarest files". This > seems like an ill-conceived operation. A REST file is the output from a > reconstruction pass, written by hd_ana or hd_root after tracking and > clustering, dirc reconstruction, etc. are all done. On the other hand, a > smeared file is a pre-reconstruction event format, containing hit objects > not tracks and clusters. I don't know of any way these could be merged and > have the output be meaningful to anyone who would want to read it. > > -Richard Jones > > On Wed, Aug 24, 2022 at 11:41 AM Jonathan Zarling < > ***@***.***> wrote: > >> I'm trying to use the hddm_merge_files program to merge smeared and >> danarest files, as a way to reduce the large number of files that MCWrapper >> produces. It looks like it doesn't work though. Both smeared files and REST >> files seem to loop endlessly, writing out way more events than were >> supplied. Maybe something in the while loop in Process_r.cc >> <https://github.com/JeffersonLab/halld_recon/blob/master/src/programs/Utilities/hddm_merge_files/Process_r.cc> >> and Process_s.cc >> <https://github.com/JeffersonLab/halld_recon/blob/master/src/programs/Utilities/hddm_merge_files/Process_s.cc> >> is busted? >> >> One can verify this behavior on the farm with the following steps: >> >> - cp -r >> /w/halld-scshelf2101/home/jzarling/github_issues/hddm_merge_REST . >> - hddm_merge_files -osmeared_out_test.hddm genEtaRegge_*_smeared.hddm >> - or hddm_merge_files -odana_merged_test.hddm dana_rest*.hddm >> >> The original two files contain 20,000 and 319 events respectively. You >> can see the merged output file quickly exceeds this number. >> >> — >> Reply to this email directly, view it on GitHub >> <#687>, or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AB3YKWDWAXNOKWJFYSRA65DV2Y7CHANCNFSM57P2US7Q> >> . >> You are receiving this because you are subscribed to this thread.Message >> ID: ***@***.***> >> >

[jonzarling]

Thanks very much for working on the solution Richard, I was able to try today. Looks like it's partially working.

For the case of merging smeared hit-level files, the fix appears to work. However, the output file appears a little more than twice the input files in size. Maybe the output compression is different?

For the case of merging REST data files, things aren't quite working yet. The updated hddm_merge_files does loop over all events and exit fine, but the file outputted is too small and when running hd_dump appears to contain no events. That might need something additional done to it.