Track image duplication due to multiple PL runs

[d-l-walker]

This issue is for tracking cases where there are multiple copies of an image in the /calibrated/working/ directory for a given SB. This is due to multiple pipeline imaging runs for the same SB, and will result in filenames like:

uid___A001_X15a0_XYZ.s12_0.Sgr_A_star_sci.spwN.cube.I.iter1.image.pbcor
uid___A001_X15a0_XYZ.s38_0.Sgr_A_star_sci.spwN.cube.I.iter1.image.pbcor

where the only differentiating point is the sXY_0 part of the filename, which corresponds to the number of the pipeline stage at which the images were generated.

As a start, I have identified cases where this occurs for the HNCO SPW (SPWs 31/22/23 for 12m/7m/TP):

12m

Region	12m cube	SPW
j	uid___A001_X15b4_X3d.s12_0.Sgr_A_star_sci.spw31.cube.I.iter1.image.pbcor/	31
j	uid___A001_X15b4_X3d.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.image.pbcor/	31

7m

Region	7m cube	SPW
a	uid___A001_X1590_X30ac.s12_0.Sgr_A_star_sci.spw16.cube.I.iter1.image.pbcor/	16
a	uid___A001_X1590_X30ac.s38_0.Sgr_A_star_sci.spw16.cube.I.iter1.image.pbcor/	16
a	uid___A001_X1590_X30ac.s12_0.Sgr_A_star_sci.spw18.cube.I.iter1.image.pbcor/	18
a	uid___A001_X1590_X30ac.s38_0.Sgr_A_star_sci.spw18.cube.I.iter1.image.pbcor/	18
a	uid___A001_X1590_X30ac.s12_0.Sgr_A_star_sci.spw22.cube.I.iter1.image.pbcor/	22
a	uid___A001_X1590_X30ac.s38_0.Sgr_A_star_sci.spw22.cube.I.iter1.image.pbcor/	22
a	uid___A001_X1590_X30ac.s12_0.Sgr_A_star_sci.spw24.cube.I.iter1.image.pbcor/	24
a	uid___A001_X1590_X30ac.s38_0.Sgr_A_star_sci.spw24.cube.I.iter1.image.pbcor/	24
a	uid___A001_X1590_X30ac.s12_0.Sgr_A_star_sci.spw26.cube.I.iter1.image.pbcor/	26
a	uid___A001_X1590_X30ac.s38_0.Sgr_A_star_sci.spw26.cube.I.iter1.image.pbcor/	26
ao	uid___A001_X15a0_X192.s13_0.Sgr_A_star_sci.spw16.cube.I.iter1.image.pbcor/	16
ao	uid___A001_X15a0_X192.s38_0.Sgr_A_star_sci.spw16.cube.I.iter1.image.pbcor/	16
ao	uid___A001_X15a0_X192.s13_0.Sgr_A_star_sci.spw18.cube.I.iter1.image.pbcor/	18
ao	uid___A001_X15a0_X192.s38_0.Sgr_A_star_sci.spw18.cube.I.iter1.image.pbcor/	18
ao	uid___A001_X15a0_X192.s13_0.Sgr_A_star_sci.spw20.cube.I.iter1.image.pbcor/	20
ao	uid___A001_X15a0_X192.s38_0.Sgr_A_star_sci.spw20.cube.I.iter1.image.pbcor/	20
ao	uid___A001_X15a0_X192.s13_0.Sgr_A_star_sci.spw22.cube.I.iter1.image.pbcor/	22
ao	uid___A001_X15a0_X192.s38_0.Sgr_A_star_sci.spw22.cube.I.iter1.image.pbcor/	22
c	uid___A001_X15a0_Xae.s11_0.Sgr_A_star_sci.spw16.cube.I.iter1.image.pbcor/	16
c	uid___A001_X15a0_Xae.s13_0.Sgr_A_star_sci.spw16.cube.I.iter1.image.pbcor/	16
c	uid___A001_X15a0_Xae.s11_0.Sgr_A_star_sci.spw18.cube.I.iter1.image.pbcor/	18
c	uid___A001_X15a0_Xae.s13_0.Sgr_A_star_sci.spw18.cube.I.iter1.image.pbcor/	18
c	uid___A001_X15a0_Xae.s11_0.Sgr_A_star_sci.spw20.cube.I.iter1.image.pbcor/	20
c	uid___A001_X15a0_Xae.s13_0.Sgr_A_star_sci.spw20.cube.I.iter1.image.pbcor/	20
c	uid___A001_X15a0_Xae.s11_0.Sgr_A_star_sci.spw22.cube.I.iter1.image.pbcor/	22
c	uid___A001_X15a0_Xae.s13_0.Sgr_A_star_sci.spw22.cube.I.iter1.image.pbcor/	22
u	uid___A001_X15a0_X11a.s12_0.Sgr_A_star_sci.spw16.cube.I.iter1.image.pbcor/	16
u	uid___A001_X15a0_X11a.s38_0.Sgr_A_star_sci.spw16.cube.I.iter1.image.pbcor/	16
u	uid___A001_X15a0_X11a.s12_0.Sgr_A_star_sci.spw18.cube.I.iter1.image.pbcor/	18
u	uid___A001_X15a0_X11a.s38_0.Sgr_A_star_sci.spw18.cube.I.iter1.image.pbcor/	18
u	uid___A001_X15a0_X11a.s12_0.Sgr_A_star_sci.spw22.cube.I.iter1.image.pbcor/	22
u	uid___A001_X15a0_X11a.s38_0.Sgr_A_star_sci.spw22.cube.I.iter1.image.pbcor/	22
v	uid___A001_X15a0_X120.s12_0.Sgr_A_star_sci.spw16.cube.I.iter1.image.pbcor/	16
v	uid___A001_X15a0_X120.s38_0.Sgr_A_star_sci.spw16.cube.I.iter1.image.pbcor/	16
v	uid___A001_X15a0_X120.s12_0.Sgr_A_star_sci.spw18.cube.I.iter1.image.pbcor/	18
v	uid___A001_X15a0_X120.s38_0.Sgr_A_star_sci.spw18.cube.I.iter1.image.pbcor/	18
v	uid___A001_X15a0_X120.s12_0.Sgr_A_star_sci.spw20.cube.I.iter1.image.pbcor/	20
v	uid___A001_X15a0_X120.s38_0.Sgr_A_star_sci.spw20.cube.I.iter1.image.pbcor/	20
v	uid___A001_X15a0_X120.s12_0.Sgr_A_star_sci.spw22.cube.I.iter1.image.pbcor/	22
v	uid___A001_X15a0_X120.s38_0.Sgr_A_star_sci.spw22.cube.I.iter1.image.pbcor/	22
v	uid___A001_X15a0_X120.s12_0.Sgr_A_star_sci.spw24.cube.I.iter1.image.pbcor/	24
v	uid___A001_X15a0_X120.s38_0.Sgr_A_star_sci.spw24.cube.I.iter1.image.pbcor/	24
v	uid___A001_X15a0_X120.s12_0.Sgr_A_star_sci.spw26.cube.I.iter1.image.pbcor/	26
v	uid___A001_X15a0_X120.s38_0.Sgr_A_star_sci.spw26.cube.I.iter1.image.pbcor/	26
y	uid___A001_X15a0_X132.s13_0.Sgr_A_star_sci.spw24.cube.I.iter1.image.pbcor/	24
y	uid___A001_X15a0_X132.s38_0.Sgr_A_star_sci.spw24.cube.I.iter1.image.pbcor/	24

Next steps

• I need to expand this to include all images, not just the HNCO cubes. I scripted this process, so it should be straightforward to extend this to the other images.
• Once the previous step is done, I'll update this issue with all of the duplicates in tabulated form.
• Once we've identified the duplicate images, then we need to determine the best way to distinguish between them so that we can discard the old versions.

Other things to think about ...

• We have various cases of manual re-imaging, where we have then uploaded the re-imaged files to the ~/upload/ folder via globus. We should verify that all such cases have been ingested with the expected file naming convention, such that they can be properly picked up by our scripts.
• Similarly, the two fields that were manually imaged during QA2 (Execution Block ID uid://A001/X15a0/X184 Sgr_A_st_am_03_TM1 #259, Execution Block ID uid://A001/X15a0/X17e Sgr_A_st_al_03_TM1 #126) will have filenames reflecting the fact that they were manually imaged. These should be updated to fit the standard convention for the same reasons.