JP-2546: Refactor wfss_contam to support multiprocessing and decrease runtime

[emolter]

Resolves JP-2546

Closes #7288

This PR addresses ...

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[emolter]

@Rplesha @Russell-Ryan This should run now. I tested that the step runs both standalone (via WfssContamStep.call) and as part of the spec2 pipeline (via from_cmdline) on NIRISS dataset jw02738-o001_20231124t145029_spec2_00013_asn.json. I set the brightest_n parameter to a small value to keep runtime small. I didn't do any additional validation of the outputs beyond what I did several months ago.

Let me know if you're able to run it, and please let me know if you run into any issues or have any questions.

Adding a known issue so I don't forget, and in case y'all encounter it too: the indexing in this step seems not to play nicely with the indexing from extract_2d when the extract_2d.wfss_nbright parameter is set to a non-default value. This currently causes the step to think the first source has ~10 million pixels in it for some reason.

[codecov]

Codecov Report

Attention: Patch coverage is 79.83539% with 49 lines in your changes missing coverage. Please review.

Project coverage is 74.06%. Comparing base (16aa8b0) to head (4a6d4ad).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
jwst/wfss_contam/wfss_contam.py	66.94%	40 Missing ⚠️
jwst/wfss_contam/observations.py	92.68%	6 Missing ⚠️
jwst/wfss_contam/wfss_contam_step.py	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9220      +/-   ##
==========================================
+ Coverage   73.66%   74.06%   +0.40%     
==========================================
  Files         368      368              
  Lines       36392    36484      +92     
==========================================
+ Hits        26808    27022     +214     
+ Misses       9584     9462     -122     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tapastro]

Adding a known issue so I don't forget, and in case y'all encounter it too: the indexing in this step seems not to play nicely with the indexing from extract_2d when the extract_2d.wfss_nbright parameter is set to a non-default value. This currently causes the step to think the first source has ~10 million pixels in it for some reason.

That kind of sounds like the step is trying to assign all pixels in the segmentation map with value 0 to the first source - perhaps the step needs to 1-index calls to the segmentation map array?

[emolter]

That kind of sounds like the step is trying to assign all pixels in the segmentation map with value 0 to the first source - perhaps the step needs to 1-index calls to the segmentation map array?

I thought of that too. But why would this only be the case when setting the parameter in extract_2d?
I haven't really tried to debug this yet. The meta-question is, should I spend time debugging the changes now? I am happy to do so but it sounded like we weren't yet sure this is a priority.

[tapastro]

Probably not worth working on unless it becomes an issue during INS testing.

[emolter]

I just fixed a bug that was causing my NIRCam test dataset (jw01076-o103_20231023t160322_spec2_00003_asn.json) to fail, so if you already downloaded a local copy of this, it would be good to update it.