JP-251: Optimal extraction support for MIRI LRS

[melanieclarke]

Resolves JP-251

Adding support for PSF-based optimal extraction. There is currently only one PSF model available, for MIRI LRS fixed slit mode, but the code is written so that only a PSF reference file is required to use optimal extraction. To test it for other modes, provide an override_psf file.

Recommended parameters for running spec2 or spec3 with optimal extraction are:
--steps.resample_spec.skip=true
--steps.pixel_replace.skip=true
--steps.extract_1d.extraction_type=optimal
--steps.extract_1d.use_source_posn=true
--steps.extract_1d.model_nod_pair=true
--steps.extract_1d.optimize_psf_location=true
--steps.extract_1d.save_profile=true
--steps.extract_1d.save_residual_image=true

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.fits_generator.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[codecov]

Codecov Report

Attention: Patch coverage is 98.48485% with 7 lines in your changes missing coverage. Please review.

Project coverage is 73.73%. Comparing base (92086e9) to head (1fa5ca5).
Report is 51 commits behind head on main.

Files with missing lines	Patch %	Lines
jwst/pipeline/calwebb_spec3.py	0.00%	4 Missing ⚠️
jwst/extract_1d/extract_1d_step.py	93.93%	2 Missing ⚠️
jwst/resample/resample_spec_step.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8967      +/-   ##
==========================================
- Coverage   78.35%   73.73%   -4.62%     
==========================================
  Files         505      373     -132     
  Lines       46057    37272    -8785     
==========================================
- Hits        36087    27482    -8605     
+ Misses       9970     9790     -180     

Flag	Coverage Δ
nightly	?

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[melanieclarke]

First regtest run here:
https://github.com/spacetelescope/RegressionTests/actions/runs/12693569699

First attempt shows many regtest failures for MIRI due to missing PSF references for non-LRS-FS modes. It looks like we need 'N/A's in CRDS for those modes.

[melanieclarke]

The MIRI rmap has been updated with a N/A fallback for the PSF type. Rerunning regression tests here:
https://github.com/spacetelescope/RegressionTests/actions/runs/13060561722

[melanieclarke]

Regtests now show only two diffs:

1. Small numerical differences in a residual data product from the new optimal extraction test. I generated the truth file locally, before we moved to numpy 2.
2. Extraction aperture centering for MIRI LRS slit spec2, from an unresampled cal file, shifted by about 1/100 of a pixel, causing some small numerical differences in the output spectrum.

I don't think either of these are concerning - I can okify them after merging, if @tapastro agrees?

[tapastro]

Agreed! Okify away.