JP-251: Optimal extraction support for MIRI LRS

changes/8967.extract_1d.rst

@@ -0,0 +1 @@
+Add options for PSF-based optimal extraction for point sources in MIRI LRS fixed slit exposures.

docs/jwst/extract_1d/arguments.rst

@@ -15,11 +15,22 @@ The following arguments apply to all modes unless otherwise specified.
 
 ``--apply_apcorr``
   Switch to select whether or not to apply an APERTURE correction during the
-  Extract1dStep processing. Default is ``True``. Has no effect for NIRISS SOSS data.
+  Extract1dStep processing. Default is ``True``. Has no effect for NIRISS SOSS data
+  or for optimal extractions.
 
 Step Arguments for Slit and Slitless Spectroscopic Data
 -------------------------------------------------------
 
+``--extraction_type``
+  Specify the extraction type.
+  If 'box', a standard extraction is performed, summing over an aperture box.
+  If 'optimal', a PSF-based optimal extraction is performed.
+  If None, optimal extraction is attempted whenever use_source_posn is True.
+  Box extraction is suitable for any input data (point sources and extended sources;
+  resampled and unresampled images).  Optimal extraction is best suited for unresampled
+  point sources. Currently, optimal extraction is only available for MIRI LRS Fixed Slit data.
+  The default extraction type is 'box'.
+
 ``--use_source_posn``
   Specify whether the target and background extraction
   region locations specified in the :ref:`EXTRACT1D <extract1d_reffile>` reference
@@ -34,6 +45,21 @@ Step Arguments for Slit and Slitless Spectroscopic Data
   Specify a number of pixels (fractional pixels are allowed) to offset the 
   extraction aperture from the nominal position.  The default is 0.
 
+``--model_nod_pair``
+  Flag to enable fitting a negative trace during optimal extraction.
+  If True, and the extraction type is 'optimal', then a negative trace
+  from nod subtraction is modeled alongside the positive source during
+  extraction.  This will be attempted only if the input data has been background
+  subtracted and the dither pattern type indicates that 2 nods were used.
+  The default value is True.
+
+``--optimize_psf_location``
+  Flag to enable PSF location optimization during optimal extraction.
+  If True, and the extraction type is 'optimal', then the placement of
+  the PSF model for the source location (and negative nod, if present)
+  will be iteratively optimized. This parameter is recommended if
+  negative nods are modeled.  The default value is True.
+
 ``--smoothing_length``
   If ``smoothing_length`` is greater than 1 (and is an odd integer), the
   image data used to perform background extraction will be smoothed in the
@@ -82,13 +108,17 @@ Step Arguments for Slit and Slitless Spectroscopic Data
   to report on progress. Default value is 50.
 
 ``--save_profile``
-  Flag to enable saving the spatial profile representing the extraction aperture.
+  Flag to enable saving the spatial profile representing the extraction aperture or model.
   If True, the profile is saved to disk with suffix "profile".
 
 ``--save_scene_model``
-  Flag to enable saving a model of the 2D flux as defined by the extraction aperture.
+  Flag to enable saving a model of the 2D flux as defined by the extraction aperture or PSF model.
   If True, the model is saved to disk with suffix "scene_model".
 
+``--save_residual_image``
+  Flag to enable saving the residual image (from the input minus the scene model)
+  If True, the model is saved to disk with suffix "residual".
+
 Step Arguments for IFU Data
 ---------------------------
 

docs/jwst/extract_1d/description.rst

@@ -18,11 +18,14 @@ The EXTRACT1D reference file is not used for Wide-Field Slitless Spectroscopy da
 the full size of the input 2D subarray or cutout for each source, or restricted to
 the region within which the world coordinate system (WCS) is defined in each cutout.
 
-For slit-like 2D input data, source and background extractions are done using
-a rectangular aperture that covers one pixel in the dispersion direction and
+For slit-like 2D input data, source and background extractions are, by default, done
+using a rectangular aperture that covers one pixel in the dispersion direction and
 uses a height in the cross-dispersion direction that is defined by parameters in
-the EXTRACT1D reference file.
-For 3D IFU data, on the other hand, the extraction options differ depending on
+the EXTRACT1D reference file.  Optionally, for point sources, a PSF-based optimal
+extraction may be performed, using a model of the spectral PSF to fit the total flux
+at each dispersion element.
+
+For 3D IFU data, the extraction options differ depending on
 whether the target is a point or extended source.  For a point
 source, the spectrum is extracted using circular aperture photometry,
 optionally including background subtraction using a circular annulus.
@@ -36,28 +39,29 @@ object and perform extraction.  For 3D NIRSpec fixed slit rateints data, the
 ``extract_1d`` step will be skipped as 3D input for the mode is not supported.
 
 
-For most spectral modes an aperture correction will be applied to the extracted
+For most spectral modes, an aperture correction will be applied to the extracted
 1D spectral data (unless otherwise selected by the user), in order to put the
 results onto an infinite aperture scale.
 This is done by creating interpolation functions based on the APCORR reference
 file data and applying the interpolated aperture correction (a multiplicative
 factor between 0 and 1) to the extracted, 1D spectral data (corrected data
 include the "flux", "surf_bright", "flux_error", "sb_error", and all flux and
-surface brightness variance columns in the output table).
+surface brightness variance columns in the output table).  For optimal extractions,
+aperture correction is not performed, since it is assumed the total flux is
+modeled by the PSF.
 
 Input
 -----
-Calibrated and potentially resampled 2D images or 3D cubes. The format should be a
+The input data are calibrated and potentially resampled 2D images or 3D cubes.
+The format should be a
 CubeModel, SlitModel, IFUCubeModel, ImageModel, MultiSlitModel, or a ModelContainer.
 For some JWST modes this is usually a resampled product, such as the "s2d" products
 for MIRI LRS fixed-slit, NIRSpec fixed-slit, and NIRSpec MOS, or the "s3d" products
 for MIRI MRS and NIRSpec IFU. For other modes that are not resampled (e.g. MIRI
 LRS slitless, NIRISS SOSS, NIRSpec BOTS, and NIRCam and NIRISS WFSS), this will
 be a "cal" or "calints" product.
 For modes that have multiple slit instances (NIRSpec fixed-slit and MOS, WFSS),
-the SCI extensions should have the keyword SLTNAME to specify which slit was extracted,
-though if there is only one slit (e.g. MIRI LRS and NIRISS SOSS), the slit name can
-be taken from the EXTRACT1D reference file instead.
+the SCI extensions should have the keyword SLTNAME to specify which slit was extracted.
 
 Normally the :ref:`photom <photom_step>` step should be applied before running
 ``extract_1d``.  If ``photom`` has not been run, a warning will be logged and the
@@ -94,13 +98,13 @@ FLUX_ERROR is the error estimate for FLUX; it has the
 same units as FLUX. The error is calculated as the square root of the sum of the
 three variance arrays: Poisson, read noise (RNOISE), and flat field (FLAT).
 SURF_BRIGHT is the surface brightness in MJy / sr, except that for point
-sources observed with NIRSpec and NIRISS SOSS, SURF_BRIGHT will be set to
+sources observed with NIRSpec and NIRISS SOSS, or optimal extractions, SURF_BRIGHT will be set to
 zero, because there is no way to express the extracted results from those modes
 as a surface brightness. SB_ERROR is the error estimate for SURF_BRIGHT, calculated
 in the same fashion as FLUX_ERROR but using the SB_VAR arrays. While it's expected
 that a user will make use of the FLUX column for point-source data and the
-SURF_BRIGHT column for an extended source, both columns are populated (except for
-NIRSpec and NIRISS SOSS point sources, as mentioned above).
+SURF_BRIGHT column for an extended source, both columns are populated
+(except as mentioned above).
 
 The ``extract_1d`` step collapses the input data from 2-D to 1-D by summing
 one or more rows (or columns, depending on the dispersion direction).
@@ -134,9 +138,9 @@ otherwise.
 
 .. _extract-1d-for-slits:
 
-Extraction for 2D Slit Data
----------------------------
-The operational details of the 1D extraction depend heavily on the parameter
+Box Extraction for 2D Slit Data
+-------------------------------
+For standard box extractions, the operational details depend heavily on the parameter
 values given in the :ref:`EXTRACT1D <extract1d_reffile>` reference file.
 Here we describe their use within the ``extract_1d`` step.
 
@@ -305,6 +309,48 @@ a separate polynomial. However, the independent variable (wavelength or pixel)
 does need to be the same for all polynomials for a given slit.
 
 
+Optimal Extraction for 2D Slit Data
+-----------------------------------
+
+Optimal extraction proceeds similarly to box extraction for 2D slit data, except that
+instead of summing over an aperture defined by the reference files, a model of the point
+spread function (PSF) is fit to the data at each dispersion element.  This generally provides
+higher signal-to-noise for the output spectrum than box extractions and has the advantage
+of ignoring missing data due to bad pixels, cosmic rays, or saturation.  Optimal extraction
+also does not require a resampled spectral image as input: it can avoid the extra interpolation
+by directly fitting the spatial profile along the curved trace at each dispersion element.
+
+Optimal extraction is suited only to point sources with known source locations, for which a
+high-fidelity PSF model is available.  Currently, only the MIRI LRS fixed slit exposure type
+has a PSF model available in CRDS.
+
+When optimal extraction is selected (`extraction_type = 'optimal'`), the aperture definitions in
+the extraction reference file are ignored, and the following parameters
+are used instead:
+
+* `use_source_posn`: Source position is estimated from the input metadata and used to
+  center the PSF model.  The recommended value is True, in order to account for spatial offsets
+  within the slit; if False, or if the source position could not be estimated, the source is
+  assumed to be at the center of the slit.
+* `model_nod_pair`: If nod subtraction occurred prior to extraction, setting this option to
+  True will allow the extraction algorithm to model a single negative trace from the nod pair
+  alongside the positive trace. This can be helpful in accounting for PSF overlap between the
+  positive and negative traces.  This option is ignored if no background subtraction occurred,
+  or if the dither pattern was not a 2-point nod.
+* `optimize_psf_location`: Since source position estimates may be slightly inaccurate,
+  it may be useful to iteratively optimize the PSF location.  When this option is set to True, the
+  location of the positive and negative traces (if used) are optimized with respect to the
+  residuals of the scene modeled by the PSF at that location.  This option is
+  strongly recommended if `model_nod_pair` is True, since the negative nod location is less
+  reliably estimated than the positive trace location.
+* `subtract_background`: Unlike during box extraction, the background levels can be modeled and removed
+  during optimal extraction without explicitly setting a background region.  It is recommended to
+  set this parameter to True if background subtraction was skipped prior to extraction. Set this
+  parameter to False if a negative nod trace is present but not modeled (`model_nod_pair = False`).
+* `override_psf`: If a custom flux model is required, it is possible to provide one by overriding
+  the PSF model reference file. Set this parameter to the filename for a FITS file matching the
+  :ref:`SpecPsfModel <psf_reffile>` format.
+
 .. _extract-1d-for-ifu:
 
 Extraction for 3D IFU Data

docs/jwst/extract_1d/extract1d_api.rst

@@ -4,3 +4,5 @@ Python interfaces for 1D Extraction
 .. automodapi:: jwst.extract_1d.extract
 .. automodapi:: jwst.extract_1d.extract1d
 .. automodapi:: jwst.extract_1d.ifu
+.. automodapi:: jwst.extract_1d.source_location
+.. automodapi:: jwst.extract_1d.psf_profile

docs/jwst/extract_1d/reference_files.rst

@@ -1,6 +1,9 @@
 Reference File
 ==============
-The ``extract_1d`` step uses an EXTRACT1D reference file and an APCORR reference file.
+For most modes, the ``extract_1d`` step uses an EXTRACT1D reference file and an
+APCORR reference file.  For optimal extraction, it additionally uses a PSF
+reference file.
 
 .. include:: ../references_general/extract1d_reffile.inc
 .. include:: ../references_general/apcorr_reffile.inc
+.. include:: ../references_general/psf_reffile.inc

docs/jwst/references_general/psf_reffile.inc

@@ -0,0 +1,44 @@
+.. _psf_reffile:
+
+PSF Reference File
+^^^^^^^^^^^^^^^^^^
+
+:REFTYPE: PSF
+:Data model: `~jwst.datamodels.SpecPsfModel`
+
+The PSF reference file contains a model of the 1-D point spread function
+by wavelength, intended to support spectral modeling and extraction.
+
+
+Reference Selection Keywords for PSF
+++++++++++++++++++++++++++++++++++++
+CRDS selects appropriate PSF references based on the following keywords.
+PSF is not applicable for instruments not in the table.
+All keywords used for file selection are *required*.
+
+========== =========================================================================
+Instrument Keywords
+========== =========================================================================
+MIRI       INSTRUME, DETECTOR, FILTER, EXP_TYPE
+========== =========================================================================
+
+.. include:: ../includes/standard_keywords.inc
+
+Reference File Format
++++++++++++++++++++++
+PSF reference files are in FITS format, with 2 IMAGE extensions.
+The FITS primary HDU does not contain a data array.
+The format and content of the file is as follows:
+
+=======  ========  =====  ==============  =========
+EXTNAME  XTENSION  NAXIS  Dimensions      Data type
+=======  ========  =====  ==============  =========
+PSF      IMAGE       2    ncols x nrows   float
+WAVE     IMAGE       1    ncols           float
+=======  ========  =====  ==============  =========
+
+The values in the ``PSF`` array give relative spectral flux values by cross-dispersion
+position, at each dispersion element specified in the ``WAVE`` array.  Detector
+pixels are subsampled by the amount specified in the SUBPIX keyword, and the PSF
+is centered on the cross-dispersion element specified in either CENTCOL (for vertical
+dispersion) or CENTROW (for horizontal dispersion).

docs/jwst/references_general/references_general.rst

@@ -100,6 +100,8 @@ documentation on each reference file.
 +                                                       +--------------------------------------------------+
 |                                                       | :ref:`APCORR <apcorr_reffile>`                   |
 +                                                       +--------------------------------------------------+
+|                                                       | :ref:`PSF <psf_reffile>`                         |
++                                                       +--------------------------------------------------+
 |                                                       | SPECKERNEL (NIRISS SOSS ATOCA only)              |
 +                                                       +--------------------------------------------------+
 |                                                       | SPECPROFILE (NIRISS SOSS ATOCA only)             |
@@ -258,6 +260,8 @@ documentation on each reference file.
 +--------------------------------------------------+-------------------------------------------------------+
 | :ref:`PHOTOM <photom_reffile>`                   | :ref:`photom <photom_step>`                           |
 +--------------------------------------------------+-------------------------------------------------------+
+| :ref:`PSF <psf_reffile>`                         | :ref:`extract_1d <extract_1d_step>`                   |
++--------------------------------------------------+-------------------------------------------------------+
 | :ref:`PSFMASK <psfmask_reffile>`                 | :ref:`align_refs <align_refs_step>`                   |
 +--------------------------------------------------+-------------------------------------------------------+
 | :ref:`READNOISE <readnoise_reffile>`             | :ref:`jump <jump_step>`                               |