Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates lanczos regridding and stacking #259

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Conversation

ljgray
Copy link
Contributor

@ljgray ljgray commented Jan 17, 2024

This PR implements two improvements for the lanczos regridder and adds new tasks to combine the regridding and stacking processes.

Lanczos updates

Signal variance estimate

Rather than using a single scalar as an estimate of the gridded signal variance, this PR adds the option to use an actual diagonal signal covariance derived from another dataset. Testing shows that using a proper covariance estimate improves interpolation around bright point sources while also reducing excess noise and artifacts introduced when using an excessively high variance on quieter parts of the sky. This PR:

  • Adds a simple task to estimate the diagonal covariance from a sidereal stream (this is just the squared, mean-subtracted power with a windowed mean applied)
  • Adds a container to hold a band diagonal covariance dataset in sidereal time
  • Adds an optional signal_cov argument in the setup method of transform.Regridder and, if provided, uses this container to make the inverse signal covariance for the interpolation
  • Cleans up the transform.Regridder and sidereal.SiderealRegridder code a bit

Artifact masking

Using a proper signal covariance seems to reduce/remove the bulk of regridding artifacts, but there are still some residual artifacts at the edge of heavily masked areas. From testing, this seems to occur when the main lobe of the regridding kernel is centred on flagged time samples. The estimate of the gridded sample comes only from the kernel side lobes, where small numerical errors end up being significant. This PR addresses this by projecting the data mask through the absolute value of the gridding kernel and masking samples below some fractional threshold.

Combined regridding and stacking

This is still somewhat experimental, and I have not yet tested whether the resulting stack is better using this method instead of just using the standard quarterstacking.
Having more time samples present in the Wiener filter regridder should in theory produce a better estimate of the underlying signal. Rather than gridding each individual day and taking a weighted average, as is done in the standard quarterstack pipeline, this PR adds tasks to split the gridding into steps which produce

  • A noise-weighted projection of the time series onto the RA grid
  • A band-diagonal noise covariance matrix
    The idea is to produce these two datasets for several days, sum them over all days, add an estimate of the gridded signal covariance, and then solve the deconvolution problem. This should have two advantages:
  • Fewer to no gaps in the data, reducing or eliminating gridding artifacts
  • A slightly better estimate of the underlying signal

In order to make this process computationally feasible, the weights (noise) dataset, and thereby the noise covariance, is factorized into a frequency-time array and a frequency-baseline array. This PR does the folllwing:

  • Adds containers for a factorized timestream and a partially gridded sidereal stream
  • Adds a task to factorize noise weights
  • Refactors the regrid.band_wiener code to separate the gridding process
  • Adds task to perform the "dirty" sidereal gridding, deconvolution, and stack-and-deconvolution processes
  • Adds a feature to core.io.Truncate to get the truncation weight dataset from a class property, allowing that dataset to be produced on the fly

@ljgray ljgray force-pushed the ljg/dirty-regrid branch 2 times, most recently from e7cb5a7 to 3a39c88 Compare May 1, 2024 21:31
@ljgray ljgray force-pushed the ljg/dirty-regrid branch 3 times, most recently from 1eae893 to 4bd01c2 Compare August 9, 2024 19:41
@ljgray ljgray changed the title Update regridder to better interface with stacking Combine regridding and stacking to reduce artifacts Aug 9, 2024
@ljgray ljgray force-pushed the ljg/dirty-regrid branch 17 times, most recently from c32ac57 to 20f7741 Compare August 16, 2024 20:36
@ljgray ljgray force-pushed the ljg/dirty-regrid branch 5 times, most recently from 47b055a to dc38569 Compare August 19, 2024 23:31
@ljgray ljgray force-pushed the ljg/dirty-regrid branch 2 times, most recently from 171912c to 17fe15e Compare September 19, 2024 19:24
@ljgray ljgray force-pushed the ljg/dirty-regrid branch 5 times, most recently from 5ebc372 to 75d2963 Compare October 9, 2024 21:15
@ljgray ljgray force-pushed the ljg/dirty-regrid branch 5 times, most recently from 44db836 to d791a9b Compare October 16, 2024 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant