-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updates lanczos regridding and stacking #259
Open
ljgray
wants to merge
12
commits into
master
Choose a base branch
from
ljg/dirty-regrid
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ljgray
force-pushed
the
ljg/dirty-regrid
branch
from
January 17, 2024 01:38
e6ef21f
to
c74f443
Compare
ljgray
force-pushed
the
ljg/dirty-regrid
branch
from
February 13, 2024 18:00
c74f443
to
c3404bb
Compare
ljgray
force-pushed
the
ljg/dirty-regrid
branch
2 times, most recently
from
May 1, 2024 21:31
e7cb5a7
to
3a39c88
Compare
ljgray
force-pushed
the
ljg/dirty-regrid
branch
3 times, most recently
from
August 9, 2024 19:41
1eae893
to
4bd01c2
Compare
ljgray
changed the title
Update regridder to better interface with stacking
Combine regridding and stacking to reduce artifacts
Aug 9, 2024
ljgray
force-pushed
the
ljg/dirty-regrid
branch
17 times, most recently
from
August 16, 2024 20:36
c32ac57
to
20f7741
Compare
ljgray
force-pushed
the
ljg/dirty-regrid
branch
5 times, most recently
from
August 19, 2024 23:31
47b055a
to
dc38569
Compare
ljgray
force-pushed
the
ljg/dirty-regrid
branch
2 times, most recently
from
September 19, 2024 19:24
171912c
to
17fe15e
Compare
ljgray
force-pushed
the
ljg/dirty-regrid
branch
from
September 27, 2024 22:39
17fe15e
to
78721e8
Compare
ljgray
force-pushed
the
ljg/dirty-regrid
branch
5 times, most recently
from
October 9, 2024 21:15
5ebc372
to
75d2963
Compare
ljgray
force-pushed
the
ljg/dirty-regrid
branch
5 times, most recently
from
October 16, 2024 21:56
44db836
to
d791a9b
Compare
ljgray
force-pushed
the
ljg/dirty-regrid
branch
from
October 23, 2024 23:56
d791a9b
to
9bbf60c
Compare
Also, add a function to estimate a signal covariance from a timestream
ljgray
force-pushed
the
ljg/dirty-regrid
branch
from
November 13, 2024 00:42
9bbf60c
to
eb114b0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements two improvements for the lanczos regridder and adds new tasks to combine the regridding and stacking processes.
Lanczos updates
Signal variance estimate
Rather than using a single scalar as an estimate of the gridded signal variance, this PR adds the option to use an actual diagonal signal covariance derived from another dataset. Testing shows that using a proper covariance estimate improves interpolation around bright point sources while also reducing excess noise and artifacts introduced when using an excessively high variance on quieter parts of the sky. This PR:
signal_cov
argument in thesetup
method oftransform.Regridder
and, if provided, uses this container to make the inverse signal covariance for the interpolationtransform.Regridder
andsidereal.SiderealRegridder
code a bitArtifact masking
Using a proper signal covariance seems to reduce/remove the bulk of regridding artifacts, but there are still some residual artifacts at the edge of heavily masked areas. From testing, this seems to occur when the main lobe of the regridding kernel is centred on flagged time samples. The estimate of the gridded sample comes only from the kernel side lobes, where small numerical errors end up being significant. This PR addresses this by projecting the data mask through the absolute value of the gridding kernel and masking samples below some fractional threshold.
Combined regridding and stacking
This is still somewhat experimental, and I have not yet tested whether the resulting stack is better using this method instead of just using the standard quarterstacking.
Having more time samples present in the Wiener filter regridder should in theory produce a better estimate of the underlying signal. Rather than gridding each individual day and taking a weighted average, as is done in the standard quarterstack pipeline, this PR adds tasks to split the gridding into steps which produce
The idea is to produce these two datasets for several days, sum them over all days, add an estimate of the gridded signal covariance, and then solve the deconvolution problem. This should have two advantages:
In order to make this process computationally feasible, the weights (noise) dataset, and thereby the noise covariance, is factorized into a frequency-time array and a frequency-baseline array. This PR does the folllwing:
regrid.band_wiener
code to separate the gridding processcore.io.Truncate
to get the truncation weight dataset from a class property, allowing that dataset to be produced on the fly