2022/08/11 Meeting notes #8
Comments
|
I have a doubt about the goal of this project. The goal is to develop a pipeline to image the sun using any data from MeerKAT right? But my doubt is why we want to imagine the sun. Initially I understood that we wanted to image the Sun to characterise the level of solar contamination and then somehow eliminate it (the Sun) from the observations, but at the last meeting I understood that the objective is actually for the pipeline to be able to make images of the Sun for any MeerKAT data. |
|
It is both.
|
|
Okay. That means that once to image the Sun we have to change the phase centre of the ms to point to the Sun and subtract the model-data from corrected-data we can then subtract the Sun and go back to the previous phase centre (eg. where a specific target we interested in are) but now without the Sun there because we have already subtracted it? |
|
Further to what Oleg said, we are still not sure what observing conditions lead to the most severe solar contamination of the data. Is there a particular separation angle between the telescope's pointing direction and the Sun for which solar interference is particularly bad? Can the Sun enter the data from behind the main dish, e.g. by direct illumination of the secondary reflector? Are sunset and sunrise particularly bad times to do observations when the sun is low on the horizon? You will be the first person to conduct a systematic investigation of these effects for MeerKAT. If you can devise a scheme to quantify the severity of the solar interference for a large range of observational conditions then we can use that information when scheduling future observations on the telescope. I think the best way to approach these questions is to first gather lots of images of the Sun. Your work can also be fed into the scheduling plans for SKA-MID, which will have a very similar design to MeerKAT, albeit in a much larger scale. There is a direct community benefit to making the observing programs for these instruments more efficient, both in terms of minimising data loss and therefore cost. |
I think in the workflow we have been assuming so far, we subtract the
I principle I think this is correct. If you can deconvolve the Sun then a model can be constructed and subtracted from the visibilities. How well the deconvolution process will work remains to be seen, as the Sun will be smeared in time even across a single scan. There may also be some bookkeeping involved, for example we might want to create a custom Lots to explore here techniques-wise, and all of it is interesting (I hope you agree!) and potentially useful to the community. But I think the first thing to do is to implement your routine solar imaging pipeline. |
|
Thank you very much, now I am clearer about what I really hope to do with
the results of the Pipeline.I imagined many things but it was not very
clear to me yet. And yes, certainly it's all very interesting and I'm very
happy to be a part of it. Thanks for your explanation.
…On Mon, Aug 15, 2022 at 3:08 PM IanHeywood ***@***.***> wrote:
we have to change the phase centre of the ms to point to the Sun and
subtract the model-data from corrected-data
I think in the workflow we have been assuming so far, we subtract the
MODEL_DATA from CORRECTED_DATA prior to changing the phase centre to
point at the Sun. The assumption is that the model visibilities will
correspond to the emission in the original target field, following the 2GC
(self-calibration) stage of the 'regular' processing pipeline.
we can then subtract the Sun and go back to the previous phase centre (eg.
where a specific target we interested in are) but now without the Sun there
because we have already subtracted it?
I principle I think this is correct. If you can deconvolve the Sun then a
model can be constructed and subtracted from the visibilities. How well the
deconvolution process will work remains to be seen, as the Sun will be
smeared in time even across a single scan. There may also be some
bookkeeping involved, for example we might want to create a custom
SOLAR_DATA model column in the MS to preserve the original model...
Lots to explore here techniques-wise, and all of it is interesting (I hope
you agree!) and potentially useful to the community. But I think the first
thing to do is to implement your routine solar imaging pipeline.
—
Reply to this email directly, view it on GitHub
<#8 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AWYM2BO3KWLE6NUKLRAPWFTVZI6OLANCNFSM56JLD6PA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
You should of course always feel free to bring your own ideas to the table! |
|
It's okay. Thanks
…On Mon, Aug 15, 2022 at 3:54 PM IanHeywood ***@***.***> wrote:
I imagined many things
You should of course always feel free to bring your own ideas to the table!
—
Reply to this email directly, view it on GitHub
<#8 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AWYM2BLGESKV5YQXU6TVL4LVZJDXTANCNFSM56JLD6PA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Greetings Iam doing the calibration step and after 1GC I did 2GC (selfcal) step using this script : selfcal:
enable: true
ncpu: 10
img_npix: 6000
img_cell: 1.5
img_niter: 1000000
img_nchans: 5
img_robust: 0
img_taper: '0'
img_nwlayers_factor: 3
cal_niter: 5
cal_timeslots_chunk: -1
start_iter: 1
img_specfit_nrcoeff: 2
img_multiscale: true
img_multiscale_scales: ''
img_nrdeconvsubimg: 1024
image:
enable: true
cleanmask_thr: [30,20,15,5]
clean_cutoff: [0.5,0.5,0.5,0.5]
col: [DATA,CORRECTED_DATA,CORRECTED_DATA,CORRECTED_DATA]
calibrate:
enable: true
model: ['1','2','3']
gain_matrix_type: ['Fslope', 'GainDiagPhase', 'GainDiag']
gsols_chan: [0,0,0]
gsols_timeslots: [50,50,50]
transfer_apply_gains:
enable: false
transfer_model:
enable: false
report: trueand I got this images: analysing I think the image are not good so I thing I cant continue with the folowing proceeds. But now I dont now exacty what I have to do to have a better image. I thoutg I have to run again selfcal in the target data solutions but iam having this error: So iam sure of what/how to do. |
|
I see 12 images, what exactly am I looking at? At the colour scale you're showing, they look superficially OK, why do you think they're not so good? But please adjust the colour scale so that we see more of the background noise/artefacts, it will be easier to judge quality then. |
|
I don't really know CARACal so @o-smirnov and @Kincaidr might be better placed to help, but: I don't think the external calibrators should be involved for self-calibration operation, and indeed I'm guessing you are working from a MS that only contains target visibilities at this point ( The maps look good, so my best guess would be this: perhaps it's trying to report on something related to the gaincal at the very end, but the gaincal is missing from the MS. Perhaps try setting As a general point, I think five (or four?) cycles of selfcal is unneccessary, and will only serve to increase the amount of time this takes, for no benefit. with this setting I think any apparent improvements between selfcal cycles will simply be due to deeper deconvolution, which you can readily do immediately. MeerKAT is (in my experience) stable enough and with a good enough PSF to just do a single cycle of selfcal, after which the map is DDE-limited. I think you can probably save a lot of time (and electricity) by changing this. Other things I'm not sure about: Is It looks like the final (?) selfcal iteration is amplitude and phase (GainDiag): Again, others please comment. Cheers. |
My familiarity with the field gives me an advantage here. :) @Victoria-Samboco are these the full band (MFS) images? |
|
@Victoria-Samboco, for future reference: when you post YaML into an issue, put triple back-quotes with "yml" at the start like so: ...and finish with a line of triple-backquotes. Your YaML will then render with nice syntax highlighting, and will preserve correct spacing. If you simply paste YaML into the comment box like you did initially, github's auto-formatting strips the leading spaces and makes it very hard to read the code. I have already edited your comment to fix this. (Likewise, you can render Python code nicely by using triple-backquote-python).
Correct. So I would ignore that error. Is just for a final HTML report. I don't use this, the radiopadre notebooks are more informative anyway. (The built-in reports pre-dated radiopadre and have far fewer features).
Agreed.
No, this enables wsclean's
Yep, @IanHeywood's interpretation is correct, and I agree, |
|
Actually, there are not 12 images, there are 6. But the second 6 are full field and the first 6 are the same images with zoom in. The images looks like this:
The first is from de DATA colum and the other ones are from CORRECTED_DATA for different solvers The last 2 images here I think are the same think, I think maybe is repeated because the nummer of calibration interaction was 5 ''' cal_niter: 5 ''' |
|
@IanHeywood yes these is a full band images . So that's mean I dont need to use |
Yes, but the error has nothing to do with the selfcal step. @Victoria-Samboco re-ran the pipeline after changing the input ms file from the raw ms to the ms containing only targets (corr-ms). When running the pipeline from selfcal (or any worker) caracal automatically runs obsconfig worker at the start as a pre-validation step. So in this case it could not find the calibrator information. So the reason why there are 6 images is because |
Yep, so the error should just be ignored (maybe post a note to the Caracal repo to adjust the error wording, to make it clear that this is only a problem if the crosscal worker is being run).
I can neither confirm nor deny, I didn't write the Crosscal worker. :) Best ask Kshitij. Or just read the logs to see what it actually did? |
Yes so the main changes are:
For the rest of the inputs, just make sure the arrays have consistent lengths, so under |
|
Hi Prof Oleg and Ian Bellow I have images from the model after Selfcal (with auto-mask 10) and the other is the model after applying deep mask with breizorro (mask 5). I would like to know why we have this black dots/regions (negative regions) on the sources.
and below are the images corresponding to these models after Selfcal (with mask 10) and after applying deep mask (5). |
|
Even though the genuine emission in a Stokes I image is always positive, the clean components can be both positive and negative. Some negative components are generally fine, for example a mixture of (mostly) positive and (some) negative components may be required to faithfully characterise an extended source. The issue above is that both the automasking and breizorro have interpreted the strong radial artefacts around the brightest source in the image as genuine emission. Clean has then deconvolved these regions, which will likely include negative values that are significantly lower than those that result from pure noise. I stopped using automasking as I never found a set of parameters that struck a good balance between completeness, avoiding artefacts, and also being applicable to a wide range of fields. For my own processing I "waste" an imaging cycle with unconstrained deconvolution and then use something breizorro-like to make a mask. The breizorro mask, having a lower threshold than the automasking, will be including a lot more genuine sources, which is why that mask looks a lot busier. However I am surprised by how many artefacts breizorro is including. Can you please post a link to the (1GC?) image that you are generating the mask from? Something strange also appears in the lower left image, where the bright source in the lower right seems to have a new source appearing next to it, but I can't quite see what is going on with the images at this resolution. |
|
For the 1GC image the link is /net/garfunkel/home/samboco/solarkat/1GC_UHF/output/continuum/image_2/ for the deep mask image is /net/garfunkel/home/samboco/solarkat/1GC_UHF/output/continuum/deep_masking_image |
I suspect this is actually the 2GC image (wsclean is imaging the If so, then my best explanation for the new source in the left hand image above is that there is a strong negative feature in the model which is being reinforced by self-calibration and appearing in the corrected data for the subsequent imaging cycle. You can see it in the image and profiles here:
The deep mask image also has this spurious negative feature (and its associated artefacts), which is probably why the artefacts are much more pronounced in the breizorro map. For mask masking with breizorro I would return to the 1GC image, iteratively raising the threshold until none of the artefacts are present, and then re-image the |
|
Oh yes sorry. The 1GC image is here |
|
Some general comments on the wsclean command that was used:
If these are caracal defaults then you might want to confer with the dev team about them. |
|
A further thought occurs. The strong, spurious negative source does not appear in your lower right image which if I understand correctly represents data that were self-calibrated, with the model being formed from the breizorro mask. This suggests that while breizorro enforces positivity when making masks perhaps wslcean's internal automasking does not. The former behaviour seems a lot safer than the latter when making a Stokes I image. |
Hi everyone
From today's meeting we discussed about what are the next steps once we already succeeded to image the Sun.
Next steps:
From UHF-band proceed with:
3.1. Split_ms
3.2. Ra/Dec
3.3. Apply chgcentre command
3.4. run wsclean with -subtract-model parameter
The text was updated successfully, but these errors were encountered: