Result of Peeling Sun with Quartical

[Victoria-Samboco]

The left figure represents the Subtracted Sun image from the pipeline, and the right image is the peeled image with Quartical. I imaged the two images in the same conditions; I would like to understand why the resultant peeled image comes with noise and requires additional cleaning.

[o-smirnov]

What were your solution intervals? Can you add the QuartiCal log?

[Victoria-Samboco]

What were your solution intervals? Can you add the QuartiCal log?

2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO: ### validated inputs
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO: cab goquartical:
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_ms.path = msdir/1671435077_sdp_l0_1024ch_GRS1747-312.ms
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_ms.data_column = DATA
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_ms.weight_column = WEIGHT_SPECTRUM
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_ms.time_chunk = 14
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_ms.freq_chunk = 0
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_ms.is_bda = False
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_ms.group_by = ['FIELD_ID', 'DATA_DESC_ID', 'SCAN_NUMBER']
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_ms.select_fields = []
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_ms.select_ddids = []
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_ms.select_uv_range = [150.0, 0.0]
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_model.recipe = MODEL_DATA:MODEL_DATA_SUN
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_model.beam_l_axis = X
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_model.beam_m_axis = Y
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_model.invert_uvw = True
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_model.source_chunks = 500
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   input_model.apply_p_jones = False
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   output.gain_directory = obs2/peeled.qc/peeled.qc
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   output.log_directory = obs2/peeled.qc/
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   output.log_to_terminal = True
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   output.overwrite = True
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   output.products = ['corrected_residual']
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   output.columns = ['CORRECTED_RESIDUAL']
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   output.flags = True
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   output.apply_p_jones_inv = False
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   output.subtract_directions = [1]
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   output.compute_baseline_corrections = False
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   output.apply_baseline_corrections = False
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   mad_flags.enable = False
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   mad_flags.whitening = disabled
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   mad_flags.threshold_bl = 5.0
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   mad_flags.threshold_global = 5.0
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   mad_flags.max_deviation = 5.0
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   solver.terms = ['G', 'dE']
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   solver.iter_recipe = [50, 50, 25, 25]
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   solver.propagate_flags = False
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   solver.robust = False
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   solver.threads = 4
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   solver.convergence_fraction = 0.99
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   solver.convergence_criteria = 1e-06
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   solver.reference_antenna = 0
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dask.threads = 8
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dask.workers = 1
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dask.scheduler = threads
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   G.type = delay
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   G.solve_per = antenna
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   G.direction_dependent = False
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   G.pinned_directions = [0]
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   G.time_interval = 7
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   G.freq_interval = 0
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   G.interp_mode = reim
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   G.interp_method = 2dlinear
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   G.respect_scan_boundaries = True
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   G.initial_estimate = False
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dE.type = complex
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dE.solve_per = antenna
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dE.direction_dependent = True
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dE.pinned_directions = [0]
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dE.time_interval = 14
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dE.freq_interval = 256
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dE.interp_mode = reim
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dE.interp_method = 2dlinear
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dE.respect_scan_boundaries = True
2023-05-25 11:18:16 STIMELA.boom.cal_and_peel_sol INFO:   dE.initial_estimate = False
### running /home/samboco/Virtual_enviroments/stimela_env/bin/goquartical input_ms.path=msdir/1671435077_sdp_l0_1024ch_GRS1747-312.ms input_ms.data_column=DATA input_ms.weight_column=WEIGHT_SPECTRUM input_ms.time_chunk=14 input_ms.freq_chunk=0 input_ms.is_bda=False input_ms.group_by=[FIELD_ID,DATA_DESC_ID,SCAN_NUMBER] input_ms.select_fields=[] input_ms.select_ddids=[] input_ms.select_uv_range=[150.0,0.0] input_model.recipe=MODEL_DATA:MODEL_DATA_SUN input_model.beam_l_axis=X input_model.beam_m_axis=Y input_model.invert_uvw=True input_model.source_chunks=500 input_model.apply_p_jones=False output.gain_directory=obs2/peeled.qc/peeled.qc output.log_directory=obs2/peeled.qc/ output.log_to_terminal=True output.overwrite=True output.products=[corrected_residual] output.columns=[CORRECTED_RESIDUAL] output.flags=True output.apply_p_jones_inv=False output.subtract_directions=[1] output.compute_baseline_corrections=False output.apply_baseline_corrections=False mad_flags.enable=False mad_flags.whitening=disabled mad_flags.threshold_bl=5.0 mad_flags.threshold_global=5.0 mad_flags.max_deviation=5.0 solver.terms=[G,dE] solver.iter_recipe=[50,50,25,25] solver.propagate_flags=False solver.robust=False solver.threads=4 solver.convergence_fraction=0.99 solver.convergence_criteria=1e-06 solver.reference_antenna=0 dask.threads=8 dask.workers=1 dask.scheduler=threads G.type=delay G.solve_per=antenna G.direction_dependent=False G.pinned_directions=[0] G.time_interval=7 G.freq_interval=0 G.interp_mode=reim G.interp_method=2dlinear G.respect_scan_boundaries=True G.initial_estimate=False dE.type=complex dE.solve_per=antenna dE.direction_dependent=True dE.pinned_directions=[0] dE.time_interval=14 dE.freq_interval=256 dE.interp_mode=reim dE.interp_method=2dlinear dE.respect_scan_boundaries=True dE.initial_estimate=False
# 2023-05-25 11:18:47 | INFO | parser:log_final_config | Final configuration state:
# 
# ------------------------------------input_ms------------------------------------
# input_ms.path                      msdir/1671435077_sdp_l0_1024ch_GRS1747-312.ms
# input_ms.data_column                                                        DATA
# input_ms.sigma_column                                                       None
# input_ms.weight_column                                           WEIGHT_SPECTRUM
# input_ms.time_chunk                                                           14
# input_ms.freq_chunk                                                            0
# input_ms.is_bda                                                            False
# input_ms.group_by                    ['FIELD_ID', 'DATA_DESC_ID', 'SCAN_NUMBER']
# input_ms.select_corr                                                        None
# input_ms.select_fields                                                        []
# input_ms.select_ddids                                                         []
# input_ms.select_uv_range                                            [150.0, 0.0]
# --------------------------------------------------------------------------------
# 
# ----------------------------------input_model-----------------------------------
# input_model.recipe                                     MODEL_DATA:MODEL_DATA_SUN
# input_model.beam                                                            None
# input_model.beam_l_axis                                                        X
# input_model.beam_m_axis                                                        Y
# input_model.invert_uvw                                                      True
# input_model.source_chunks                                                    500
# input_model.apply_p_jones                                                  False
# --------------------------------------------------------------------------------
# 
# -------------------------------------output-------------------------------------
# output.gain_directory                                   obs2/peeled.qc/peeled.qc
# output.log_directory                                             obs2/peeled.qc/
# output.log_to_terminal                                                      True
# output.overwrite                                                            True
# output.products                                           ['corrected_residual']
# output.columns                                            ['CORRECTED_RESIDUAL']
# output.flags                                                                True
# output.apply_p_jones_inv                                                   False
# output.subtract_directions                                                   [1]
# output.net_gains                                                            None
# output.compute_baseline_corrections                                        False
# output.apply_baseline_corrections                                          False
# --------------------------------------------------------------------------------
# 
# -----------------------------------mad_flags------------------------------------
# mad_flags.enable                                                           False
# mad_flags.whitening                                                     disabled
# mad_flags.threshold_bl                                                       5.0
# mad_flags.threshold_global                                                   5.0
# mad_flags.max_deviation                                                      5.0
# --------------------------------------------------------------------------------
# 
# -------------------------------------solver-------------------------------------
# solver.terms                                                         ['G', 'dE']
# solver.iter_recipe                                              [50, 50, 25, 25]
# solver.propagate_flags                                                     False
# solver.robust                                                              False
# solver.threads                                                                 4
# solver.convergence_fraction                                                 0.99
# solver.convergence_criteria                                                1e-06
# solver.reference_antenna                                                       0
# --------------------------------------------------------------------------------
# 
# --------------------------------------dask--------------------------------------
# dask.threads                                                                   8
# dask.workers                                                                   1
# dask.address                                                                None
# dask.scheduler                                                           threads
# --------------------------------------------------------------------------------
# 
# ---------------------------------------G----------------------------------------
# G.type                                                                     delay
# G.solve_per                                                              antenna
# G.direction_dependent                                                      False
# G.pinned_directions                                                          [0]
# G.time_interval                                                                7
# G.freq_interval                                                                0
# G.load_from                                                                 None
# G.interp_mode                                                               reim
# G.interp_method                                                         2dlinear
# G.respect_scan_boundaries                                                   True
# G.initial_estimate                                                         False
# --------------------------------------------------------------------------------
# 
# ---------------------------------------dE---------------------------------------
# dE.type                                                                  complex
# dE.solve_per                                                             antenna
# dE.direction_dependent                                                      True
# dE.pinned_directions                                                         [0]
# dE.time_interval                                                              14
# dE.freq_interval                                                             256
# dE.load_from                                                                None
# dE.interp_mode                                                              reim
# dE.interp_method                                                        2dlinear
# dE.respect_scan_boundaries                                                  True
# dE.initial_estimate                                                        False
# --------------------------------------------------------------------------------
# 
# 
# 2023-05-25 11:18:47 | INFO | preprocess:transcribe_recipe | The following model sources were obtained from --input-model-recipe:
#    Columns: {'MODEL_DATA', 'MODEL_DATA_SUN'}
#    Sky Models: None
# 2023-05-25 11:18:47 | INFO | ms_handler:read_xds_list | Antenna table indicates 62 antennas were present for this observation.
# 2023-05-25 11:18:47 | INFO | ms_handler:read_xds_list | Polarization table indicates 4 correlations are present in the measurement set - ['XX', 'XY', 'YX', 'YY'].
# 2023-05-25 11:18:47 | INFO | ms_handler:read_xds_list | Field table indicates phase centre is at (4.6721671405519585 -0.5457751229996112).
# 2023-05-25 11:18:48 | INFO | ms_handler:write_xds_list | Outputs will be written to FLAG, FLAG_ROW, CORRECTED_RESIDUAL.
# 2023-05-25 11:18:53 | SUCCESS | executor:_execute | 6.01 seconds taken to build graph.
# 2023-05-25 11:18:53 | INFO | executor:_execute | Computation starting. Please be patient - log messages will only appear per completed chunk.
# 2023-05-25 11:26:10 | INFO | logging:log_chisq | 
#     FLD: 0 SPW: 0 SCN: 12 T_CHUNK: 5 F_CHUNK: 0 CHISQ: 1.86 -> 1.81
# 2023-05-25 11:26:11 | INFO | logging:log_chisq | 
#     FLD: 0 SPW: 0 SCN: 12 T_CHUNK: 0 F_CHUNK: 0 CHISQ: 1.84 -> 1.78
# 2023-05-25 11:26:12 | INFO | logging:log_chisq | 
#     FLD: 0 SPW: 0 SCN: 12 T_CHUNK: 7 F_CHUNK: 0 CHISQ: 1.86 -> 1.80
# 2023-05-25 11:26:13 | INFO | logging:log_chisq | 
#     FLD: 0 SPW: 0 SCN: 12 T_CHUNK: 1 F_CHUNK: 0 CHISQ: 1.85 -> 1.83
# 2023-05-25 11:26:14 | INFO | logging:log_chisq | 
#     FLD: 0 SPW: 0 SCN: 12 T_CHUNK: 3 F_CHUNK: 0 CHISQ: 1.86 -> 1.84
# 2023-05-25 11:26:15 | INFO | logging:log_chisq | 
#     FLD: 0 SPW: 0 SCN: 12 T_CHUNK: 2 F_CHUNK: 0 CHISQ: 1.85 -> 1.84
# 2023-05-25 11:26:26 | INFO | logging:log_chisq | 
#     FLD: 0 SPW: 0 SCN: 12 T_CHUNK: 6 F_CHUNK: 0 CHISQ: 1.86 -> 1.82
# 2023-05-25 11:26:27 | INFO | logging:log_chisq | 
#     FLD: 0 SPW: 0 SCN: 12 T_CHUNK: 4 F_CHUNK: 0 CHISQ: 1.85 -> 1.82
# 2023-05-25 11:26:35 | SUCCESS | executor:_execute | 461.45 seconds taken to execute graph.
# 2023-05-25 11:26:35 | INFO | logging:log_summary_stats | 
# Final post-solve chi-squared summary, colourised by deviation from the mean:
#  <= 3*sigma   <= 5*sigma   <= 10*sigma   > 10*sigma 
#   CHUNK  FLD0
#          SPW0
#          SCN12
# 
#   T0F0   1.78
#   T1F0   1.83
#   T2F0   1.84
#   T3F0   1.84
#   T4F0   1.82
#   T5F0   1.81
#   T6F0   1.82
#   T7F0   1.80
# 
2023-05-25 11:26:35 STIMELA.boom.cal_and_peel_sol INFO: goquartical returns exit code 0 after 0:08:19