Question About Warning

[AKHughes1994]

Hi,

I've received this warning (see image) when self-calibrating a particular data set. It doesn't do anything since I still see dramatic improvements in the image fidelity post-self-cal. I'm unsure what/how to interpret this error, and I would like to know if you could provide any insight. When I open the ms file and plot the dimension of the MODEL_DATA column, it most certainly has defined dimensions.

This is a vague question, so I apologize. Let me know if you need any information, i.e., msfiles. I will note that I have averaged this ms file from 0.15s to 6s timing. So I wonder if this has something to do with the averaging issue, as I haven't seen the issue in non-averaged data sets.

[JSKenyon]

Hi @AKHughes1994! As you mention, this should be a harmless warning. As far as I know, this comes directly from dask-ms and has something to do with variably shaped/partially initialized columns. If you are interested in verifying that that is the case, I would recommend opening the MS using python-casacoreand calling getdminfo and getcoldesc on the MODEL_DATA column. You can post the results of those calls here, at which point I can probably commit to a more definite answer.

[AKHughes1994]

Hi Sorry For the delayed reply,

Forgive me if this is not the input you are looking for, I used tb.getdminfo() in casa and the output with the (i think we are looking for is):

{'COLUMNS': array(['FLAG_ROW', 'MODEL_DATA'], dtype='<U16'),
 'NAME': 'FLAG_ROW',
 'SEQNR': 6,
 'SPEC': {'BUCKETSIZE': 8192,
  'IndexLength': 53452,
  'MaxCacheSize': 2,
  'PERSCACHESIZE': 2},
 'TYPE': 'StandardStMan'}

Similarly, for tb.getcoldesc(),

{'comment': '',
 'dataManagerGroup': 'FLAG_ROW',
 'dataManagerType': 'StandardStMan',
 'keywords': {},
 'maxlen': 0,
 **'ndim': -1**,
 'option': 0,
 'valueType': 'complex'}

More information, it doesn't seem to be dependant on a single ms file (I see the same message for other ms files). I am populating the model column with WSCLEAN v3.3, whereas I don't see the same issue in previouse container versions that use WSCLEAN v3.2, so maybe that ndim: -1 is something to do with the version change. I'll check that out and update the discussion.

[AKHughes1994]

I checked its not the WSCLEAN version. I have no idea why it's the model data column is has those ndim terms now (whereas other ms files do not). I'll keep looking.

[JSKenyon]

Are you doing some sort of channel selection in wsclean which you weren't doing previously? In other words, is wsclean adding the model column, but only at some locations? That would be my first thought. The likely culprit will be whatever software/step actually adds the MODEL_DATA column.

[AKHughes1994]

I don't know what is different, the imaging call should be exactly the same. The major difference is that I have two containers with slightly different versions for... everything. One container doesn't show the error, and another container does. I'll play around with it over the next couple of weeks, and when I find out what the difference is, I'll update the discussion. Given that the error doesn't seem to do anything, though, it's not urgent!

Thanks for the advice so far.