Error in find_10thetas() in pyOWC

[matdever]

I have been trying to use the pyOWC package, which used to work for me, but I now encounter the same consistent error that prevents me from completing the analysis. I tried to troubleshoot by making a new environment, re-install the packages anew, no success. The error persists over several floats and several datasources. Is it an error with my setup?

The error relates to the find_10thetas() function within owc.calibration.calc_piecewisefit(), and seem to relate to the dimensions of the arrays. Here is the full error:

---

ValueError Traceback (most recent call last)
Cell In[3], line 75
72 owc.configuration.set_calseries("", FLOAT_NAME, USER_CONFIG)
74 # Calculate the fit of each break and calibrate salinities
---> 75 owc.calibration.calc_piecewisefit("", FLOAT_NAME, USER_CONFIG)
77 # Results figures
78 owc.plot.dashboard("", FLOAT_NAME, USER_CONFIG, levels=10)

File /opt/anaconda3/envs/OWC_analysis/lib/python3.10/site-packages/pyowc/calibration.py:590, in calc_piecewisefit(float_dir, float_name, system_config)
587 unique_mapped_sal_1 = copy.deepcopy(unique_mapped_sal)
588 unique_mapsalerrors_1 = copy.deepcopy(unique_mapsalerrors)
--> 590 theta, p, index, var_s_th, th = find_10thetas(copy.deepcopy(unique_sal),
591 copy.deepcopy(unique_ptmp),
592 copy.deepcopy(unique_pres),
593 copy.deepcopy(unique_mapped_ptmp),
594 use_theta_lt, use_theta_gt,
595 use_pres_lt, use_pres_gt,
596 use_percent_gt)
598 index = np.array(index, dtype=int)
599 pp = np.argwhere(np.isnan(index) == 0)

File /opt/anaconda3/envs/OWC_analysis/lib/python3.10/site-packages/pyowc/core/finders.py:665, in find_10thetas(sal, ptmp, pres, la_ptmp, use_theta_lt, use_theta_gt, use_pres_lt, use_pres_gt, use_percent_gt)
661 # interpolate theta level, if possible
662 if (k_index != theta_index and ~np.isnan(sal[theta_index, depth]) and
663 ~np.isnan(sal[k_index, depth]) and ~np.isnan(ptmp[theta_index, depth]) and
664 ~np.isnan(ptmp[k_index, depth])):
--> 665 interp_ptmp_sal = interpolate.interp1d([ptmp[theta_index, depth],
666 ptmp[k_index, depth]],
667 [sal[theta_index, depth],
668 sal[k_index, depth]])
670 sal_temp[level, depth] = interp_ptmp_sal(theta_levels[level])
672 # else we use the closest points
673 else:

File /opt/anaconda3/envs/OWC_analysis/lib/python3.10/site-packages/scipy/interpolate/_interpolate.py:500, in interp1d.init(self, x, y, kind, axis, copy, bounds_error, fill_value, assume_sorted)
496 def init(self, x, y, kind='linear', axis=-1,
497 copy=True, bounds_error=None, fill_value=np.nan,
498 assume_sorted=False):
499 """ Initialize a 1-D linear interpolation class."""
--> 500 _Interpolator1D.init(self, x, y, axis=axis)
502 self.bounds_error = bounds_error # used by fill_value setter
504 # copy keyword semantics changed in NumPy 2.0, once that is
505 # the minimum version this can use copy=None.

File /opt/anaconda3/envs/OWC_analysis/lib/python3.10/site-packages/scipy/interpolate/_polyint.py:57, in _Interpolator1D.init(self, xi, yi, axis)
55 self.dtype = None
56 if yi is not None:
---> 57 self._set_yi(yi, xi=xi, axis=axis)

File /opt/anaconda3/envs/OWC_analysis/lib/python3.10/site-packages/scipy/interpolate/_polyint.py:121, in _Interpolator1D._set_yi(self, yi, xi, axis)
118 if axis is None:
119 raise ValueError("no interpolation axis specified")
--> 121 yi = np.asarray(yi)
123 shape = yi.shape
124 if shape == ():

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

[kamwal]

Hi Matt, could you provide the list of float WMO, datasets and configurations setups you used to run the code?

gmaze Do you think this could be something with the packages or more like a bug in the code?

[gmaze]

very hard to tell, we did not pinned dependencies versions

but it is very likely that it is related to the environnement, otherwise, the error would have pop up before

1st thing is to try to reproduce the error

@matdever beyond the OWC setup data asked by @kamwal , would be good to have your environment versions

(you can have this easily: install argopy in the same environment you're having the pyowc error, and then give us the output of the argopy.show_versions() command)

[thogar-computer]

I can also take a look from the code side. I would agree that this is likely a non-pinned dependencies issues. thought it would be useful to know from @matdever if you still have your old environment? as a compared between the two would be really useful.

You also mention that it did work. Can you remember the last time it worked? Again, that would help to see if a new version of a package came out around the same time.

Cheers
Tom

[kamwal]

I run few example floats which were working some time ago(I didn't run this code since a while): ["3902398"], ["1901914"], ["3901960"] using the CTD_for_DMQC_2024V01 and ARGO_for_DMQC_2023V03 reference data.
My Python version is 3.10.9
env.txt
config.txt

I am not sure why it is like that. There have not been made any changes to this code from a long time

I have found the same error message:

Using precalculated data: L:\users\argo\kamwal\OWC_Python_Test_2\argodmqc_owc\data\float_mapped\map_6901933.mat

---

TOTAL TIME ELAPSED: 27.102484703063965
Using parameters found in data/float_calib/\calseries_6901933.mat
To use new parameters, delete this file
Traceback (most recent call last):
File "L:\users\argo\kamwal\OWC_Python_Test_2\argodmqc_owc\start_with_pycharm.py", line 32, in
owc.calibration.calc_piecewisefit("/", flt, USER_CONFIG)
File "L:\users\argo\kamwal\OWC_Python_Test_2\argodmqc_owc\pyowc\calibration.py", line 590, in calc_piecewisefit
theta, p, index, var_s_th, th = find_10thetas(copy.deepcopy(unique_sal),
File "L:\users\argo\kamwal\OWC_Python_Test_2\argodmqc_owc\pyowc\core\finders.py", line 665, in find_10thetas
interp_ptmp_sal = interpolate.interp1d([ptmp[theta_index, depth],
File "C:\Users\kamwal\Programs\Anaconda3\envs\dmqc_pcm\lib\site-packages\scipy\interpolate\interpolate.py", line 436, in init
_Interpolator1D.init(self, x, y, axis=axis)
File "C:\Users\kamwal\Programs\Anaconda3\envs\dmqc_pcm\lib\site-packages\scipy\interpolate\polyint.py", line 54, in init
self._set_yi(yi, xi=xi, axis=axis)
File "C:\Users\kamwal\Programs\Anaconda3\envs\dmqc_pcm\lib\site-packages\scipy\interpolate\polyint.py", line 118, in _set_yi
yi = np.asarray(yi)
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

Process finished with exit code 1

[matdever]

thank you so much for the feedback. Here are the requested info:

My env versions:

SYSTEM
------
commit: None
python: 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:54:21) [Clang 16.0.6 ]
python-bits: 64
OS: Darwin
OS-release: 23.4.0
machine: arm64
processor: arm
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_CA.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.14.3
libnetcdf: 4.9.2

INSTALLED VERSIONS: CORE
------------------------
aiohttp     : 3.9.3       
argopy      : 0.1.15      
erddapy     : 2.2.0       
fsspec      : 2023.10.0   
netCDF4     : 1.6.5       
packaging   : 23.2        
requests    : 2.31.0      
scipy       : 1.11.3      
toolz       : 0.12.0      
xarray      : 2023.6.0    

INSTALLED VERSIONS: EXT.UTIL
----------------------------
gsw         : 3.6.18      
tqdm        : -           
zarr        : -           

INSTALLED VERSIONS: EXT.PERF
----------------------------
dask        : -           
distributed : -           
h5netcdf    : -           
pyarrow     : -           

INSTALLED VERSIONS: EXT.PLOT
----------------------------
IPython     : 8.20.0      
cartopy     : -           
ipykernel   : 6.28.0      
ipywidgets  : -           
matplotlib  : 3.8.4       
pyproj      : -           
seaborn     : -           

INSTALLED VERSIONS: DEV
-----------------------
aiofiles    : -           
black       : 24.3.0      
bottleneck  : 1.3.7       
cfgrib      : -           
cftime      : 1.6.2       
conda       : -           
flake8      : 7.0.0       
nc_time_axis: -           
numpy       : 1.26.4      
pandas      : 2.1.1       
pip         : 23.3.1      
pytest      : -           
pytest_cov  : -           
pytest_env  : -           
pytest_localftpserver: -           
setuptools  : 68.2.2      
sphinx      : 5.0.2       

INSTALLED VERSIONS: PIP
-----------------------
pytest-reportlog: -     

My code to run the OWC analysis:

import os, shutil
from pathlib import Path

import pyowc as owc
import argopy
from argopy import DataFetcher
import gsw

for WMO in ['7902010']:

    # Define float to calibrate:
    FLOAT_NAME = WMO#"1902573"

    # Set-up where to save OWC analysis results:
    results_folder = './OWC_analysis/%s' % FLOAT_NAME
    Path(results_folder).mkdir(parents=True, exist_ok=True)
    shutil.rmtree(results_folder) # Clean up folder content
    Path(os.path.sep.join([results_folder, 'float_source'])).mkdir(parents=True, exist_ok=True)
    Path(os.path.sep.join([results_folder, 'float_calib'])).mkdir(parents=True, exist_ok=True)
    Path(os.path.sep.join([results_folder, 'float_mapped'])).mkdir(parents=True, exist_ok=True)
    Path(os.path.sep.join([results_folder, 'float_plots'])).mkdir(parents=True, exist_ok=True)

    # fetch the default configuration and parameters
    USER_CONFIG = owc.configuration.load()

    # Fix paths to run at Ifremer:
    for k in USER_CONFIG:
        if "FLOAT" in k and "data/" in USER_CONFIG[k][0:5]:
            USER_CONFIG[k] = os.path.abspath(USER_CONFIG[k].replace("data", results_folder))
    USER_CONFIG['CONFIG_DIRECTORY'] = os.path.abspath('./OWC_analysis/constants')
    USER_CONFIG['HISTORICAL_DIRECTORY'] = os.path.abspath('./OWC_analysis/')  # where to find ARGO_for_DMQC_2020V03 and CTD_for_DMQC_2021V01 folders
    USER_CONFIG['HISTORICAL_ARGO_PREFIX'] = 'ARGO_for_DMQC_2022V03/argo_'
    USER_CONFIG['HISTORICAL_CTD_PREFIX'] = 'CTD_for_DMQC_2021V02/ctd_'
    USER_CONFIG['FLOAT_PLOTS_FORMAT'] = 'png'
    USER_CONFIG['CONFIG_MAX_CASTS'] = 127

    print(owc.configuration.print_cfg(USER_CONFIG))

    # Create float source data with argopy:
#    fetcher_for_real = DataFetcher(src='localftp', local_ftp='/Volumes/T7', cache=True, mode='expert').float(FLOAT_NAME)
    fetcher_for_real = DataFetcher(src='erddap', cache=True, mode='expert').float(FLOAT_NAME)
    ds = fetcher_for_real.load().data

    ds.argo.create_float_source(path=USER_CONFIG['FLOAT_SOURCE_DIRECTORY'], force='default')

    # Prepare data for calibration: map salinity on theta levels
    owc.calibration.update_salinity_mapping("", USER_CONFIG, FLOAT_NAME)

    # Set the calseries parameters for analysis and line fitting
    owc.configuration.set_calseries("", FLOAT_NAME, USER_CONFIG)

    # Calculate the fit of each break and calibrate salinities
    owc.calibration.calc_piecewisefit("", FLOAT_NAME, USER_CONFIG)

    # Results figures
    owc.plot.dashboard("", FLOAT_NAME, USER_CONFIG, levels=10)

if you still have your old environment?
I'm sorry, I do not. It was on a different computer.

[matdever]

any progress on this issue? Can I help in any way?

Thank you!

[gmaze]

I put this on my plate this morning 🥐

[gmaze]

@kamwal @thogar-computer did you managed to reproduce the error ?

[kamwal]

I run few example floats which were working some time ago(I didn't run this code since a while): ["3902398"], ["1901914"], ["3901960"] using the CTD_for_DMQC_2024V01 and ARGO_for_DMQC_2023V03 reference data. My Python version is 3.10.9 env.txt config.txt

I am not sure why it is like that. There have not been made any changes to this code from a long time

I have found the same error message:

Using precalculated data: L:\users\argo\kamwal\OWC_Python_Test_2\argodmqc_owc\data\float_mapped\map_6901933.mat

TOTAL TIME ELAPSED: 27.102484703063965 Using parameters found in data/float_calib/\calseries_6901933.mat To use new parameters, delete this file Traceback (most recent call last): File "L:\users\argo\kamwal\OWC_Python_Test_2\argodmqc_owc\start_with_pycharm.py", line 32, in owc.calibration.calc_piecewisefit("/", flt, USER_CONFIG) File "L:\users\argo\kamwal\OWC_Python_Test_2\argodmqc_owc\pyowc\calibration.py", line 590, in calc_piecewisefit theta, p, index, var_s_th, th = find_10thetas(copy.deepcopy(unique_sal), File "L:\users\argo\kamwal\OWC_Python_Test_2\argodmqc_owc\pyowc\core\finders.py", line 665, in find_10thetas interp_ptmp_sal = interpolate.interp1d([ptmp[theta_index, depth], File "C:\Users\kamwal\Programs\Anaconda3\envs\dmqc_pcm\lib\site-packages\scipy\interpolate\interpolate.py", line 436, in init _Interpolator1D.init(self, x, y, axis=axis) File "C:\Users\kamwal\Programs\Anaconda3\envs\dmqc_pcm\lib\site-packages\scipy\interpolate\polyint.py", line 54, in init self._set_yi(yi, xi=xi, axis=axis) File "C:\Users\kamwal\Programs\Anaconda3\envs\dmqc_pcm\lib\site-packages\scipy\interpolate\polyint.py", line 118, in _set_yi yi = np.asarray(yi) ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

Process finished with exit code 1

Yes, I found the same error when I ran the code on my side a few weeks ago.
Sorry, I didn't had a chance to work on this since.

[matdever]

I just thought I would check on this issue as @kamwal told me in Trieste that it was getting solved. Any progress? Sorry to bother - I rely on this to develop some of the DMQC corrections for biases due to CTD's compressibility.

[kamwal]

Hi Matt,
Thank you for your patience. Following our meeting in Trieste, we had an initial discussion with dirkslawinski , who may potentially have a solution to the issue with the falling code. I will follow up with him to verify this and will keep you updated.

[jamclar]

I've had a bit of a play around with @kamwal 's code and been able to make it work with a change to the dependency versions. I've made the following PR: #108

Would anyone be free to give it a try to see if it fixes it on their end too?

@matdever @gmaze

Many thanks