CI failures significantly increased with automl_timeseries_forecasting_with_pycaret.ipynb

[amotl]

About

We are observing elevated frequencies of test failures on this notebook, both through CI runs triggered by PRs, and nightly scheduled ones, so the notebook in its current form should be considered super flaky, at the latest with immediate effect.

FAILED [test.py::test_notebook[automl_timeseries_forecasting_with_pycaret.ipynb]

Problem 0

self = <joblib.parallel.BatchCompletionCallBack object at 0x7f4f737cb910>

    def _return_or_raise(self):
        try:
            if self.status == TASK_ERROR:
>               raise self._result
E               ValueError: Input contains NaN.

• AutoML: CI trips with ValueError: Input contains NaN. #298

Problem 1

=========================== short test summary info ============================
FAILED test.py::test_notebook[automl_timeseries_forecasting_with_pycaret.ipynb] - nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 60 seconds.
The message was: Cell execution timed out.
Here is a preview of the cell contents:
-------------------
['target_data = pd.read_csv(', '    "https://data.4tu.nl/file/539debdb-a325-412d-b024-593f70cba15b/a801f5d4-5dfe-412a-ace2-a64f93ad0010"', ')', 'related_data = pd.read_csv(', '    "https://data.4tu.nl/file/539debdb-a325-412d-b024-593f70cba15b/f2bd27bd-deeb-4933-bed7-29325ee05c2e",']

-- https://github.com/crate/cratedb-examples/actions/runs/12819074099/job/35745960097?pr=811#step:6:1626

Evaluation

It looks like loading times of those resources are currently very high, so executing the notebook which acquires them runs into a timeout error. Sure enough, it also does not provide a good experience when actually using them.

• https://data.4tu.nl/file/539debdb-a325-412d-b024-593f70cba15b/a801f5d4-5dfe-412a-ace2-a64f93ad0010
• https://data.4tu.nl/file/539debdb-a325-412d-b024-593f70cba15b/f2bd27bd-deeb-4933-bed7-29325ee05c2e

References

• Update to crate-2.0.0, which uses orjson for JSON marshalling #811

/cc @wierdvanderhaar, @simonprickett, @ckurze, @kneth

[amotl]

Re: Problem 1

While it started working again, so this isn't exactly a blocker, maybe it would still be good to snatch the relevant dataset from https://data.4tu.nl/ into cratedb-datasets, and load it from there, when possible?

[amotl]

Problem 2

Just enumerating/sampling them, not investigating them at all. ¹

HTTPError: HTTP Error 502: Bad Gateway
FAILED test.py::test_file[automl_timeseries_forecasting_with_pycaret.py] - urllib.error.HTTPError: HTTP Error 502: Bad Gateway

-- https://github.com/crate/cratedb-examples/actions/runs/12860637559/job/35852836619?pr=816#step:6:1677

Footnotes

1. Most of them will go away by re-running CI workflows, that's unfortunate but acceptable. However, the frequency of trippings raised significantly in the last weeks, that's why those spots gained a dedicated tracking ticket (this one) now. Causing continuous troubles is not acceptable of course, because it distracts from other things. ↩

[amotl]

Problem 3

Once in a while, we also receive those errors on CI validation runs against automl_timeseries_forecasting_with_pycaret.py:

 >       if not isinstance(obj, dask.dataframe.core.DataFrame):
E       AttributeError: module 'dask.dataframe.core' has no attribute 'DataFrame'

/opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/sktime/datatypes/_adapter/dask_to_pd.py:137: AttributeError

-- https://github.com/crate/cratedb-examples/actions/runs/12860637559/job/35894717404?pr=816#step:6:1443

Inside here, there is also this to be observed, in this case also flagged as an ERROR:

ERROR    traitlets:__init__.py:98 Notebook JSON is invalid: Additional properties are not allowed ('metadata' was unexpected)

Failed validating 'additionalProperties' in stream:

On instance['cells'][22]['outputs'][3]:
{'metadata': {'nbreg': {'diff_ignore': ['/outputs']}},
 'name': 'stdout',
 'output_type': 'stream',
 'text': 'Fitting 3 folds for each of 10 candidates, totalling 30 fits\n'}
WARNING  traitlets:client.py:1234 No handler found for comm target 'dash'

-- https://github.com/crate/cratedb-examples/actions/runs/12860637559/job/35894717404?pr=816#step:6:1368

[amotl]

Thoughts 1

However, the frequency of trippings raised significantly in the last weeks, that's why [...]

[...] I am elevating this tracking ticket to bug Something isn't working level now, even if validation still succeeds occasionally, so it is not completely defunct ;].

The right opportunity sweet spot to improve the situation here might be modernizations in this regard, we need to do anyway:

• Migrate from PyCaret to AutoGluon crate-clients-tools#207

[amotl]

Problem 4

ValueError: could not convert string to float: 'No' is also emerging, either through bad connectivity / fluctuations at the data source vs. robustness problems of pycaret/sklearn, or a combination of both.

 /path/to/python3.11/site-packages/pycaret/utils/generic.py:585: UserWarning: Traceback (most recent call last):
    File " /path/to/python3.11/site-packages/pycaret/utils/generic.py", line 580, in _calculate_metric
      calculated_metric = score_func(y_test, target, sample_weight=weights, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[...]
     File " /path/to/python3.11/site-packages/sklearn/utils/_array_api.py", line 521, in _asarray_with_order
      array = numpy.asarray(array, order=order, dtype=dtype)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ValueError: could not convert string to float: 'No'

[amotl]

Thoughts 2

I can only hope migrating to AutoGluon, paired with relevant spring cleaning, can lead to a better micro-product, even if it's only scoped to this little notebook right now.

I dearly think anomaly detection and forecasting is an equally if not even more important topic for CrateDB on the ML domain, when compared against topics on LLMs that are listed on the same documentation page.