Feat/scalar with window

[madtoinou]

Checklist before merging this PR:

• Mentioned all issues that this PR fixes or addresses.
• Summarized the updates of this PR under Summary.
• Added an entry under Unreleased in the Changelog.

Fixes #1540.

Summary

• Possibility to pass Scaler/Pipeline to historical_forecasts(), backtest() and gridsearch() to avoid data-leakage
  • When enable_optimization=True, the Pipeline must already be fitted (when applicable), all the series are transformed in one pass
  • Pipeline are systematically fitted and applied to the series between each forecast horizon (regardless of the retrain parameter value)

Other Information

The input series to these methods must always be "un-processed" when providing data_transformers in order to avoid "double scaling" of the series.

This PR is based on #2021.

[codecov]

Codecov Report

Attention: Patch coverage is 92.98246% with 8 lines in your changes missing coverage. Please review.

Project coverage is 94.10%. Comparing base (d103a05) to head (a633564).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
darts/models/forecasting/forecasting_model.py	88.46%	3 Missing ⚠️
darts/utils/historical_forecasts/utils.py	94.33%	3 Missing ⚠️
...essing/transformers/invertible_data_transformer.py	80.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2529      +/-   ##
==========================================
- Coverage   94.15%   94.10%   -0.05%     
==========================================
  Files         139      139              
  Lines       14890    14992     +102     
==========================================
+ Hits        14019    14108      +89     
- Misses        871      884      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

dennisbader commented on 2024-11-12T09:30:45Z
----------------------------------------------------------------

Line #8.        "enable_optimization": False,

maybe remove?

madtoinou commented on 2024-11-12T10:51:33Z
----------------------------------------------------------------

I added it to hide the warnings, but I don't mind removing it.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

dennisbader commented on 2024-11-12T09:30:46Z
----------------------------------------------------------------

Here I think it would be great to leave the example without data_transformers in historical_forecasts, and then show in a next example how it's done with the transformers. Also:

• The plot is identical to before, shouldn't it have some diffs at least?
• we should really transform the series as well

madtoinou commented on 2024-11-12T11:13:35Z
----------------------------------------------------------------

Updated the notebook; two examples (one with, one without the auto-scaling) while minimizing the changes

[madtoinou]

I added it to hide the warnings, but I don't mind removing it.

---

View entire conversation on ReviewNB

[madtoinou]

Updated the notebook; two examples (one with, one without the auto-scaling) while minimizing the changes

---

View entire conversation on ReviewNB

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

dennisbader commented on 2024-11-14T14:59:43Z
----------------------------------------------------------------

I think the note about parallelization was removed, let's add it back again?

[dennisbader]

This looks really good @madtoinou 🚀

I added a couple of suggestions, and two things which we should add:

• support data transformers in residuals
• support pre-trained data transformers regardless of global_fit

[dennisbader]

Beautiful 😍 I pushed some minor changes:

• adapt the docs a bit here and there
• renamed idx_series to series_idx
• converted transform() of fitted_values (in gridsearch) to inverse_transform() (should be inverse transform I think ;) )

There is one last change that we throw a warning instead of an error when global_fit=True and len(series)>1

[dennisbader]

Great work @madtoinou and @JanFidor 🚀 Thanks for all the work!