How do I use validation_method=custom where validation_indexes is most recent datetime in dataframe

[shashikiran-maf]

I'm unable to use custom as validation_method. It errs out saying the training period should be greater than forecast_length.
My dataframe is has date range - 01-12-2021 to 31-03-2022 (four months data)
now with custom validation_method, I want to utilize the last month i.e., March 2022 as validation_set (validation_indexes).
I'm passing validation_indexes=[[pd.to_datetime('28-02-2022')]] at predict function, and my num_validation is set to 1.

This setup errs out with "training dataset should be greater than forecast_length, try altering min_allowed_train_percent"

[winedarksea]

Possibly two problems here @shashikiran-maf. One is the min allowed train percent arg, the second is that validation indexes is expecting a date range, not a single date.
Try passing something like this to the arguments:

AutoTS(
    # other args here
    min_allowed_train_percent=0,
).fit(
    # other args here
    validation_indexes=[
            pd.date_range("2022-03-01", "2022-03-31"),
    ],
)

[shashikiran-maf]

@winedarksea
However, I'm not sure if this is related this thread but it is throwing some different error now -

KeyError: "Evaluation Metrics are missing and all models have failed, by an error in template or metrics.\n Usually this means you are missing required packages for the models like fbprophet or gluonts,\n or that the models in model_list are inappropriate for your data.\n A new starting template may also help. KeyError('smape',)"

[shashikiran-maf]

@winedarksea
although you have answered this thread, unmarking it just to grab your attention to recent replies.

[winedarksea]

Unfortunately the error you are getting there is the general "All Models Failed" error. I can't really answer why without more code. Which model_list are you using? You should look at the other error messages printed earlier and see why the models tried are failing.

[adanebehr]

Hi, I am trying to use the custom validation method as well, to perform a "rolling validation" (or the hedgehog method, as I like to think about it). I have annual granularity data going back to the 1980s, so I am trying this:

model = AutoTS(forecast_length=5, frequency="A", num_validations=0, validation_method='custom')

validation_indexes=[pd.date_range(start="12/31/2019", end="12/31/2023", freq='A'), pd.date_range(start="12/31/2018", end="12/31/2022", freq='A')]

model=model.fit(df, validation_indexes=validation_indexes)

---

The custom validation indexes are not working as expected. I also am getting the "training dataset should be greater than forecast_length, try altering min_allowed_train_percent". Looking at the source code, it seems to me that the custom validation option causes a subset of the data (based on validation_indexes) to be passed to a training_testing split (with a testing tail as long as the forecast length), rather than using the indexes specified by validation_indexes as the test. And I think that is why the error about not having enough training data is being thrown.
Is that how it is meant to work? If so, do you happen to have an example code where you have successfully implemented this before?

Thanks!

[winedarksea]

@adanebehr
so firstly, you might benefit from using the 'seasonal 364' method which does pretty much the same thing, going backwards with each new window starting 364 days prior (in this example). Just 'seasonal' attempts to do an automatic fit of most relevant seasonal windows. And just "backwards" could work too with a full year forecast length. Those are the recommended options for simplicity. You can view model.validation_indexes to see what it chose after .fit is run - and I recommend trying it because this validation_indexes is exactly what you need to produce for custom validations.

So the only error I think you are making for the custom validations is you need to include the full date range including the training period. Yes, that is confusing, but allows constraining to different training start points as well. Also be aware that num_validations should be 1 less than the total number of rounds (it is one "evaluation" then n "validations" hence the syntax). So in your example num_validations should probably be 1.

validation_indexes=[
    pd.date_range("2021-01-01", "2022-05-02"),
    pd.date_range("2021-01-01", "2022-02-02"),
    pd.date_range("2021-01-01", "2022-03-03"),
]

In your case, the "2021-01-01" would be replaced by "1980-01-01" or whatever your first year of data is. Might be safest to do something like df.index[0] as the first value to generalize.