Feature importance of regressors, and forecast length longer than length of data

[k-dhingra]

Hi,

Firstly, thanks a lot for creating such a great package! It has definitely added significant value in my ts forecasting methods and I appreciate you putting this together.

I typically use the future_regressor function with multiple regressors given there are a number of variables that are known based on other forecasts. Does the AutoTS function provide details of which regressors are chosen and how they are weighted?

Secondly, do the models account for leads/lags of these future regressors? i.e. if I am forecasting GDP, but I have data on interest rate forecasts, which might typically impact GDP 6 months after interest rates increase/fall. Do the models account for this and find the optimal lag of future regressors? Can this data also then be used to create a longer forecast? i.e. if current interest rate rises will impact the GDP in 6 months time, but I also have a forecast of the next 6 months for interest rate rises, which would impact GDP in 12 months time. This should theoretically allow me to forecast 12 months ahead, as opposed to the 6 months of data I have available for future_regressor.

I understand the complexities of this, given different models will have different lags and impacts of future regressors, but I was wondering if this was possible?

[winedarksea]

@k-dhingra I'm glad you like it!

AutoTS does not itself provide details of future regressor usage other than the regressor_used param, which is whether all were used, or not (and of course, some models never use regressors). The one exception is the Cassandra model, which has regressor coefficients available for each (let me know if you need help with using that, as it isn't well-documented). Some Statsmodels package models would also provide this information, but that is not accessible directly - you'd have to rerun the model in statsmodels directly to get at it.

Does it find the optimal lag for future_regressors? No, it does not. That is something which is on my wishlist to figure out, but the only way I know right now is to loop through all possible lags. That is extremely computationally intensive, and often times finds the wrong lag anyway due to spurious correlations and the like. You can of course manually lag the data forward if you already know what the optimal lag is - the datetime index of the future regressor is for when you want the data applied, it doesn't have to be for when the data came from. For your case, it sounds like manually shifting the data forward would work just fine, giving you the 12 months. You could also pass a future_regressor that contains the same data at different lags - should not hurt, and might help.
If you have any good idea for finding the optimal lag, I am happy to work on implementing them as I can.

[k-dhingra]

Just as a follow up on this, does the ARDL model automatically test for specific lags in the exogenous variables/regressors? It looks like statsmodels accounts for this to find the optimal lag of regressors.

This was the best model chosen for one of the runs:

ARDL
{'fillna': 'linear', 'transformations': {'0': 'SeasonalDifference', '1': 'Detrend', '2': 'Slice'}, 'transformation_params': {'0': {'lag_1': 7, 'method': 'Median'}, '1': {'model': 'Linear', 'phi': 1, 'window': None, 'transform_dict': None}, '2': {'method': 0.5}}}
{'lags': 1, 'trend': 't', 'order': 1, 'regression_type': 'User'}
SMAPE: 51.48949824587886
MAE: 113454.19341554903
SPL: 0.37137185239116294

[winedarksea]

An SMAPE of 51% is pretty bad, usually - although maybe that is the best possible with a noisy dataset.

ARDL is from statsmodels, you could rebuild outside autots (using the GeneralTransformer for continuing to use the preprocessing) if you wanted to access more of the statistics that are provided.
The way I have it configured, ARDL only tests lags of 1, 2, 3, and 4. I think I remember testing it and much larger lags became quite slow, but I might be confusing it with another model.

[k-dhingra]

That's helpful to know, and all makes sense.
Yeah definitely a high SMAPE, but it unfortunately was due to the lack of available data (annual time series since only 2003 - i.e. less than 20 datapoints). A relatively low SMAPE has generally led to overfitting with this type of data.
Not sure if you have potential methods to overcome this either? I know Dynamic Factor Models try to work with a large number of regressors with (relatively) small sample sizes. But I'm a little confused about its implementation and interpretation.

[winedarksea]

@k-dhingra The best solution would be to get more data - monthly, if possible.
For dynamic factor models, you've probably already seen this:
https://www.statsmodels.org/dev/examples/notebooks/generated/statespace_dfm_coincident.html
Unobserved Components models also have worked well for me on limited data, and are a cousin of Dynamic Factor models.
With annual data, you lose all the seasonality information, so it might be just as well to ignore the time series nature and frame it as some other kind of statistical regression, with year as just one variable. Especially if you have external regressors, as it sounds like you do.

[k-dhingra]

That's really helpful to know. Thanks a tonne for all your help on this!

[winedarksea]

*additional comment deleted as spam. First time I've seen that on GitHub