Best Likelihood for Univariate Time Series

[jackalack]

Would it be possible to introduce a method that approximates the best likelihood to use for a provided univariate time series? We are exploring TFT with the various likelihoods for forecasting electricity consumption.

[hrzn]

Unfortunately I'm not really aware of such a method, except trying and measuring how well each likelihood works :D
Did you have something in mind?

A few things to know related to this topic:

• Distribution likelihoods (such as Gaussian, Laplace, etc) will (in principles) capture the "aleatoric uncertainty" in the data. That is, if you suspect that your series follow e.g., some Gaussian distribution with time-varying parameters, it's a good idea to use the Gaussian likelihood.
• (Unfortunately, there is a bug in GaussianLikelihood as of Darts 0.21.0. It will be fixed in the next release, but you'll get typically poor confidence intervals if you use it now. Sorry for that).
• Even if the kind of distribution is a good fit for your data, not all likelihoods will work well in practice. This is because the log-likelihood used to perform the optimization may not always provide good gradients to train the neural nets. The best way is to try.
• If you don't have any idea of what analytical distribution could be a good match, QuantileRegression provides non-parametric estimates of the distribution's quantiles directly and often works well in practice.
• If you're interested in capturing "epistemic uncertainty" (i.e., the uncertainty about the model itself), you can consider setting mc_dropout=True when calling predict(), which for dropout-enabled models will activate Monte Carlo dropout, effectively sampling over the family of models defined by all the dropout activation patterns.

[jackalack]

Hey @hrzn thanks for the quick response. Agreed that there is probably nothing more to be done other than cycling through the likelihoods and seeing which works best, but figure might as well centralize that code here rather than clients implement the logic on the periphery.

Very helpful pointer on the GaussianLikelihood bug. We are running experiments on our time series and did notice our CRPS was lower than expected relative to other likelihoods for GaussianLikelihood

Agree with your point that the best likelihood for the data still might not be the best likelihood to use, but when optimizing for the likelihood, model, hyper-parameters, etc it becomes a very expensive job, so hoping to have an initial analytics based step to figure out an appropriate subset of likelihoods

[astrogilda]

Hey @hrzn , thanks for the detailed response! Wrt to your suggestion of using mc_dropout=True with model.predict, I couldn't find any such functionality with model.historical_forecasts. Does that mean I'd have to write my own wrapper around predict, essentially copying over historical_forecasts's functionality?

[hrzn]

@astrogilda yes for now that would be the way to go. We have a fix for that tracked here: #1294
I'll close this issue in the meantime.

[astrogilda]

Awesome, thanks!