-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sparse GPs #4
Comments
@theogf By Sparse GPs, are referring to approximation using pseudo points? In this case, with non-Gaussian likelihoods? I think this is the right place for them. @willtebbutt What are your thoughts on this? |
Yes I meant the inducing points approach. And the Gaussian likelihood is also a usecase |
Yeah, sparse approximations are definitely something that we should include here. It's worth noting that we've already got some infrastructure for this in AbstractGPs. It might be worth considering how we could build sparse approximations for problems involving non-Gaussian likelihoods on top of this. Currently it assumes that you just want the closed-form solution, which isn't necessarily the case and should probably be relaxed i.e. perhaps you want to do stochastic variational inference (SVI) and iteratively find the optimal values for the approximate posterior mean and precision at the diagonalised pseudo-points. This kind of thing would also be helpful for this package, because you're typically going to be optimising the parameters of your approximate posterior process. Thinking about this has made me realise that it might have been a mistake to make the The issue is that under the current edit: Hmm I might have been a bit hasty above. We might be totally fine as we are. |
With our current API, the following is possible: # Set up prior and initialise approximate posterior.
f = GP(k)
f_approx_post = ApproxPosteriorGP(f, some_parameters_to_be_optimised)
lf_approx_post = LatentGP(f_approx_post(x), BernoulliLikelihood())
# Maybe you're doing SVI, so you compute an estimator of the ELBO.
elbo_estimator(lf_approx_post, y)
# Make some predictions at new locations under the approximate posterior.
rand(f_approx_post(x_pred), BernoulliLikelihood()) This approach has the advantage of not requiring any new objects in If we renamed # Set up prior and initialise approximate posterior.
f = GP(k)
f_approx_post = ApproxPosteriorGP(f, some_parameters_to_be_optimised)
lf_approx_post = LatentGP(f_approx_post, BernoulliLikelihood()) # note no `x`
# Maybe you're doing SVI, so you compute an estimator of the ELBO.
elbo_estimator(lf_approx_post(x), y) # Note the introduction of `x` here.
# Make some predictions at new locations under the approximate posterior.
rand(lf_approx_post(x_pred)) Note that @sharanry @theogf @devmotion any preferences between the above two approaches? edit: the second approach would introduce a discrepancy between how |
I think I'd prefer the second approach, it feels natural to me to only state the likelihood once. |
I agree with @devmotion, second approach seems cleaner and more natural. I also think we should update how @yebai had also suggested considering |
yes, it would be great to keep all the types and interface methods related to GP in one place, i.e. It is ok to have a separate package for non-Gaussian |
This is not true, and it's not the case that incorporating one's likelihood in with a GP type is strictly a win from an API perspective. First the API: the main distinction is that when working with the hierarchical Having a larger API would make implementing new This lack of a observed / latent proesses in the Also, a Furthermore, the distinction between an
In short,
I am always open to suggestions for better names. |
There is some confusion around my suggestion I think. Firstly, I am not suggesting removing the current Secondly, what I mean there is always some likelihood is probably not clear. I didn't mean the standard regression or classification setting, where when specifying a GP model, we often simultaneously also define a likelihood model. For example, in a time-series setting, where we apply a state-space model. In such a case, the likelihood is multi-level, the latent transitioning dynamics could have a Gaussian likelihood, while the observation process often has a separate likelihood model. It is also possible, the latent dynamics don't have a likelihood at all, while the observation process has a Gaussian likelihood. However, this setting still significantly differs from a GP with a Gaussian likelihood. In summary, the likelihood always exists but sometimes the specification is delayed (i.e. not immediately specified after the GP prior). In a more standard use case of GPs for regression and classification, the user can combine |
Regarding the name
|
I'm also a bit confused now haha.
I don't have partcularly strong feelings either way here. Provided that
Agreed.
I agree with this.
I agree with this but, again, I think it makes sense to define the likelihoods wherever we define the |
I am thinking that |
Okay. We should probably define the |
Sounds good -- I suggest that we move |
Do we want to include Sparse Gaussian Processes in this package or create an extra one for it?
In my experience a lot of things can be written once for both sparse and non-sparse models, so having them in the same package might simplify things
The text was updated successfully, but these errors were encountered: