Model Selection and Component Pruning in Spectral Mixture Gaussian Process Regression

Gaussian process (GP) regression (Rasmussen and Williams, 2006) is a flexible, powerful, nonparametric regression method, which can be applied to sequential data as well as spatial data. One elegant property of GPs is that they are only dependent on a mean function and kernel (covariance function). The kernel determines which types of functions can be modeled with the GP. Interesting for this thesis project is the highly flexible spectral mixture (SM) kernel (Wilson, 2016). The SM kernel fits a mixture of Gaussians over the spectral density of the data. If this mixture contains sufficiently many Gaussians, or spectral mixture components, the SM kernel can approximate any stationary Gaussian process kernel.

In the SM kernel, the weights of irrelevant spectral mixture components are automatically driven to zero (this is a form of automatic relevance determination, see Wilson (2016) and Rasmussen and Williams (2006)). Model selection via the log marginal likelihood tends to be biased towards models with more spectral mixture components, which can lead to overfitting. Strategies for pruning spectral mixture components during training include a variance-based split/merge strategy (as in Chen and Van Laarhoven (2024)), or via variational inference (using, for instance, a Bayesian Gaussian mixture model).

Dependent on your interests, there are several possible research directions:

• (straightforward) Create a comparison of existing component pruning strategies; analyze prediction quality, computational complexity and/or interpretability.
• (medium) Investigate model selection for the SM kernel in the context of (some form of) sparse Gaussian process regression.
• (advanced) Extend an existing pruning approach, by for instance defining a better pruning heuristic, or attempt to develop your own pruning approach.

Some familiarity with GP regression is helpful. There exists an implementation for the SM kernel, written in GPFlow. Datasets could be synthesized or taken from existing GP papers.

If you would like to propose your own research direction within this topic, or if you would like to work on another GP related project, feel free to mail (mail address), we can discuss the possibilities.

Contact: Yuliya Shapovalova, Janneke Verbeek.

Supervision: Janneke Verbeek (Radboud University), Yuliya Shapovalova (Radboud University).

References

Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press.

Wilson, A. G. and Adams, R. P. (2013). "Gaussian Process Kernels for Pattern Discovery and Extrapolation." Proceedings of the 30th International Conference on Machine Learning (ICML).

Chen, K., van Laarhoven, T., & Marchiori, E. (2024). Compressing spectral kernels in Gaussian Process: Enhanced generalization and interpretability. Pattern Recognition, 155, 110642.