Abstract: Additive models with interactions have been considered extensively in the literature, using estimation methods such as maximum likelihood, Tikhonov regularization or Gaussian process regression. We present an alternative empirical-Bayes approach to selecting interaction effects using the I-prior approach (WB, 2020). Using a parsimonious specification of hierarchical interaction spaces, model selection is simplified. Furthermore, we present an efficient EM algorithm for estimating the key hyperparameters, not available for competing approaches. The EM algorithm facilitates finding the global maximizer of the marginal likelihood.
Simulations for linear regressions indicate competitive performance with methods such as the lasso and Bayesian variable selection using spike and slab priors or g-priors. However, the proposed methodology is more general and can also be used with interacting nonlinear regression functions.
The regression functions live in hierarchical interaction spaces for which we consider reproducing kernel Krein spaces (RKKSs), which generalize the too restrictive reproducing kernel Hilbert spaces (RKHSs) by loosening the positive definiteness requirement. Regardless of this, the Fisher information for the regression function is positive definite, hence defining an RKHS; loosely speaking, the norm of a function in this RKHS measures the difficulty of estimating it. Then, the I-prior maximizes entropy subject to aconstant difficulty of estimating the regression function, leading to a proper Gaussian prior, ie its paths are a.s. in the