Which observations change a GP's hyperparameters?

E.g. a Gaussian process fits most observations well, except for one point away from the rest. It seems reasonable to call that point influential. But influential for what? The posterior mean nearby, the noise variance, and the fitted lengthscale are different quantities. A point can affect one considerably without changing the others very much.

This note looks at a specific question: how does a small change in one observed response affect hyperparameters estimated by marginal likelihood? This is response sensitivity, not deletion of an observation or contamination of the whole data distribution. Those require different perturbations.

The fitted parameters also depend on the data

Assume a zero-mean GP with a differentiable covariance C(θ) = K(θ) + σ²I. Inputs are fixed, and θ includes whichever covariance and noise parameters are being estimated. Apart from a constant, the negative log marginal likelihood is

At an interior local optimum θ̂, the gradient with respect to θ is zero. Change y_i to y_i + ε and follow the same local optimum. Differentiating that zero-gradient condition gives

Everything is evaluated at the fitted parameters and original data. This follows from implicit differentiation, provided the local optimum varies smoothly and H is nonsingular. Related influence-function methods use the same gradient-and-curvature structure, although the perturbation must be specified rather than assumed. [1]

There are two separate effects here. The vector b_i describes how the observation changes the parameter gradient. H determines how much the parameters move in response. A small gradient change can produce a large parameter change along a direction with weak curvature.

Where the observation enters

For this Gaussian model, set α = C⁻¹y and C_j = ∂C/∂θ_j. Direct differentiation gives

So sensitivity is not a function of the residual at point i alone. It involves the inverse covariance, the response vector, and the way the covariance changes with each parameter. Two observations with similar residuals can produce very different parameter updates.

For a squared-exponential kernel, changing the lengthscale changes covariance according to pairwise separation. An observation in a tightly sampled region interacts with many nearby points. An isolated observation has a different set of covariance relationships. Neither location is automatically more influential; the expression above makes clear why a residual ranking cannot decide it.

There is also a difference between moving the response and moving the input. A response perturbation leaves C fixed before refitting. Moving x_i changes a row and column of C directly. The formula for b_i above should not be reused for that case.

A flat optimum changes the interpretation

Lengthscale, signal variance and noise variance can compensate for each other over some datasets. The objective may then have a direction along which several parameter combinations fit similarly well. A perturbation aligned with that direction can move θ̂ substantially while changing predictions very little.

Suppose H = VΛV^T. In the local sensitivity, each component of b_i along an eigenvector is divided by its eigenvalue. Small positive eigenvalues amplify the response. An unusually large influence estimate may therefore be evidence of weak identification, rather than evidence that the observation is incorrect.

If H is singular, the local inverse is not defined. Adding damping makes a numerical solve possible, but it changes the sensitivity being reported. Close to a boundary or where optimisation switches between local optima, a first-order calculation can also be misleading.

Parameter movement and prediction movement

For a fixed test input, the posterior mean is μ* = k*^TC⁻¹y. Its total response sensitivity has a direct term and a refitting term:

The first term changes the observation while holding the hyperparameters fixed. The second accounts for refitting them. Looking only at the first misses the possibility that one response changes smoothness or noise assumptions across the domain. Looking only at parameter movement misses the possibility that the predictive mean barely changes.

Posterior variance has its own derivative. Under the assumptions here, its fixed-parameter value does not depend on y, so response sensitivity enters through refitting. That is another reason to distinguish sensitivity of the fitted model from sensitivity of its mean.

Using the calculation as a diagnostic

Parameter coordinates need to be chosen explicitly. A change of 0.1 in the lengthscale and a change of 0.1 in log lengthscale do not mean the same thing. Positive hyperparameters are often fitted in log coordinates; the Hessian, gradients and reported sensitivities should all use consistent coordinates. Response perturbations also need a meaningful scale.

A useful numerical check is to perturb one response by a small positive and negative amount, refit near the original solution, and compare the observed parameter changes with the linear prediction. Repeat with smaller perturbations. Agreement should improve locally when the assumptions hold. Poor agreement can indicate inaccurate optimisation, ill-conditioning, or a perturbation already too large for the approximation.

Observation removal is a separate test. Removing a point changes the covariance system and is not equivalent to sending its response towards zero. Local response sensitivity should not be presented as a leave-one-out result.

The purpose is to identify which observations and parameter directions need closer examination. It does not provide a rule for deleting inconvenient points. A highly influential observation may be the only one informing an important part of the function.

References and related work

1. Koh & Liang (2017). Understanding Black-box Predictions via Influence Functions. ICML. Background on local influence calculations; the response derivatives above are derived for the stated GP objective.
2. Sedai & Medda (2026). Influence functions for Gaussian process hyperparameter estimation. Related research listed on this site's publications page.

Comments or corrections: a.sedai@ucl.ac.uk.