E.g. two outputs are measured at the same inputs and appear strongly related. Fitting them together seems sensible. A multi-task GP can learn their dependence, so it should predict better than fitting each output separately. Sometimes it does. But the correlation is not enough to establish that.
There is a useful special case where the task covariance can be strongly non-diagonal and the predictive mean for each task still depends only on its own observed values. Working through this case helps separate dependence in the prior from information transfer after conditioning.
A shared kernel and a task covariance
Consider a zero-mean GP over inputs x and tasks t, with separable covariance
The matrix B describes task relationships. The kernel k describes how dependence varies over inputs. In this model, each task pair has the same input dependence up to a scalar multiplier. That is a substantial assumption: B cannot make two tasks share a broad trend but disagree only at short distances.
Assume every task is observed at the same n inputs X, observations are noise-free, and B and K = k(X, X) are positive definite. Stack the observations task by task into y. The covariance is B ⊗ K, where ⊗ denotes the Kronecker product.
This complete, common-input design is important. The result below is not a general statement about all multi-task models. Cancellation under these assumptions is discussed by Bonilla, Chai and Williams. [1]
Where the other tasks disappear
For a test input x*, let k* = k(X, x*). The covariance between ft(x*) and all training observations is Bt: ⊗ k*T. Conditioning gives the posterior mean
= (etT ⊗ k*TK−1)y
= k*TK−1yt.
The task row multiplied by B−1 becomes etT, selecting only task t. This is exactly the single-task, noise-free predictive mean using the same input kernel. No off-diagonal entry of B appears in the final expression.
For two tasks, take B with diagonal entries 1 and off-diagonal entries ρ, with |ρ| < 1. Holding the input kernel fixed, changing ρ does not change either predictive mean. There is no need for a numerical experiment to establish this particular result; it follows from the factorisation.
The posterior covariance between tasks s and t is
= Bst [k(x*, x*) − k*TK−1k*].
So the tasks can remain dependent in the posterior. Cancellation of cross-task contributions to the mean does not mean that the joint posterior has become independent. For a fixed Btt and k, the marginal posterior variance also matches the corresponding single-task result.
What changes with noise or missing observations?
With independent observation noise by task, the training covariance becomes B ⊗ K + D ⊗ I, where D is the diagonal matrix of task noise variances. In general its inverse no longer separates in the way used above, so observations from other tasks can contribute to the mean.
Noise does not automatically guarantee transfer, either. If B is diagonal, there is no task coupling. More unusually, if D is proportional to B, the covariance remains separable and the cancellation can persist. The covariance structure needs to be examined, rather than treating the presence of noise as a sufficient explanation.
Missing outputs or different sampling locations also remove the complete Kronecker design. An observation from a related task may then inform a region where the target task has little data. This is one reason a multi-task model can be more useful for partially observed outputs than for a fully observed, noise-free grid.
Another option is a sum of separable components, Σq B(q)kq(x, x′). Different components can express task relationships at different scales. The simple cancellation need not apply, although introducing extra components brings more parameters and does not by itself establish a predictive improvement.
Shared fitting is also a form of sharing
The derivation held k and B fixed. In practice, several tasks may be used to estimate common lengthscales or other kernel parameters. Predictions can therefore change when another task is added, even in a setting where direct cross-task contributions cancel at fixed parameters.
It helps to distinguish these mechanisms. One is conditioning the target prediction on other task values. The other is using those values to learn a shared model. A comparison that independently optimises the single-task and multi-task kernels includes both effects.
This distinction also explains a possible source of negative transfer. If one task needs a short lengthscale and another needs a long one, forcing them to use the same input kernel may compromise one or both fits. Large raw correlation does not resolve that mismatch.
Checking whether sharing actually helps
A useful initial check holds the parameters fixed and changes the auxiliary task values. Under the noise-free separable construction above, the target mean should remain unchanged. This can serve as an implementation check as well as a conceptual example.
For a predictive comparison, use a target-only baseline and evaluate each task separately. Report uncertainty quality as well as mean error. If the intended application involves missing outputs, the holdout should reproduce that pattern rather than only removing the same locations from every task.
A second comparison can fit the models independently, with a consistent tuning procedure. The difference between fixed-parameter and refitted results helps show whether any improvement comes from direct conditioning or shared parameter estimation. An independent-task model with shared kernel parameters can be an additional useful baseline.
A task-covariance heatmap describes part of the fitted model. It is not evidence on its own that the model benefits the target prediction. That requires checking what information is still useful after the target task's own observations have been accounted for.
Reference
- Bonilla, Chai & Williams (2007). Multi-task Gaussian Process Prediction. NeurIPS. The cancellation derivation above is a worked explanation of this established result, not a new result.
Comments or corrections: a.sedai@ucl.ac.uk.