Suppose you have a sample drawn from a multivariate normal distribution in dimension . From this observation, you want to find a “good” estimate for . We will define our “good” estimate as such that expected value of the Euclidean distance between and is small. An obvious and reasonable choice would be to take . Surprisingly (at first), there are other estimators which are better. The most well-known estimator which provably dominates this estimate is the James-Stein estimator:
Notice this shrinks our naive estimate toward the origin. With this in mind, the surprise (somewhat) fades as the pictures below give pretty clear general intuition.
There are many variants of the James-Stein estimator. For example, there is nothing special about shrinking toward the origin. Also note the red points in the image on the left; for points with small norms, the multiplier in the James-Stein estimator is negative and actually pushes the estimate away from the origin. This too can be easily remedied. It seems the only magic in James-Stein concerns the amount of shrinkage applied. Where does the multiplier in the James-Stein estimator come from? We cannot shrink too much or too little. If we had some a priori idea for what should be, we could chose this as our point to shrink toward and hope to get something like the image on the left as opposed to the one on the right. This feels very Bayesian. In fact, the multiplier can be readily obtain from a Bayesian estimate of .
Let’s assume we don’t know , but we do know (our likelihood). We know this conditional distribution as it represents a known measurement process. We believe (our prior). We are taking this prior centered at , but the mean of the prior will dictate towards which point your estimate shrinks. For now, let’s also suppose we somehow know . Our goal is then to estimate (our posterier). This is a direct application of Bayes’ theorem which states the pdf of the posterior is proportional to pdfs of likelihood times the prior. Using this and the pdfs for normal distributions, it is not too hard to derive that
With this posterior distribution, we should take our estimate of to be
Indeed we have shrunk our estimate according to our prior, but this isn’t yet the James-Stein estimator. Recall we assumed we knew . What if we have some idea for what may be, but we have no clue about ? Well, we can estimate from the data ; this would fit into the category of empirical Bayes methods.
Given , if our goal is to estimate , we don’t really care about . Thus we look at the marginal distribution from which was drawn. Again, using the pdfs for our prior and likelihood, we integrate over and discover the marginal distribution so that
As we want to estimate , we should take
where we had to take the expected value of an inverse chi-squared distribution. Finally, we substitute our estimate for into our previous Bayes estimate , and we get the James-Stein estimate:
We see the James-Stein estimator is precisely the expected value of the poterior for using an empirical Bayes’ estimate. That’s great, but is it really that much better than the usual maximum likelihood estimate ? Like most things, it depends. If truly is , we can calculate
So if our prior is correct, the savings can be huge in high dimensions. Of course, the improvement deteriorates with worse priors, but we can calculate more generally
In the Bayes’ estimate we see the effect of a poor prior in the term, but averaging over all , this is an improvement over the max likelihood estimate. It would be nice to have , but this has proven to be a difficult term to evaluate due to the norm in the denominator. In fact, all the James-Stein related theorems I’ve found prove the James-Stein estimator dominates the max likelihood estimate for any but say nothing about the magnitude of the improvement in terms of . Still, in practice, the benefits are often significant which is why the James-Stein estimate is more than just a novelty.