The Neyman-Pearson lemma is foundational and important result in the theory of hypothesis testing. When presented in class the proof seemed magical and I had no idea where the ideas came from. I even drew a face like this 😲 next to the usual in my book when the proof was finished. Later in class we learnt the method of undetermined multipliers and suddenly I saw where the Neyman-Pearson lemma came from.
In this blog post, I’ll first give some background and set up notation for the Neyman-Pearson lemma. Then I’ll talk about the method of undetermined multipliers and show how it can be used to derive and prove the Neyman-Pearson lemma. Finally, I’ll write about why I still think the Neyman-Pearson lemma is magical despite the demystified proof.
In the set up of the Neyman-Pearson lemma we have data which is a realisation of some random variable . We wish to conclude something about the distribution of from our data by doing a hypothesis test.
In the Neyman-Pearson lemma we have simple hypotheses. That is our data either comes from the distribution or from the distribution . Thus, our null hypothesis is and our alternative hypothesis is .
A test of against is a function that takes in data and returns a number . The value is the probability under the test of rejecting given the observed data . That is, if , we always reject and if we never reject . For in-between values , we reject with probability .
An ideal test would have two desirable properties. We would like a test that rejects with a low probability when is true but we would also like the test to reject with a high probability when is true. To state this more formally, let and be the expectation of under and respectively. The quantity is the probability we reject when is true. Likewise, the quantity is the probability we reject when is true. An optimal test would be one that minimises and maximises .
Unfortunately the goals of minimising and maximising are at odds with one another. This is because we want to be small in order to minimise and we want to be large to maximise . In nearly all cases we have to trade off between these two goals and there is no single test that simultaneously achieves both.
To work around this, a standard approach is to focus on maximising while requiring that remains below some threshold. The quantity is called the power of the test . If is a number between and , we will say that has level– if . A test is said to be most powerful at level-, if
- The test is level-.
- For all level- tests , the test is more powerful than . That is,
Thus we can see that finding a most powerful level- test is a constrained optimisation problem. We wish to maximise the quantity
subject to the constraint
With this in mind, we turn to the method of undetermined multipliers.
The method of undetermined multipliers
The method of undetermined multipliers (also called the method of Lagrange multipliers) is a very general optimisation tool. Suppose that we have a set and two function and we wish to maximise subject to the constraint .
In the context of hypothesis testing, the set is the set of all tests . The objective function is given by . That is, is the power of the test . The constraint function is given by so that if and only if has level-.
The method of undetermined multipliers allows us to reduce this constrained optimisation problem to a (hopefully easier) unconstrained optimisation problem. More specifically we have the following result:
Proposition: Suppose that is such that:
- There exists , such that maximises over all .
Then maximises under the constraint .
Proof: Suppose that satisfies the above two dot points. We need to show that for all , if , then . By assumption we know that and maximises . Thus, for all ,
Now suppose . Then, and so and so as needed.
The constant is the undetermined multiplier. The way to use the method of undetermined is to find values that maximise for each . The multiplier is then varied so that the constraint is satisfied.
Proving the Neyman-Pearson lemma
Now let’s use the method of undetermined multipliers to find most powerful tests. Recall the set which we are optimising over is the set of all tests . Recall also that we wish to optimise subject to the constraint . The method of undetermined multipliers says that we should consider maximising the function
where . Suppose that both and have densities1 and with respect to some measure . We can we can write the above expectations as integrals. That is,
Thus the function is equal to
We now wish to maximise the function . Recall that is a function that take values in . Thus, the integral
is maximised if and only if when and when . Note that if and only if . Thus for each , a test maximises if and only if
The method of undetermined multipliers says that if we can find so that the above is satisfied and , then is a most powerful test. Recall that is equivalent to . By summarising the above argument, we arrive at the Neyman-Pearson lemma,
Neyman-Pearson Lemma2: Suppose that is a test such that
then is most powerful at level- for testing against .
The magic of Neyman-Pearson
By learning about undetermined multipliers I’ve been able to better understand the proof of the Neyman-Pearson lemma. I now view it is as clever solution to a constrained optimisation problem rather than something that comes out of nowhere.
There is, however, a different aspect of Neyman-Pearson that continues to surprise me. This aspect is the usefulness of the lemma. At first glance the Neyman-Pearson lemma seems to be a very specialised result because it is about simple hypothesis testing. In reality most interesting hypothesis tests have composite nulls or composite alternatives or both. It turns out that Neyman-Pearson is still incredibly useful for composite testing. Through ideas like monotone likelihood ratios, least favourable distributions and unbiasedness, the Neyman-Pearson lemma or similar ideas can be used to find optimal tests in a variety of settings.
Thus I must admit that the title of this blog post is a little inaccurate and deceptive. I do believe that, given the tools of undetermined multipliers and the set up of simple hypothesis testing, one is naturally led to the Neyman-Pearson lemma. However, I don’t believe that many could have realised how useful and interesting simple hypothesis testing would be.
- The assumption that and have densities with respect to a common measure is not a restrictive assumption since one can always take and the apply Radon-Nikodym. However there is often a more natural choice of such as Lebesgue measure on or the counting measure on .
- What I call the Neyman-Pearson lemma is really only a third of the Neyman-Pearson lemma. There are two other parts. One that guarantees the existence of a most powerful test and one that is a partial converse to the statement in this post.