INE266 4.1 Confidence Intervals on Variance and Proportion

Confidence Intervals for Variance of Normal

On the previous board we discussed confidence intervals for estimating the mean of a normal random variable. We considered settings where the variance was known ("-intervals") and where the variance was estimated using sample variance ("-intervals"). Now we'll look at the sample variance itself as an estimator.
We previously computed that , so we know this is unbiased.
But what about standard error and confidence intervals?

I think that the standard error is ... but we don't need this to make confidence intervals. Note that shouldn't be normally distributed (since in particular it can never be ; its distribution should be skewed with a right tail), so we don't expect confidence intervals for to look like those for .

To make confidence intervals for we will use the fact that if Normal() then
        "chi-squared with degrees of freedom"
"Chi-squared with degrees of freedom" is the distribution of "the sum of squares of independent normal() random variables".

Converting to , we see that if Normal(), then
        "chi-squared with degrees of freedom"

Shape of the (Chi-Squared) Distribution

The distribution is skewed. It is always positive, with a tail to the right.

•    ,   the number of degrees of freedom
• ,   twice the number of degrees of freedom

As , the distribution .
(Once degrees of freedom is around , you can pretty much use a normal distribution instead.)

Example:

The graphs below show the distribution for various degrees of freedom.

Solving
   
for the variance yields
               
Note: , so for big quantiles and for small quantiles.

Substituting in quantile values
  • quantile with
       
  • quantile with
       
results in a confidence interval for :
   

Note: The square root function is 1-1 and increasing, so taking square roots of the expression above yields confidence intervals for .
   

Example:

The code below computes different sample variances, each using sample points. It marks the actual variance as a blue line. Each sample variance is then marked and its confidence interval is drawn. Confidence intervals are each drawn at different heights so that the plot is more readable. If the actual variance is not included in a resulting CI then the interval is drawn in red. In the background is a chi-squared distribution renormalized to show the expected distribution of sample variances. Notice that, since is skewed with a tail to the right, so are the confidence intervals.

For the confidence interval, we expect of the resulting intervals to be marked in red (missing the true value).

Bootstrap Confidence

The graphs above look pretty reasonable. Since this is a confidence interval, we'd like to see of the intervals marked in red (not containing the actual variance). To test this more thoroughly (and for more values of ), I'll compute bootstrap confidence values for confidence intervals below. The bootstrap values below come from computing confidence intervals and calculating the proportion of them which contain the true variance. I do this for many different values of n to check whether sample size has any effect. As we would hope, the resulting plot is pretty close to .

Confidence Intervals for Population Proportion

Population proportion is a common application for confidence intervals and hypothesis testing. This is the situation which arises when you want to approximate the proportion of a large population with some property. Instead of testing every individual in the population, you instead sample a subset and test them. The observed sample proportion is an (unbiased) estimator for the true population proportion .

If the total population is sufficiently large compared to the sample size, then the observed sample is Binomial with size and probability -- this has mean and variance . Thus, the point estimator has expected value and standard error .

We would like to make a confidence interval for around . But this is unexpectedly difficult. The problem is that proportion is continuous, while the binomial distribution is discrete. Due to this, it isn't possible to get an estimator with the same quality as our previous examples.

The Wald CI for Population Proportion

The earliest and simplest way to make a confidence interval around is to note that, if is big, then the Binomial distribution is approximately Normal, according to the Central Limit Theorem. So as long as our sample is big enough,
      where  

So for big we could try to use a normal CI around , like the very first example of confidence intervals we covered. Unfortunately, since we don't know , we don't know .... but we know that -intervals should still be valid for big enough even when we approximate by .

Hardening our hearts against all doubts and fears, we can blindly combine all of these approximations to construct a "confidence interval" around . The result is called the "Wald" interval. This is the standard confidence interval for statistics textbooks to teach to students.

•     where is the normal quantile

Note: The Wald confidence interval is really, really bad. Since it doesn't account for the standard error of at all, the resulting confidence intervals are always too small, even for .

Example:

In the example below, we'll make calculations of Wald confidence intervals for with true population proportion . Our textbook's rule of thumb is that Wald CI are okay as long as and . Even though and fall in this range, you will note that many more than of the intervals below are colored red.

Bootstrap confidence (Comparing different methods)

Over the years, many different methods have been developed for computing confidence intervals for population proportion. The most well-known methods are the following

• The classical "Wald" interval, from approximating binomial by normal and using for gives intervals which are too small -- so the computed " CI" may actually capture the true value only of the time. This is especially pronounced for small and big values.
• In 1927, the Wilson "score" interval was proposed. Instead of using for , as the Wald interval does, Wilson suggested solving the equation
     
  for . [Our textbook confuses this with the later Agresti-Coull interval.] The score interval is pretty good, and is the interval method used by R's prop.test(..) function. Unfortunately the resulting interval bounds are rather intimidating.
     
• In 1934, the Clopper—Pearson "exact" interval was proposed using the beta distribution (from inverting binomial tests). The goal of Clopper-Pearson was to ensure that the " CI" captured the correct value at least of the time. [Clopper-Pearson is designed to be too big.] This is the interval method used by R's binom.test(..) function. It also results in intimidating formulas...
• In 1998, Agresti and Coull proposed a "corrected Wald" interval. Agresti and Coull did a detailed bootstrap analysis of different confidence interval methods and concluded that people should use the score interval. But then they note that, if you really want to use Wald, then there is a simple way to "correct" it for CI: It is enough to do Wald intervals with modified data and (add two "successes" and two "failures" to data).

For an excellent discussion of different intervals and their bootstrap confidence levels, I suggest looking at the 1998 paper of Agresti-Coull [https://math.unm.edu/~james/Agresti1998.pdf] which is quite readable even for non-statisticians.

Let's compare a few different methods!

1. The Wald interval. Approximate by normal, using for . This should give intervals which are too small; since we aren't accounting for the standard error of .. but if isn't too big or too small, and is large, then maybe if we pray hard enough things won't be too bad.
2. The Wilson score interval. (See the crazy formula above or in the textbook.) This is the standardly accepted "best choice".
3. Maximum likelihood estimated interval. I just made this one up because I was bored and wanted to try my hand at this problem.
   Instead of doing clever theory, I take advantage of the computer to brute-force a search for maximum likelihood values where
      
      
   In theory this should give good estimates... at the price of being VERY SLOW.

Note: For symmetric distributions (like normal or t) the MLE bounds are equivalent to the usual confidence interval bounds. But this is not true for the binomial distribution!

I'm a little bit sad that my "MLE interval" appears to be too big when n gets large. I guess its not too bad for something I made up. Possibly I'm not handling the endpoints of my interval correctly, and this could be fixed with a bit more thought....

Let's also do a comparison with "adjusted Wald".