Skip to content

Confidence Intervals

Evaluating a point estimator for a parameter gives an approximation . If we were to run the experiment multiple times, then we expect to see multiple different values for . The values of would cluster around the mean of the estimator with a spread given by the standard error of the estimator .

Example:
mu<-2; sigma<-5; n<-100
sample_means <- rowMeans(matrix(rnorm(100*n,mu,sigma),      # generate 100 sample means
								nrow=100))   
plot(density(sample_means),                                 # plot density estimate
	 main='Density estimate of sample means',               #  title
	 xlab='sample mean',ylab='probability density' )        #  axis labels

points(sample_means,rep(0,100))                             # mark sample means on x-axis
abline(v=mu, lwd=2, col='blue')                             #      expected value of estimator
abline(v=mu+2*sigma/sqrt(n), lwd=2, col='blue', lty=2)      #      two standard errors up
abline(v=mu-2*sigma/sqrt(n), lwd=2, col='blue', lty=2)      #      two standard errors down

If the point estimator is unbiased, then the expected value , which is great. But none of the individual values of are likely equal to . So what can we do if we only have one value of ? The best we can do is try to make explicit exactly how approximate our estimator is.

  •                point estimate for
  •       margin of error estimate for
  •   confidence interval for

Note that above should be proportional to the standard error .

More generally we can hope to replace the point estimate by a confidence interval     around such that contains the true value of the parameter with some probability -- i.e. is the probability of missing the parameter. (Note: and are random variables themselves -- usually functions of .) In general, the width of the confidence interval will be proportional to the standard error of ; so the more samples that are used, the narrower the confidence interval, and the better the estimate, will be.

Confidence Intervals for Mean of Normal (Using )

Suppose we are using to estimate for a distribution with known variance .
(Or the distribution isn't but we have using sufficiently many samples for CLT to apply.)

  •  
  •  
Applying this to error estimates for yields
  •  
  •   where  

The margin of error estimate for will be given by using values for to include of the probabilty distribution.
             where
      where

The confidence interval around the observed sample mean will be
   
The mean may not equal , but with probability , is at least inside this interval.

Example:
# Some values of z_alpha
cat('z.₀₅  = ' ,signif( qnorm(1-.05), 3),'\n')
cat('z.₀₂₅ = ' ,signif( qnorm(1-.025),3),'\n')
cat('z.₀₁  = ' ,signif( qnorm(1-.01), 3),'\n')
cat('z.₀₀₅ = ' ,signif( qnorm(1-.005),3),'\n')

Note that, in this case, the interval could also be described as
   
where and are the values with
        (which means )
 
Where is distributed with mean (the observed sample mean) and variance .

Example:

The code below computes different sample means, each using sample points. It marks the actual mean as a blue line through the center of the distribution the samples were drawn from. Each sample mean is then marked and its confidence interval is drawn. Confidence intervals are each marked at a different height so that the plot is more readable. If the actual mean is not included in a resulting CI then the interval is drawn in red. The expected width of confidence intervals is marked with dotted lines.

For the confidence interval, we expect of the resulting intervals to be marked in red (missing the actual mean).

###  CI.z(..)  and  plot_CI(..)  functions 


################################  z interval  ############
#' Make z confidence intervals for mean of normal rv
#'  
#' @ mu     mean of X
#' @ sigma  variance of X   
#' @ n      sample size for variance
#' @ alpha  desired confidence level
#' @ num_estimates  number of confidence intervals to construct
#' @ data   boolean: return data or return bootstrap confidence
#'
#' returns either data.frame: $x_bar, $min, $max, $in_CI
#'              or    number: bootstrap confidence level
#'
CI.z <- function(mu=0, sigma=1, n=100, alpha=.05, num_estimates=10000, data=FALSE) {

	sample_means <- rowMeans(matrix(      # generate num_estimates sample means
				 	  rnorm(n*num_estimates, mu, sigma),  
					  nrow=num_estimates
				     ))

	CI <- data.frame(  
		min = qnorm(  alpha/2, sample_means, sigma/sqrt(n)),
		max = qnorm(1-alpha/2, sample_means, sigma/sqrt(n))
	)

	# T/F vector telling if CI contains actual mean
	in_CI <- ((mu > CI$min) & (mu < CI$max))

	if (data==TRUE) { return(cbind(x_bar=sample_means, CI, in_CI)) }
	
	return (sum(in_CI)/num_estimates)   # return bootstrap confidence
}
################################


################################  plot_CI  ####################
#' plots confidence intervals spread vertically across graph
#'  
#' @ CI  data.frame $p_hat, $min, $max, $in_CI
#' @ y.max  confidence intervals plotted at heights from 0 to y.max
#'
plot_CI <- function(CI, y.max) {
	n <- nrow(CI)
	
	color <- rep('red', n)  # bad CI will be red and thick
	width <- rep(2,     n) 

	color[CI$in_CI] <- 'black'  # good CI will be black and thin
	width[CI$in_CI] <- 1
	
	height <- (1:n) * y.max / n # spread CI vertically across plot
	
	for (i in 1:n) {
		lines(c(CI$min[i], CI$max[i]),
			  c(height[i], height[i]),
			  col=color[i], lwd=width[i])
		points(CI$x_bar[i], height[i],
			  col=color[i], pch=16)
	}
}
Hidden code

Bootstrap Confidence of Interval

In the graphs above, it looks like approximately of the confidence intervals contain the mean, as desired. Just to verify, we can make bootstrap confidence estimates. The code below will make confidence intervals and calculate the proportion of them containing the mean. This is the bootstrap confidence. We'll calculate the bootstrap confidence for sample sizes ranging from to and plot the result. If we're making confidence intervals, then we hope for the bootstrap confidence to be approximately .

Hidden code

Confidence Intervals for Mean of (Using )

If we don't know the variance ahead of time, then we could try replacing the variance by its estimator, the sample variance . Unfortunately, this introduces even more error... now there is standard error coming from the estimate as well as further standard error coming from the estimate ... this results in confidence intervals being too small!

  •         (unless is big, this estimate is much too narrow!)

Example (directly substituting sample variance for variance):

The code below copies the setup from the previous example, but now each sample mean's confidence interval uses the corresponding sample variance instead of . In order to avoid the CLT, each sample mean is computed from only samples. (If we used samples as before, then the extra error would disappear.)

Though this code is attempting to compute confidence intervals; more than of the resulting intervals are marked red (don't include the actual mean). These intervals are too small!

1 hidden cell
Hidden code

Bootstrap Confidence

Looking at the graphs above, it seems that less than of the confidence intervals contain the mean. This matches our intuition that these CI should be too small due to us not accounting for the extra standard error coming from . For big this effect should be small. In the plot below we'll compute bootstrap confidence for from to .

As you can see, once is around , we can pretty much just replace by in the previous theory (though intervals will still be just a little too small).

Hidden code
‌
‌
‌