If you were a man applying to Berkeley's graduate school in 1973, you were almost twice as likely to be admitted as your female peers. On the surface, this seems to have been a flagrant case of gender discrimination. However, a closer inspection of the data reveals that women were more likely to apply to departments where the admission rate was lower overall, which was the true reason for any difference between the sexes. The Berkeley problem is a classic example of Simpson's paradox – an important concept in statistics where an effect disappears or even reverses when you control for other factors. Knowledge of this concept can prove critical in areas such as education policy, human resources, or any other field where bias or discrimination is a concern.
- 1Welcome to Berkeley
- 2Acceptance rate for men and women
- 3Visualizing the discrepancy
- 4Acceptance rate by department
- 5Alternative explanations
- 6Binary logistic regression: part i
- 7Binary logistic regression: part ii
- 8Behold Simpson's paradox
- 9Bias or discrimination?
Decision Scientist at Facebook
Joshua Feldman is a Decision Scientist at Facebook, where he uses data insights to help drive brand sentiment, user growth and engagement, and marketing effectiveness. He mainly codes in R and SQL, taking a specialist interest in causal inference, natural language processing and data visualization. He holds an MSc in quantitative research methodology from the London School of Economics.