Unit 3

Statistical inference for linear regression models

This unit introduces statistical inference for single‑predictor linear regression. Building on the examples from last week, we will learn about estimators and their properties, such as standard errors and confidence intervals. We will then apply these concepts to linear regression modeling, deriving an unbiased estimator of the regression slope and understanding how hypothesis tests work for single-predictor regression models. Regression to the mean will also be covered.

Material:

• Notes and exercises
• Colab notebook (2024)

Concepts

Estimators, bias and consistency, Linear regression (single predictor) model and assumptions, Covariance, correlation and their relationship to regression slope. Regression to the mean. Least squares interpretation of covariance formula. Hypothesis testing for regression models.

Things to practice

• Identify whether a simple estimator is biased or not, by hand if possible, or using simulations in Python.
• Use the CLT to approximate the sample distribution of an estimator.
• Stating the assumptions of a linear regression model and identify (e.g. based on plots) when they are invalid.
• Be able to explain the phenomenon of regression to the mean.
• Interpret the output of a fitted linear regression model and access the goodness of fit.
• Recalling the basic facts quantities such as $p$-values and $R^2$ depend on the sample size.

Wikipedia References

• Linear regression
• Estimator
• Bias variance trade-off
• Ordinary least squares (OLS)
• Covariance and correlation
• Maximum likelihood estimation (optional)
• Confidence interval
• Statistical hypothesis testing
• Regression toward the mean