Unit 3

Statistical inference for linear regression models

This unit introduces statistical inference for single‑predictor linear regression. Building on the examples from last week, we will learn about estimators and their properties, such as standard errors and confidence intervals. We will then derive unbiased estimators of the paramaters in a linear regression model. We will learn about coefficient of determination and correlations – two key quantities we use to access the relationship between variables in a linear regression model. Regression to the mean will also be covered.

Material:

• Notes and exercises
• Colab notebook from class (2024)
• Colab notebook from class (2025)

Concepts

Estimators, bias and consistency, Linear regression (single predictor) model and assumptions, Covariance, correlation and their relationship to regression slope. Regression to the mean. Least squares interpretation of covariance formula. Hypothesis testing for regression models.

Things to practice

• Identify whether a simple estimator is biased or not, by hand if possible, or using simulations in Python.
• Use the CLT to approximate the sample distribution of an estimator.
• Stating the assumptions of a linear regression model and identify (e.g. based on plots) when they are invalid.
• Be able to relate regression paramaters and means and variance of the predictor to the marginals.
• Understand the relationship between model paramaters, correlations, coefficient of determination. Know the meaning of correlations and cod.
• Interpret the output of a fitted linear regression model and access the goodness of fit (coefficient of determination).

Wikipedia References

• Linear regression
• Estimator
• Bias variance trade-off
• Ordinary least squares (OLS)
• Covariance and correlation
• Maximum likelihood estimation (optional)
• Confidence interval
• Statistical hypothesis testing
• Regression toward the mean