Week 1

Probability theory background

You should be familiar with the idea of sample spaces, outcomes, events, conditional probability, independence, marginalization, Bayes’ theorem, LLN and CLT. These are all covered in the textbook (1.1, 1.3, 1.4) and are easy to find elsewhere (or have ChatGPT explain it).

I will review Chebyshev’s inequality (L 1.15), Jensen’s inequality (1.16) since these are important later on.

In terms of distributions, you should be comfortable with Bernoulli, Binomial, Exponential, Poisson and Normal distributions.

You do not need to have any background in measure theory. You can skim 1.2 and 1.5, but we will circle back to that later as preparation for the more abstract theory of stochastic processes. Markov chains on countable state spaces will be handled without any measure theory, so we will begin with Chapter 3.

Discrete time Markov Chains

The main topic of week 1 will be discrete time Markov chains. The plan is to cover

• Transition probabilities
• Stochastic matrix
• P 3.4 (Chapman-Kolmogorov equation)
• Stationary distribution/invariant distribution
• L 3.5 (spectral radius of stochastic matrix)
• D 3.6 (Irreducibility)
• L 3.7
• T 3.8
• D 1.10
• T 3.11

Exercises

• 1.4
• 1.6
• 1.8
• 1.9
• 1.10
• 3.1 They say ``Analyze’’ the stationary distribution, which is open ended. Try writing down the recursive equation for the entries of the stationary distribution. You may recognize what the solution to this is. If not, you can look at up and try to justify it intuitively (it is a distribution you know).
• 3.2 I believe by $P^i$ they mean $P^{i_m}$
• 3.3
• 3.4