Where We Are

• Weeks 1 to 4: \(f\) known. The problem was to sample from it and to integrate against it.
• Weeks 5 and 6: \(F\) unknown, and the sample stood in for it. The parameter \(\theta\) was a fixed number. All the randomness was in \(\hat\theta\).
• This week: the same unknown \(\theta\), now described by a probability distribution that the data update.
• Weeks 1 and 3 already wrote that distribution down: \(\pi(\theta \mid \mathbf{x}) \propto L(\theta)\,\pi(\theta)\).

where \(\mathbf{x}\): the observed data; \(L(\theta)\): the likelihood, the model’s density at \(\mathbf{x}\) as a function of \(\theta\); \(\pi(\theta)\): the density of \(\theta\) before the data; \(\pi(\theta \mid \mathbf{x})\): its density after the data.

Two Readings of \(\theta\)

where \(L(\theta)\): the likelihood, the model’s density at the observed data as a function of \(\theta\); \(\pi(\theta \mid \mathbf{x})\): the density of \(\theta\) given the observed data \(\mathbf{x} = (x_1, \ldots, x_n)\).

• In both readings \(\theta\) is one number. The Bayesian distribution is about what we know of it: before the data (prior) and after (posterior).

One Dataset: 7 Heads in 10 Flips

• A coin with unknown probability of heads \(\theta\) is flipped \(n = 10\) times and shows \(k = 7\) heads.
• Model: \(X \mid \theta \sim \Bin(10, \theta)\).
• Likelihood: \(L(\theta) = \binom{10}{7}\,\theta^7(1-\theta)^3\).
• The task: learn about \(\theta\) from these data.

where \(X\): the number of heads in the 10 flips; \(k = 7\): its observed value.

The Likelihood Function

\[L(\theta) = \binom{10}{7}\,\theta^7(1-\theta)^3, \qquad 0 \le \theta \le 1\]

where \(\theta\): the probability of heads; the data, 7 heads in 10 flips, are held fixed.

• The likelihood is the model’s probability of the observed data, read as a function of \(\theta\).
• Same formula as the binomial probability, read the other way: there \(\theta\) is fixed and the count varies, here the count is fixed and \(\theta\) varies.
• It ranks the values of \(\theta\): a larger \(L(\theta)\) means the observed data were more probable under that \(\theta\).
• Both readings of \(\theta\) start from this function: the frequentist maximizes it, the Bayesian multiplies it by a prior.

The Likelihood for the Coin

• Orange curve: \(L(\theta)\) for 7 heads in 10 flips. Dotted line: its maximizer, \(\hat\theta = 0.7\). Points: \(\theta = 0.5\) and \(\theta = 0.7\).
• \(L(0.7) = 0.267\) and \(L(0.5) = 0.117\): the data are \(2.3\) times as probable under \(\theta = 0.7\) as under \(\theta = 0.5\).
• As a function of \(\theta\) the likelihood does not integrate to 1. The area under this curve is \(\binom{10}{7}\int_0^1 \theta^7(1-\theta)^3\,d\theta = 1/11\).

The Frequentist Answer

• MLE: \(\hat\theta = k/n = 0.70\), the maximizer of \(L(\theta)\).
• 95% Wald interval: \(\hat\theta \pm 1.96\sqrt{\hat\theta(1-\hat\theta)/n} = [0.42,\ 0.98]\).
• What the 95% means: if we repeat this experiment many times, 95% of such intervals will contain the true \(\theta\).
• The 95% describes the procedure over repeated experiments. It is no probability that \(\theta\) lies in \([0.42,\ 0.98]\): \(\theta\) is fixed, and this one interval either contains it or does not.

where \(\hat\theta\): the observed proportion of heads; \(1.96\): the 0.975 quantile of the standard normal.

The Bayesian Answer

• Before the flips every \(\theta\) between 0 and 1 is equally plausible: a flat prior.
• After the flips, the posterior: the distribution of \(\theta\) given the data. With a flat prior it is the likelihood curve, rescaled to area 1.
• Posterior mean \(0.67\). Shaded: the 95% credible interval \([0.39,\ 0.89]\). Given the data, \(\theta\) lies in it with probability 0.95.
• A direct question has a direct answer: \(P(\theta > 0.5 \mid \mathbf{x}) = 0.887\), the area to the right of the dashed line.

Same Data, Two Answers

where \([l, u]\): the credible interval, here the 2.5% and 97.5% quantiles of the posterior.

• The first 95% is a property of the procedure over repeated data. The second is a probability about \(\theta\), given these data.

Bayes’ Rule for Events

• Conditional probability: \(P(A \mid B) = P(A \cap B)/P(B)\), for \(P(B) > 0\).
• Write \(P(A \cap B)\) both ways: \(P(A \mid B)\,P(B) = P(B \mid A)\,P(A)\).

\[P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}, \qquad P(B) = P(B \mid A)\,P(A) + P(B \mid A^c)\,P(A^c)\]

where \(A\), \(B\): events; \(A^c\): the complement of \(A\); the second equation: the law of total probability.

• The rule turns \(P(B \mid A)\), which a model gives, into \(P(A \mid B)\), which we want.

A Medical Test

• A disease affects 1% of a population: \(P(\text{disease}) = 0.01\).
• A screening test is positive in 95% of the people who have the disease: \(P(+ \mid \text{disease}) = 0.95\), the sensitivity.
• It is negative in 95% of the people who are healthy: \(P(- \mid \text{healthy}) = 0.95\), the specificity.
• A person picked at random tests positive. We want \(P(\text{disease} \mid +)\).

where \(+\), \(-\): a positive and a negative test result.

The Common Mistake

• A common answer is 95%.
• That is \(P(+ \mid \text{disease})\): the probability of the test result, given the disease.
• The task asks for \(P(\text{disease} \mid +)\): the probability of the disease, given the test result.
• The two conditional probabilities are different numbers. Bayes’ rule turns one into the other, and it needs the 1%.

The Test as a Bayes Table

• \(P(+) = 0.95 \times 0.01 + 0.05 \times 0.99 = 0.059\), by the law of total probability: \(5.9\%\) of the population tests positive.
• \(P(\text{disease} \mid +) = 0.0095 / 0.059 = 0.161\).
• A positive test multiplies the probability of disease by \(16\), from 1% to \(16\%\). That is still far from 95%.
• Nobody disputes this calculation: whether a random person has the disease is a random event. The dispute starts when the unknown is a parameter.

Ten Thousand People

• Screen 10,000 people: 100 have the disease and 9,900 are healthy.
• Of the 100 with the disease, 95 test positive.
• Of the 9,900 healthy, 495 test positive.
• 590 positives in all, and 95 of them have the disease: \(95/590 = 0.161\).
• The false positives outnumber the true positives more than five to one, because the healthy outnumber the sick 99 to one.

Bayes’ Rule for a Discrete Unknown

• A bag holds two coins: one fair, \(\theta = 0.5\), and one biased, \(\theta = 0.8\).
• One coin is drawn at random and flipped 10 times: 7 heads. We want the probability that it is the biased coin.
• The unknown \(\theta\) has two possible values, with prior probabilities \(\pi(0.5) = \pi(0.8) = 1/2\).

\[\pi(\theta_j \mid \mathbf{x}) = \frac{L(\theta_j)\,\pi(\theta_j)}{\sum_{l} L(\theta_l)\,\pi(\theta_l)}\]

where \(\theta_1, \theta_2\): the possible values of \(\theta\); \(\pi(\theta_j)\): the prior probability of \(\theta_j\); \(L(\theta_j) = \binom{10}{7}\theta_j^7(1-\theta_j)^3\): the likelihood; \(\pi(\theta_j \mid \mathbf{x})\): the posterior probability of \(\theta_j\) given the data \(\mathbf{x}\) (7 heads in 10 flips).

The Bayes Table

• Multiply prior by likelihood, row by row. Divide by the column sum \(m(\mathbf{x}) = 0.159\).
• The data move the probability of the biased coin from \(0.50\) to \(0.63\).
• With prior probabilities \((0.90,\ 0.10)\) the same data give \((0.84,\ 0.16)\): with little data the prior matters.

where \(m(\mathbf{x}) = \sum_l L(\theta_l)\,\pi(\theta_l)\): the normalizing constant, the sum of the fourth column.

Bayes’ Rule for a Continuous Unknown

\[\pi(\theta \mid \mathbf{x}) = \frac{L(\theta)\,\pi(\theta)}{m(\mathbf{x})}, \qquad m(\mathbf{x}) = \int L(\theta)\,\pi(\theta)\,d\theta\]

• A continuous \(\theta\) has a row of the table for every value: the sum becomes an integral.
• \(\pi(\theta)\), the prior: our beliefs before seeing data.
• \(L(\theta) = \prod_{i=1}^n f(x_i \mid \theta)\), the likelihood: how data relates to parameters.
• \(\pi(\theta \mid \mathbf{x})\), the posterior: updated beliefs after seeing data.
• \(m(\mathbf{x})\), the normalizing constant (the evidence): the number that makes the posterior integrate to 1.

where \(f(x \mid \theta)\): the model’s density (or probability mass function) for one observation; \(\mathbf{x} = (x_1, \ldots, x_n)\): the observed data.

The Normalizing Constant Is the Hard Part

• Computing \(m(\mathbf{x}) = \int L(\theta)\,\pi(\theta)\,d\theta\) is often intractable: high-dimensional \(\theta\), no closed form, numerical integration fails.
• It does not depend on \(\theta\). It only rescales the curve \(L(\theta)\,\pi(\theta)\) so that it integrates to 1.
• One way around it: priors for which the integral is never computed.
• Another way: methods that use the posterior only up to a constant, such as accept-reject and MCMC.

where \(L(\theta)\): the likelihood; \(\pi(\theta)\): the prior density; \(m(\mathbf{x})\): the normalizing constant.

Posterior \(\propto\) Likelihood \(\times\) Prior

\[\pi(\theta \mid \mathbf{x}) \propto L(\theta)\,\pi(\theta)\]

where \(\pi(\theta \mid \mathbf{x})\): the posterior density; \(L(\theta)\): the likelihood; \(\pi(\theta)\): the prior density; \(\propto\): proportional to, as a function of \(\theta\).

• The two sides differ by a factor that is free of \(\theta\).
• Any factor free of \(\theta\) can be dropped: \(m(\mathbf{x})\), \(\binom{n}{k}\), \(1/x_i!\), the constant of the prior.
• The dropped constant comes back at the end: the posterior integrates to 1.
• Recall unnormalized targets in accept-reject? The same idea: the shape is enough.

Kernels

• The kernel of a density is the part that depends on the variable. The rest is the constant that makes it integrate to 1.
• Recognize the kernel, name the distribution, read off its parameters.

where \(a\), \(b\): the parameters of the named distribution, with \(b > 0\), and \(a > 0\) for the Beta and the Gamma; \(\Gam(a, b)\) has shape \(a\) and rate \(b\); \(\Normal(a, b^2)\) has mean \(a\) and variance \(b^2\).

Notation

\(\pi\) is always a density of the unknown. What stands behind the bar says what has been observed.

The Beta Distribution

\[\pi(\theta) = \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\,\Gamma(\beta)}\,\theta^{\alpha-1}(1-\theta)^{\beta-1}, \qquad 0 < \theta < 1\]

where \(\alpha, \beta > 0\): the two parameters; \(\Gamma\): the gamma function; the fraction: the constant that makes the density integrate to 1.

• A distribution for a probability: it lives on \((0, 1)\).
• Mean \(\dfrac{\alpha}{\alpha+\beta}\) and variance \(\dfrac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}\).
• Read \(\alpha\) as prior successes and \(\beta\) as prior failures: the prior is worth \(\alpha + \beta\) trials.
• With the same mean, a larger \(\alpha + \beta\) gives a narrower density.

Four Beta Densities

• \(\Betad(1, 1)\): flat, every \(\theta\) equally plausible, worth 2 trials.
• \(\Betad(5, 5)\) and \(\Betad(20, 20)\): both centered at \(0.5\), worth 10 and 40 trials, with standard deviations \(0.15\) and \(0.08\).
• \(\Betad(2, 10)\): mean \(1/6\), worth 12 trials, skewed toward small \(\theta\).

Choosing a Beta Prior

• Two numbers fix the prior: its mean \(m\) and the number of trials \(n_0\) it is worth.

\[\alpha = m\,n_0, \qquad \beta = (1 - m)\,n_0\]

where \(m = \alpha/(\alpha+\beta)\): the prior mean; \(n_0 = \alpha + \beta\): the prior sample size.

• A coin believed fair, worth 10 flips: \(m = 0.5\) and \(n_0 = 10\) give \(\Betad(5, 5)\).
• A die believed fair, worth 12 rolls: \(m = 1/6\) and \(n_0 = 12\) give \(\Betad(2, 10)\) for the probability of a six.
• Nothing to bring to the data: \(\Betad(1, 1)\), the flat prior.

Beta–Binomial: Setup

• Data: \(X \mid \theta \sim \Bin(n, \theta)\), observed \(X = k\).
• Prior: \(\theta \sim \Betad(\alpha, \beta)\).

\[\pi(\theta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\,\Gamma(\beta)}\,\theta^{\alpha-1}(1-\theta)^{\beta-1}, \qquad L(\theta) = \binom{n}{k}\,\theta^k(1-\theta)^{n-k}\]

where \(\alpha, \beta > 0\): the parameters of the prior; \(\Gamma\): the gamma function; \(k\): the number of successes in \(n\) trials. Both leading constants are free of \(\theta\).

Beta–Binomial: Posterior

\[\pi(\theta \mid \mathbf{x}) \propto \theta^k(1-\theta)^{n-k} \cdot \theta^{\alpha-1}(1-\theta)^{\beta-1} = \theta^{\alpha+k-1}(1-\theta)^{\beta+n-k-1}\]

where \(\mathbf{x}\): the data, \(k\) successes and \(n - k\) failures; \(\alpha\), \(\beta\): the parameters of the prior.

• This is the kernel of a Beta density: \(\theta \mid \mathbf{x} \sim \Betad(\alpha + k,\ \beta + n - k)\).
• Interpretation: prior pseudo-counts \((\alpha, \beta)\) plus observed counts \((k, n-k)\).
• No integral was computed. The constant is the Beta constant with the new parameters.

Just Add Counts

• Prior \(\Betad(\alpha, \beta)\): “I have seen \(\alpha\) successes and \(\beta\) failures before.”
• Data: \(k\) more successes and \(n - k\) more failures.
• Posterior \(\Betad(\alpha + k,\ \beta + n - k)\): “My count is now \(\alpha + k\) successes and \(\beta + n - k\) failures.”
• Updating with a conjugate prior pools the prior’s counts with the data’s counts.

where \(\alpha\), \(\beta\): the parameters of the prior; \(k\): the number of successes in \(n\) trials.

Two Priors for the Coin

• Flat prior, \(\Betad(1, 1)\): every \(\theta\) between 0 and 1 equally plausible before the data.
• With 7 heads and 3 tails the posterior is \(\Betad(1 + 7,\ 1 + 3) = \Betad(8, 4)\).
• Fair-coin prior, \(\Betad(5, 5)\): centered at 0.5, and worth 10 earlier flips, 5 heads and 5 tails.
• With the same data the posterior is \(\Betad(5 + 7,\ 5 + 3) = \Betad(12, 8)\).

where \(\Betad(\alpha, \beta)\): the prior, with pseudo-counts \(\alpha\) (heads) and \(\beta\) (tails).

Prior, Likelihood, Posterior

• Blue dashed: the prior. Orange band: the likelihood, rescaled to integrate to 1. Red: the posterior. Dotted line: the observed proportion of heads, \(k/n = 0.7\).
• Flat prior: the posterior is the rescaled likelihood, with mean \(0.667\).
• Fair-coin prior: the posterior sits between prior and likelihood, with mean \(0.600\).

A Posterior Is Likelihood Times Prior, Rescaled

• A Bayesian answer is a distribution for \(\theta\) given the data.
• Bayes’ rule is one computation in three settings: events, a discrete unknown, a continuous unknown.
• Proportionality takes the normalizing constant out of the work. The kernel identifies the posterior.
• Beta prior and binomial data give a Beta posterior: pseudo-counts plus counts.

The Loaded Die

• You find a die and suspect it is loaded toward six.
• You roll it 20 times and see 8 sixes, \(40\%\) of the rolls.
• A fair die shows a six in 1 roll of 6, about 17%.
• Model: \(X \mid \theta \sim \Bin(20, \theta)\), observed \(X = 8\).
• The task: the probability that the die favors six, \(P(\theta > 1/6 \mid \mathbf{x})\).

where \(\theta\): the probability of a six on one roll; \(X\): the number of sixes in the 20 rolls; \(\mathbf{x}\): the observed rolls.

A Prior for the Die

• Most dice are fair: center the prior at \(m = 1/6\).
• Make it worth \(n_0 = 12\) rolls: a weak opinion against 20 rolls of data.
• Then \(\alpha = 12 \times 1/6 = 2\) and \(\beta = 12 \times 5/6 = 10\): the prior is \(\Betad(2, 10)\).
• It reads “2 sixes in 12 earlier rolls”.
• Before any roll it gives \(P(\theta > 1/6) = 0.43\).

where \(m\): the prior mean; \(n_0\): the number of rolls the prior is worth.

The Die: Posterior by Hand

\[\pi(\theta \mid \mathbf{x}) \propto \underbrace{\theta^{8}(1-\theta)^{12}}_{\text{likelihood}} \cdot \underbrace{\theta^{2-1}(1-\theta)^{10-1}}_{\text{kernel of the prior}} = \theta^{9}(1-\theta)^{21} = \theta^{10-1}(1-\theta)^{22-1}\]

where \(\mathbf{x}\): the data, 8 sixes and 12 other faces; the dropped factors, \(\binom{20}{8}\) and the constant of the prior, are free of \(\theta\).

• This is the kernel of a Beta density: \(\theta \mid \mathbf{x} \sim \Betad(10, 22)\).
• Just add counts: \(2 + 8 = 10\) sixes and \(10 + 12 = 22\) other faces.
• Three steps: write the likelihood, multiply by the kernel of the prior, recognize the kernel.

Prior, Likelihood, Posterior for the Die

• Blue dashed: the prior \(\Betad(2, 10)\). Orange band: the rescaled likelihood. Red: the posterior \(\Betad(10, 22)\).
• Grey dashed line: a fair die, \(\theta = 1/6\). Dotted line: the observed proportion, \(0.4\). Shaded: the posterior probability that \(\theta > 1/6\).
• The posterior sits between the prior and the likelihood, nearer the likelihood: 20 rolls of data against a prior worth 12.

What the Posterior Says about the Die

• Posterior mean: \(10/32 = 0.3125\), between the prior mean \(0.167\) and the observed proportion \(0.4\).
• 95% credible interval: \([0.167,\ 0.480]\), the 2.5% and 97.5% quantiles of \(\Betad(10, 22)\).
• The task: \(P(\theta > 1/6 \mid \mathbf{x}) = 0.975\). Before the rolls it was \(0.43\).
• The 2.5% quantile of the posterior is \(1/6\) to three decimals, so \(P(\theta > 1/6 \mid \mathbf{x})\) is its complement, \(0.975\).
• In R: 1 - pbeta(1/6, 10, 22) and qbeta(c(0.025, 0.975), 10, 22).

The Same Rolls under Three Priors

• The first prior is flat. The other two are centered at \(1/6\), and the third is five times as strong as the second.
• The stronger the prior belief in a fair die, the smaller the posterior probability of a loaded one: \(0.996\), \(0.975\), \(0.901\).
• All three stay above one half: the conclusion survives the choice of prior, its strength changes.
• A posterior probability is a statement under a stated prior. Report the prior with it.

Two Questions about the Die

where \(X\): the number of sixes in 20 rolls; \(0.011\): the p-value of the one-sided test of \(\theta = 1/6\).

• \(0.011\) is no probability that the die is fair, and \(1 - 0.011\) is no probability that it is loaded.
• The two numbers answer different questions. Only the second is a probability about \(\theta\).

Conjugacy

• Definition (Conjugacy). A prior \(\pi(\theta)\) is conjugate to the likelihood \(L(\theta)\) if the posterior \(\pi(\theta \mid \mathbf{x})\) is in the same family as the prior.

• Advantage: a closed-form posterior, with no integration needed.

Sequential Updating

• First 10 flips, 7 heads, flat prior: posterior \(\Betad(8, 4)\).
• Use it as the prior for 30 more flips with 21 heads: posterior \(\Betad(8 + 21,\ 4 + 9) = \Betad(29, 13)\).
• One update on all 40 flips, 28 heads, from the flat prior: \(\Betad(1 + 28,\ 1 + 12) = \Betad(29, 13)\).
• The two routes agree: the order of the data does not matter.
• A prior can be read as data seen earlier.

The Gamma Distribution

• A distribution for a positive rate \(\lambda\), with shape \(\alpha\) and rate \(\beta\): mean \(\alpha/\beta\) and variance \(\alpha/\beta^2\).
• \(\Gam(2, 1)\) and \(\Gam(8, 4)\): both with mean 2, standard deviations \(1.41\) and \(0.71\).
• \(\Gam(4, 1)\) and \(\Gam(40, 10)\): both with mean 4, standard deviations \(2.00\) and \(0.63\).
• With the same mean, a larger \(\beta\) gives a narrower density.

Gamma–Poisson: The Data

• Recall the Poisson counts of the accept-reject lab? Ten counts: 3, 5, 4, 6, 3, 4, 5, 7, 4, 3.
• Model: \(X_1, \ldots, X_n \mid \lambda \iid \Pois(\lambda)\), with \(n = 10\), \(\sum_i x_i = 44\), \(\bar x = 4.4\).
• Prior: \(\lambda \sim \Gam(2, 1)\), with mean \(2\).
• The lab stated the posterior, \(\Gam\left(\alpha + \sum_i x_i,\ \beta + n\right)\), and sampled from it.

where \(\lambda\): the Poisson rate; \(\Gam(\alpha, \beta)\): shape \(\alpha\) and rate \(\beta\), with density \(\dfrac{\beta^\alpha}{\Gamma(\alpha)}\,\lambda^{\alpha-1}e^{-\beta\lambda}\) and mean \(\alpha/\beta\).

Gamma–Poisson: Posterior

\[L(\lambda) = \prod_{i=1}^n \frac{\lambda^{x_i}e^{-\lambda}}{x_i!} \propto \lambda^{\sum_i x_i}\,e^{-n\lambda}, \qquad \pi(\lambda) \propto \lambda^{\alpha-1}e^{-\beta\lambda}\]

\[\pi(\lambda \mid \mathbf{x}) \propto \lambda^{\sum_i x_i}\,e^{-n\lambda} \cdot \lambda^{\alpha-1}e^{-\beta\lambda} = \lambda^{\alpha+\sum_i x_i-1}\,e^{-(\beta+n)\lambda}\]

where \(x_1, \ldots, x_n\): the observed counts; \(\alpha\), \(\beta\): the shape and rate of the prior; the dropped factors \(1/x_i!\) and \(\beta^\alpha/\Gamma(\alpha)\) are free of \(\lambda\).

• This is the kernel of a Gamma density: \(\lambda \mid \mathbf{x} \sim \Gam\left(\alpha + \sum_i x_i,\ \beta + n\right)\).
• Interpretation: \(\alpha\) is a prior total count and \(\beta\) a prior number of observations. The data add \(\sum_i x_i\) and \(n\).

Gamma–Poisson: Prior, Likelihood, Posterior

• Blue dashed: the prior \(\Gam(2, 1)\). Orange band: the rescaled likelihood. Red: the posterior \(\Gam(46, 11)\). Dotted lines: the prior mean (blue), the sample mean \(\bar x\) (dark), the posterior mean (red).
• Posterior mean \(4.18\), standard deviation \(0.62\), 95% credible interval \([3.06,\ 5.47]\) (the 2.5% and 97.5% posterior quantiles).
• The posterior mean lies between the prior mean \(2\) and \(\bar x = 4.4\), close to \(\bar x\).

Normal–Normal: The Result

• Data: \(X_1, \ldots, X_n \mid \mu \iid \Normal(\mu, \sigma^2)\), with \(\sigma^2\) known.
• Prior: \(\mu \sim \Normal(\mu_0, \tau_0^2)\).
• Posterior: \(\mu \mid \mathbf{x} \sim \Normal(\mu_n, \tau_n^2)\), with

\[\frac{1}{\tau_n^2} = \frac{1}{\tau_0^2} + \frac{n}{\sigma^2}, \qquad \mu_n = \tau_n^2\left(\frac{\mu_0}{\tau_0^2} + \frac{n\bar x}{\sigma^2}\right)\]

where \(\mu_0\), \(\tau_0^2\): the prior mean and variance; \(\mu_n\), \(\tau_n^2\): the posterior mean and variance; \(\bar x\): the sample mean; \(\sigma^2\): the known variance of one observation.

• Precision is one over variance. Precisions add: the prior’s, plus \(n\) times that of one observation.

Normal–Normal: Completing the Square

• Likelihood in \(\mu\): \(L(\mu) \propto \exp\left\{-\dfrac{1}{2\sigma^2}\sum_i (x_i - \mu)^2\right\} \propto \exp\left\{-\dfrac{n}{2\sigma^2}\left(\mu^2 - 2\bar x\mu\right)\right\}\).
• Prior: \(\pi(\mu) \propto \exp\left\{-\dfrac{1}{2\tau_0^2}\left(\mu^2 - 2\mu_0\mu\right)\right\}\).
• Product: \(\pi(\mu \mid \mathbf{x}) \propto \exp\left\{-\dfrac{1}{2}\left[\left(\dfrac{1}{\tau_0^2} + \dfrac{n}{\sigma^2}\right)\mu^2 - 2\left(\dfrac{\mu_0}{\tau_0^2} + \dfrac{n\bar x}{\sigma^2}\right)\mu\right]\right\}\).
• A normal kernel has this form: \(\exp\left\{-\dfrac{1}{2}\left[\dfrac{1}{\tau_n^2}\,\mu^2 - 2\,\dfrac{\mu_n}{\tau_n^2}\,\mu\right]\right\}\). Match the coefficient of \(\mu^2\), then that of \(\mu\).

where every dropped factor is free of \(\mu\): \(e^{-\sum_i x_i^2/(2\sigma^2)}\), \(e^{-\mu_0^2/(2\tau_0^2)}\), \(e^{-\mu_n^2/(2\tau_n^2)}\), and the constants of the two densities.

A Measurement, Updated by Hand

• An instrument measures a quantity \(\mu\) with normal error of known standard deviation \(\sigma = 10\).
• Earlier work puts \(\mu\) near 100, give or take 5: prior \(\mu \sim \Normal(100, 5^2)\).
• Four new measurements: 104, 118, 97, 121, with \(\bar x = 110\).
• Precisions: prior \(1/5^2 = 0.04\), data \(4/10^2 = 0.04\), posterior \(0.08\).

\[\tau_n^2 = \frac{1}{0.08} = 12.5, \qquad \mu_n = 12.5\,(100 \times 0.04 + 110 \times 0.04) = 105\]

where \(\tau_n^2\), \(\mu_n\): the posterior variance and mean; precision: one over variance.

Prior, Likelihood, Posterior for the Measurement

• Blue dashed: the prior. Orange band: the rescaled likelihood, \(\Normal(\bar x, \sigma^2/n)\). Red: the posterior. Dotted lines: the prior mean \(\mu_0\) (blue), the sample mean \(\bar x\) (dark), the posterior mean \(\mu_n\) (red).
• Posterior \(\Normal(105, 12.5)\): standard deviation \(3.54\), 95% credible interval \([98.1,\ 111.9]\).
• Prior and data have equal precision, so the posterior mean is halfway between \(\mu_0\) and \(\bar x\). The posterior is narrower than both.

The Posterior Mean Is a Weighted Average

\[\E[\lambda \mid \mathbf{x}] = \frac{\alpha + \sum_i x_i}{\beta + n} = \underbrace{\frac{\beta}{\beta + n}}_{w}\cdot\frac{\alpha}{\beta} \;+\; \underbrace{\frac{n}{\beta + n}}_{1 - w}\cdot\bar x\]

where \(\alpha/\beta\): the prior mean; \(\bar x = \frac{1}{n}\sum_i x_i\): the sample mean; \(w\): the weight on the prior mean.

• Gamma–Poisson: the posterior mean is a weighted average of the prior mean and the sample mean.
• The weight on the prior is \(w = \beta/(\beta + n)\): the prior counts as \(\beta\) observations against the \(n\) in the data.
• The lab’s counts: \(w = 1/11\), and \(\frac{1}{11} \times 2 + \frac{10}{11} \times 4.4 = 4.18\).

The Weighted Average for Three Pairs

\[\text{posterior mean} = w \times \text{prior mean} + (1 - w) \times \text{data estimate}, \qquad w = \frac{n_0}{n_0 + n}\]

where \(n_0\): the number of observations the prior is worth; \(n\): the number in the data; \(w\): the prior’s weight.

What Happens as \(n\) Grows

• The weight on the prior, \(w = n_0/(n_0 + n)\), goes to 0: the posterior mean approaches the data estimate, which is the MLE in all three pairs.
• The posterior variance goes to 0 at rate \(1/n\): the posterior concentrates.
• Two analysts with different priors end with nearly the same posterior: the data swamp the prior.
• With little data the prior matters, so it is stated, and the analysis is rerun under another one.
• For large \(n\) the 95% credible interval and the 95% confidence interval are numerically close. Their meanings stay different.

where \(n_0\): the prior sample size; \(n\): the number of observations; \(w\): the weight on the prior mean.

Two Priors, More Data

• Each panel: the coin’s posterior after \(n\) flips with 70% heads. Solid: under the flat prior. Dashed: under the fair-coin prior \(\Betad(5, 5)\). Dotted line: the observed proportion, \(0.7\).
• Posterior means at \(n = 10\): \(0.667\) and \(0.600\). At \(n = 640\): \(0.699\) and \(0.697\).
• The weight on the fair-coin prior falls from \(0.50\) to \(0.015\). The two posteriors merge, and both narrow around \(0.7\).

A Flat Prior Returns the Classical Numbers

• Normal data with known \(\sigma\) and prior \(\mu \sim \Normal(\mu_0, \tau_0^2)\): let the prior variance \(\tau_0^2\) grow without bound.
• The prior sample size \(n_0 = \sigma^2/\tau_0^2\) goes to 0, and so does the weight on the prior.
• The posterior becomes \(\Normal(\bar x,\ \sigma^2/n)\): \(\mu_n \to \bar x\) and \(\tau_n^2 \to \sigma^2/n\).
• The 95% credible interval becomes \(\bar x \pm 1.96\,\sigma/\sqrt{n}\): the confidence interval, number for number.
• The two intervals differ when \(n\) is small or the prior is strong. Their meanings differ always.

where \(\mu_n\), \(\tau_n^2\): the posterior mean and variance; \(\bar x\): the sample mean of the \(n\) observations.

Two Rates: an A/B Test

• A website shows each visitor one of two designs, chosen at random.
• Control: 10 conversions in 100 visitors. Treatment: 15 conversions in 100 visitors.
• Model: \(X_C \mid \theta_C \sim \Bin(100, \theta_C)\) and \(X_T \mid \theta_T \sim \Bin(100, \theta_T)\), independent.
• Priors: \(\theta_C\) and \(\theta_T\) independent \(\Betad(1, 1)\). Just add counts: \(\theta_C \mid \mathbf{x} \sim \Betad(11, 91)\) and \(\theta_T \mid \mathbf{x} \sim \Betad(16, 86)\).
• The task: \(P(\theta_T > \theta_C \mid \mathbf{x})\), the probability that the new design is the better one.

where \(\theta_C\), \(\theta_T\): the conversion rates of the two designs; \(X_C\), \(X_T\): the two counts of conversions; \(\mathbf{x}\): both observed counts.

Two Posteriors

• Dashed: the control’s posterior, mean \(0.108\). Solid: the treatment’s, mean \(0.157\). Dotted lines: the observed proportions, \(0.1\) and \(0.15\).
• The two posteriors overlap: 100 visitors per design leave room for either one to be the better.
• The task is a probability under both posteriors at once: a double integral, with no formula worth deriving.

Posterior Summaries by Monte Carlo

• A named posterior can be sampled: rbeta(), rgamma(), rnorm(). A probability is then the share of the draws in a region: week 1’s Monte Carlo.

set.seed(536)
S <- 1e5
theta_C <- rbeta(S, 11, 91)            # the control's posterior
theta_T <- rbeta(S, 16, 86)            # the treatment's posterior
p_hat <- mean(theta_T > theta_C)       # P(theta_T > theta_C | x)
se <- sqrt(p_hat * (1 - p_hat) / S)    # its Monte Carlo standard error

• Result with \(S = 10^5\) draws: \(0.851\), Monte Carlo standard error \(0.001\). Numerical integration gives \(0.853\).
• The same draws answer any other question about the two rates, with no new algebra.
• When the posterior has no name, draws are still enough: accept-reject produces them, and so does MCMC.

The Posterior of the Difference

• Histogram: the \(10^5\) draws of \(\theta_T - \theta_C\). Red: the draws with \(\theta_T > \theta_C\), a share of \(0.851\). Grey: the rest.
• The difference has posterior mean \(0.049\) and 95% credible interval \([-0.043,\ 0.142]\), which contains 0.
• The treatment is probably the better design. These data do not settle it.

When Conjugacy Fails

• Most real problems are non-conjugate: hierarchical models, mixture models, non-linear models, robust models (for example \(t\)-distributed errors), most machine learning models.
• Example: logistic regression with a normal prior on the coefficients. The posterior has no closed form.
• The posterior is still known up to a constant: \(\pi(\theta \mid \mathbf{x}) \propto L(\theta)\,\pi(\theta)\).
• In low dimensions a grid, accept-reject, or importance sampling handles such a posterior. In high dimensions each one breaks down.
• Markov chain Monte Carlo is the method for the high-dimensional case.

where \(L(\theta)\): the likelihood; \(\pi(\theta)\): the prior density; \(\pi(\theta \mid \mathbf{x})\): the posterior density.

Punchlines, and What Comes Next

• A Bayesian answer is a distribution: the posterior, proportional to likelihood times prior.
• A conjugate prior keeps the posterior in the prior’s family. Beta–Binomial, Gamma–Poisson, and Normal–Normal each come from matching a kernel.
• The posterior mean is a weighted average of the prior mean and the data estimate, with weight \(n_0/(n_0 + n)\) on the prior.
• As \(n\) grows the data swamp the prior, and the posterior concentrates.
• A question with no formula, such as which of two rates is larger, is answered with draws from the posterior.

The frequentist asks how an estimate would vary over repeated samples. The Bayesian asks what the one sample in hand says about the unknown, and answers with a distribution. Both start from the same likelihood.