Today’s Big Question

Can we compress \(n\) data points into a simpler summary without losing any information about \(\theta\)?

When is this possible? And how do we find the best compression?

Motivating Example: Bernoulli

\(X_1, \ldots, X_n \stackrel{\text{iid}}{\sim} \text{Bernoulli}(p)\), where \(p\) is unknown.

The data: A sequence of 0s and 1s, e.g., \((1, 0, 1, 1, 0, 0, 1, 0, 1, 1)\)

Claim: \(T = \sum_{i=1}^n X_i\) tells us everything about \(p\).

Why? Once you know the total number of successes, the specific order of 0s and 1s is just noise — it carries no additional information about \(p\).

Motivating Example: Normal

\(X_1, \ldots, X_n \stackrel{\text{iid}}{\sim} N(\mu, \sigma^2)\), with \(\sigma^2\) known.

Claim: \(\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i\) captures all information about \(\mu\).

• The full dataset has \(n\) numbers.
• \(\bar{X}\) is just one number.
• Yet nothing about \(\mu\) is lost!

But wait — if both \(\mu\) and \(\sigma^2\) are unknown, we need more:

\[T = \left(\sum_{i=1}^n X_i, \;\sum_{i=1}^n X_i^2\right)\]

Two numbers for two parameters. This is not a coincidence.

Sufficient Statistic: Definition

The Factorization Theorem (Fisher–Neyman)

This is the workhorse of sufficiency. You will use this on every homework and exam.

Example: Bernoulli

\(X_1, \ldots, X_n \stackrel{\text{iid}}{\sim} \text{Bernoulli}(p)\).

Setup:

\[f(\mathbf{x} \mid p) = \prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} = p^{\sum x_i}(1-p)^{n - \sum x_i}\]

Your turn: Factor this as \(g(T(\mathbf{x}), p) \cdot h(\mathbf{x})\).

Example: Normal (Both Parameters Unknown)

\(X_1, \ldots, X_n \stackrel{\text{iid}}{\sim} N(\mu, \sigma^2)\), both \(\mu\) and \(\sigma^2\) unknown.

Setup:

\[f(\mathbf{x} \mid \mu, \sigma^2) = \left(\frac{1}{2\pi\sigma^2}\right)^{n/2} \exp\left\{-\frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \mu)^2\right\}\]

Goal: Factor and identify the sufficient statistic.

Hint: Expand \((x_i - \mu)^2 = x_i^2 - 2\mu x_i + \mu^2\).

Example: Poisson

\(X_1, \ldots, X_n \stackrel{\text{iid}}{\sim} \text{Poisson}(\lambda)\).

\[f(\mathbf{x} \mid \lambda) = \prod_{i=1}^n \frac{e^{-\lambda}\lambda^{x_i}}{x_i!} = \underbrace{\frac{e^{-n\lambda}\lambda^{\sum x_i}}{}}_{g(\sum x_i,\; \lambda)} \cdot \underbrace{\frac{1}{\prod_{i=1}^n x_i!}}_{h(\mathbf{x})}\]

\(T = \sum_{i=1}^n X_i\) is sufficient for \(\lambda\).

The Uniform Example

\(X_1, \ldots, X_n \stackrel{\text{iid}}{\sim} \text{Uniform}(0, \theta)\), where \(\theta > 0\).

• Not \(\bar{X}\)! Not \(\sum X_i\)! The maximum is sufficient.
• The indicator \(\mathbf{1}(x_{(n)} \leq \theta)\) is the key — it depends on \(\theta\) and on \(\mathbf{x}\) only through \(x_{(n)}\).

Order Statistics — Quick Review

Why here? We need order statistics because the Uniform sufficient statistic is \(X_{(n)}\), not a simple sum.

Recall from Module 1: The pdf of \(X_{(n)}\) for Uniform\((0,\theta)\) is:

\[f_{X_{(n)}}(t) = \frac{n\, t^{n-1}}{\theta^n}, \quad 0 \leq t \leq \theta\]

Discussion: Why Not \(\bar{X}\) for the Uniform?

Consider two datasets with the same mean but different maxima:

Same \(\bar{X}\), very different conclusions about \(\theta\)!

Sufficiency is Not Unique

Observation: If \(T\) is sufficient for \(\theta\), then so is any one-to-one function of \(T\).

More importantly: The entire sample \(\mathbf{X} = (X_1, \ldots, X_n)\) is always sufficient.

Proof: \(f(\mathbf{x} \mid \theta) = f(\mathbf{x} \mid \theta) \cdot 1\). Take \(T(\mathbf{x}) = \mathbf{x}\), \(g = f\), \(h = 1\).

But that is no compression at all!

We want the MOST compressed sufficient statistic — the one that throws away the most while keeping everything about \(\theta\).

This is minimal sufficiency.

Minimal Sufficient Statistic

Intuition: \(T\) achieves maximum data compression while preserving all information about \(\theta\).

• It has the coarsest partition of the sample space among all sufficient statistics.
• Any further compression would lose information.

Minimal Sufficiency Examples

Midpoint Summary

What we’ve established (Lecture 4):

The Uniform\((0,\theta)\) story so far:

• \(T = X_{(n)}\) is sufficient (Factorization Theorem)
• \(T = X_{(n)}\) is minimal sufficient (Lehmann–Scheffe)

Completeness

In words: The only function of \(T\) with mean zero everywhere is the zero function (a.s.).

Intuition: \(T\) has “no redundant information.” There is no nontrivial unbiased estimator of zero based on \(T\) alone.

Exponential Families and Completeness

Full rank means the natural parameter space \(\{\eta(\theta) : \theta \in \Theta\}\) contains an open set in \(\mathbb{R}^k\).

This immediately gives completeness for: Bernoulli, Poisson, Normal, Exponential, Gamma, Beta, …

Completeness Examples

The Uniform is NOT an exponential family (the support depends on \(\theta\)).

Completeness of \(X_{(n)}\) must be verified directly.

Basu’s Theorem

Ancillary = distribution-free of \(\theta\) (e.g., \(X_i - \bar{X}\) for location families).

Why this matters: Independence results are usually hard to prove. Basu gives them for free!

Example: For \(X_1, \ldots, X_n \stackrel{\text{iid}}{\sim} N(\mu, \sigma^2)\) with \(\sigma^2\) known, \(\bar{X}\) is complete sufficient for \(\mu\), and the sample range \(R = X_{(n)} - X_{(1)}\) is ancillary. By Basu’s theorem, \(\bar{X} \perp R\) — no calculation needed!

Rao–Blackwell: The Uniform Running Example

\(X_1, \ldots, X_n \stackrel{\text{iid}}{\sim} \text{Uniform}(0, \theta)\). Estimate \(\theta\).

Step 1: Start with a crude unbiased estimator.

\[\delta = 2X_1\]

Check: \(E[2X_1] = 2 \cdot \theta/2 = \theta\). Unbiased!

Step 2: Rao–Blackwellize. Condition on \(T = X_{(n)}\) (sufficient).

\[\delta^* = E[2X_1 \mid X_{(n)}]\]

Step 3: Compute.

\[\delta^* = \frac{n+1}{n}\, X_{(n)}\]

Variance Comparison

The Full Uniform Story

Summary and Looking Ahead

Module 3 — The Data Reduction Toolkit: