Estimation

From a Point Estimate to an Interval

You Only Ever See One Sample

You compute \(\hat\theta\) from the one dataset you have. A colleague with a different sample from the same population gets a different number. Neither of you is wrong.

The object this whole deck is about

The sampling distribution of \(\hat\theta\): the distribution of the numbers you would get by repeating the study. Every idea that follows is a statement about one of its features.

  • Judge an estimator: bias, variance, consistency
  • Build one: losses, maximum likelihood, GMM — and the bridges between them
  • Report it: efficiency, confidence intervals, p-values

A point estimate is one draw from that distribution. An interval reports its width.

Where the Distribution Sits

What Makes an Estimator “Unbiased”?

Intuition: imagine repeating your study many times, each time computing \(\hat\theta\) from a fresh sample.

  • If the average of all those estimates equals the true \(\theta\), the estimator is unbiased.
  • If it systematically overshoots or undershoots, it’s biased — like a bathroom scale that always reads 2 kg too heavy.

Formal Definition

\(\hat\theta\) is unbiased for \(\theta\) if \[\operatorname{E}[\hat\theta] = \theta \quad \text{for all } \theta \in \Theta\] Its bias is \(\operatorname{Bias}(\hat\theta) = \operatorname{E}[\hat\theta] - \theta\).

Unbiased or Biased? Some Familiar Examples

Unbiased

  • \(\bar{X}\) for the mean \(\mu\)
  • \(S^2 = \frac{1}{n-1}\sum (X_i - \bar{X})^2\) for \(\sigma^2\)

Biased

  • \(\hat{\sigma}^2 = \frac{1}{n}\sum (X_i - \bar{X})^2\)
  • Most MLEs (coming up), in finite samples

\(\operatorname{E}[\hat\sigma^2] = \frac{n-1}{n}\sigma^2\) — the bias vanishes as \(n\) grows, which is the first hint that unbiased and consistent are different promises.

Unbiasedness, Visualized

Every sample is drawn honestly from the true distribution — nothing is shifted. Both rows are handed the same data and differ only in the divisor.

The dotted rule is where each estimator is expected to land. Add repetitions and each solid average walks onto its own dotted rule — one of which is not \(\sigma^2\).

Pull the repetitions down and the averages become unreliable: unbiasedness is a statement about the long run, not about one study.

Bias vs. Variance: Two Ways to Be Wrong

Think of an estimator as arrows thrown at a target, where the bullseye is the true \(\theta\):

  • High bias: the arrows cluster tightly, but off to one side — consistently wrong in the same way
  • High variance: the arrows scatter widely around the bullseye — right on average, but unpredictable in any one sample
  • An ideal estimator has both low bias and low variance — but in practice we often have to trade one off against the other

Bias and Variance, Visualized

Rows differ in bias — whether the cloud is centred on the bullseye. Columns differ in variance — how tightly it is packed.

At one or two shots the four targets are indistinguishable. The pattern is a property of the distribution, not of any single estimate you will ever compute.

Keep in mind: each dot is an estimate, not a data point!

Mean Squared Error: Combining Both

Definition and Decomposition

\[\operatorname{MSE}(\hat{\theta}) = \underbrace{\operatorname{Var}(\hat{\theta})}_{\text{Variance}} + \underbrace{[\operatorname{Bias}(\hat{\theta})]^2}_{\text{Bias}^2}\]

We don’t only look for unbiased estimators: a slightly biased estimator with much lower variance can have lower MSE overall. Drag the slider to read the trade-off at each point.

Example: A Little Bias Can Pay Off

For \(X_1, \ldots, X_n \sim N(\mu, 1)\), compare two estimators of \(\mu\):

  • \(\hat{\mu}_1 = \bar{X}\) (unbiased)
  • \(\hat{\mu}_2 = c\bar{X}\) for some \(0 < c < 1\) (biased — “shrunk” toward 0)

Their mean squared errors are: \[\operatorname{MSE}(\hat{\mu}_1) = \frac{1}{n} \qquad\qquad \operatorname{MSE}(\hat{\mu}_2) = \frac{c^2}{n} + (1-c)^2\mu^2\]

Takeaway

When \(\mu\) is close to 0, the biased \(\hat\mu_2\) can have smaller MSE than the unbiased \(\bar{X}\). This is the idea behind shrinkage estimators like Ridge regression.

A Little Bias Can Pay Off, Visualized

Shaded: where the shrunk estimator has the lower MSE.

Shrinking hard (\(c\) small) wins big near \(\mu = 0\) and loses badly far from it. Raising \(n\) narrows the shaded band — with enough data there is little left to gain by biasing the estimate.

What More Data Buys

The Law of Large Numbers, One Path at a Time

Every property so far was stated for a fixed \(n\). Now let \(n\) grow: the running average settles down near the true \(\mu\).

The dashed funnel is \(\mu \pm \sigma/\sqrt n\) — the path is squeezed by arithmetic, not by luck. Hold on to that shape: by the end of this deck it will be an interval, drawn the other way round.

Consistency: Getting It Right Eventually

Unbiasedness is about being right on average for a given \(n\). Consistency instead asks: does the estimator get closer and closer to the truth as we collect more data?

Definition

\(\hat{\theta}_n\) is consistent if \(\hat{\theta}_n \xrightarrow{p} \theta\) as \(n \to \infty\), i.e. for every \(\epsilon > 0\): \[\lim_{n \to \infty} P\left(|\hat{\theta}_n - \theta| > \epsilon\right) = 0\]

An Easy Way to Check It

If both of these hold, \(\hat\theta_n\) is automatically consistent: \[\lim_{n \to \infty} \operatorname{Bias}(\hat{\theta}_n) = 0 \qquad \text{and} \qquad \lim_{n \to \infty} \operatorname{Var}(\hat{\theta}_n) = 0\]

Consistency in Pictures

The whole sampling distribution collapses onto \(\theta\); the faint curves are the smaller sample sizes this one grew out of.

Consistency is the shaded percentage going to 100 for every \(\epsilon\), however small. It says nothing about how fast — that question returns with efficiency, later in the deck.

Estimation as Minimizing a Loss

What Does “Best” Mean?

Every estimator answers the question which value fits the data best? A loss function is how you say what “best” means.

Loss function

\(L(y, c)\) is the cost of reporting \(c\) when the outcome is \(y\). How it charges for a miss is your choice, not a fact about the data.

The recipe

Report the value with the smallest average loss in your sample: \[\hat\theta \;=\; \arg\min_c\ \frac{1}{n}\sum_{i=1}^n L(y_i, c)\]

The sample average stands in for the population average \(\operatorname{E}[L(Y, c)]\). The law of large numbers from the last section is why that substitution works as \(n\) grows.

Which Loss Leads Where

Loss \(L(y, c)\) How a miss is charged Best single value \(c\) With regressors
Squared \(\ (y-c)^2\) quadratically — big misses dominate the mean least squares (conditional mean)
Absolute \(\ \lvert y-c\rvert\) linearly — every unit the same the median median (LAD) regression
Asymmetric \(\tau\) per unit too low, \(1-\tau\) per unit too high the \(\tau\)-quantile quantile regression
0–1 \(\ \mathbf{1}\{y \neq c\}\) every miss the same, however large the mode classification
Neg. log-likelihood \(\ -\ln f(y;\theta)\) by how surprising \(y\) is under a model the MLE maximum likelihood

Choose the loss, and you have chosen the estimator

The mean and the median answer different questions: the average income, or the typical one? The last row is the bridge to maximum likelihood.

Minimizing a Loss, Live

Top: 13 incomes and your candidate \(c\); the grey lines are the misses. Bottom: the average loss of reporting \(c\), for every \(c\). The green line is where it bottoms out.

Add the top earner: under squared loss the minimum jumps, under absolute loss it barely moves. Squared loss gives a far-away point a say proportional to its squared distance.

What the Choice of Loss Buys

  • Squared loss lets one large miss outweigh many small ones, so the mean chases outliers. It is the loss inside the MSE of the first section, and the one behind least squares
  • Absolute loss charges every unit of a miss the same, so the median is robust: one top earner cannot drag it
  • Asymmetric loss is for misses whose two directions cost differently: running out of stock vs. an unsold unit, a central bank under- vs. over-forecasting inflation

The same recipe with regressors

Replace the constant \(c\) by a prediction \(f(x_i;\theta)\) and minimize the same average loss over \(\theta\). Squared loss then fits the conditional mean \(\operatorname{E}[Y \mid X]\) — that is OLS. Absolute loss fits the conditional median.

Maximum Likelihood

The Idea: Make the Data Least Surprising

A loss needs no story about where the data came from. Maximum likelihood starts from one.

  • Write down a model: a distribution \(f(y;\theta)\) that says how probable each outcome is, for each value of \(\theta\)
  • For every candidate \(\theta\), ask: how probable would the data I actually saw have been?
  • Report the \(\theta\) under which that probability is largest

Maximum likelihood estimator

\[\hat\theta_{ML} \;=\; \arg\max_\theta \ \prod_{i=1}^n f(y_i;\theta) \;=\; \arg\max_\theta \ \sum_{i=1}^n \ln f(y_i;\theta)\] Taking logs turns a product of tiny numbers into a sum; the maximizer does not move.

Likelihood, Live

We model \(n\) workers’ log hourly wages as \(N(\mu, \sigma^2)\). Which \(\mu\) and \(\sigma\) make the wages we actually observed most probable?

Each observation contributes the height of the density at its value — the stems. The likelihood multiplies them, so one point the curve barely reaches (pink) drags the whole product down.

Move \(\mu\) and \(\sigma\) until the top picture matches the bottom one: that is maximum likelihood. Too narrow a density misses the outer points; too wide a one spreads itself thin over all of them.

Familiar Estimators Are MLEs

Data Model MLE Remark
yes / no answers Bernoulli\((p)\) \(\hat p = k/n\) the share of yes answers
counts (patents, doctor visits) Poisson\((\lambda)\) \(\hat\lambda = \bar y\) the sample mean
durations (unemployment spells) Exponential\((\lambda)\) \(\hat\lambda = 1/\bar y\) a rate: one over the mean
continuous outcomes (log wages) Normal\((\mu, \sigma^2)\) \(\hat\mu = \bar y\), \(\ \hat\sigma^2 = \frac1n\sum(y_i-\bar y)^2\) \(\hat\sigma^2\) is the biased “\(\div n\)” estimator from the first section — yet consistent

Not exotic — but no longer a guess

For simple models, MLE returns the formulas you would have guessed, together with a reason for them. For models where there is nothing obvious to guess — logit and probit, duration and count regressions — it is the default recipe.

What Maximum Likelihood Assumes

What you must assume

  • The right model: the data come from \(f(y;\theta)\) — the whole distribution, not just its mean
  • Independence, so the likelihood is a product
  • Identification: different \(\theta\), different distributions
  • Regularity: a smooth likelihood; the possible values of \(y\) do not depend on \(\theta\)

In return, as n grows

  • Consistent: \(\hat\theta \to \theta\)
  • Approximately normal around \(\theta\)
  • Efficient: the smallest possible variance — made precise later
  • Invariant: the MLE of \(g(\theta)\) is \(g(\hat\theta)\)

And if the model is wrong?

The MLE still converges — to the best-fitting wrong model. Sometimes harmless (the normal MLE of a mean is \(\bar y\), whatever the true distribution); sometimes exactly the number you did not want.

Building the Bridge

Two Roads to the Same Estimator

Big picture: a loss and a likelihood both answer the same question — which parameter values fit the data best? They just define “best” differently.

  • Least squares (LS): pick \(\theta\) that minimizes the total squared prediction error — a loss, no model of the errors needed
  • Maximum likelihood (MLE): pick \(\theta\) that makes the observed data most probable — needs a full model of the errors

Key Insight

Under the right assumptions, these two different ideas lead to the exact same estimator. The next slide shows why.

From Maximum Likelihood to Least Squares

Suppose \(Y = f(X;\theta) + \epsilon\) with \(\epsilon \sim N(0, \sigma^2)\).

The likelihood of the data is a product of normal densities: \[\text{MLE} = \arg\max_\theta \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(y_i - f(x_i;\theta))^2}{2\sigma^2}}\]

Taking logs and dropping the parts that don’t involve \(\theta\)… \[\text{MLE} = \arg\min_\theta \sum_{i=1}^n (y_i - f(x_i; \theta))^2 = \textbf{Least Squares}\]

Takeaway

If the errors are normal, least squares is maximum likelihood — and inherits every guarantee on the MLE’s list, efficiency included.

Every Likelihood Is a Loss

Maximizing \(\sum \ln f\) is minimizing \(\sum -\ln f\). So every error distribution you assume hands you a loss:

If you assume the error is… then \(-\ln f\) is, up to constants… a loss called and the MLE is
Normal \((y - f(x;\theta))^2 / 2\sigma^2\) squared least squares
Laplace (fatter tails) \(\lvert y - f(x;\theta)\rvert \,/\, b\) absolute median (LAD) regression
Bernoulli, \(P(y = 1) = p(x;\theta)\) \(-y\ln p - (1-y)\ln(1-p)\) log loss logit / probit

Read it left to right

Assuming a distribution chooses your loss: MLE is loss minimization with a loss derived from a model.

Read it right to left

Choosing a loss quietly assumes a distribution. Squared loss suits roughly normal errors — and pays for it when tails are fat.

Generalized Method of Moments

From Method of Moments to GMM

A third road. Instead of a loss or a likelihood, write down facts that should hold on average at the true parameter, like \(\operatorname{E}[Y] = \mu\). Classical method of moments: one such equation per parameter, solved exactly.

But what if you have more valid equations than parameters? You can’t satisfy all of them exactly at once, so instead you get as close as possible to satisfying all of them simultaneously.

This is the idea behind GMM

  • Classical MM: \(p\) parameters, \(p\) moment conditions (exactly identified)
  • GMM: \(p\) parameters, \(q \geq p\) moment conditions (over-identified)

The GMM Estimator

Suppose economic theory implies a set of conditions that should hold on average at the true parameter value:

\[\operatorname{E}[g(X_i, \theta)] = 0, \qquad g: \mathbb{R}^d \times \mathbb{R}^p \to \mathbb{R}^q\]

Their sample counterparts will rarely hit zero exactly, so we make them as close to zero as possible:

\[\bar{g}_n(\theta) = \frac{1}{n}\sum_{i=1}^n g(X_i, \theta)\]

GMM Estimator

\[\hat{\theta}_{GMM} = \arg\min_\theta \; \bar{g}_n(\theta)'W_n\bar{g}_n(\theta)\] where \(W_n\) is a \(q \times q\) weight matrix that decides how much each condition counts.

How GMM Relates to the Others

Minimize a loss or maximize a likelihood, and the optimum sits where the derivative is zero. Averaged over the sample, that first-order condition is a moment condition.

Estimator Built from Its moment condition
Mean squared loss \(\operatorname{E}[Y - \mu] = 0\)
Median absolute loss \(P(Y \le m) - \tfrac12 = 0\)
OLS squared loss, with regressors \(\operatorname{E}[X(Y - X'\beta)] = 0\)
MLE a full distribution \(\operatorname{E}\big[\tfrac{\partial}{\partial\theta}\ln f(Y;\theta)\big] = 0\)
IV nothing but the moment itself \(\operatorname{E}[Z(Y - X'\beta)] = 0\)

So GMM is the umbrella

Every estimator in this deck is GMM with a particular choice of moments. What differs is how much you have to believe: MLE needs the whole distribution, GMM only the moments you write down. The last row has no loss or likelihood behind it at all — that is where GMM earns its keep.

Choosing the Weights

Not all moment conditions are equally reliable — noisier ones should count for less.

Two-Step Procedure

  1. Start with equal weights (\(W = I\)) to get a first-pass estimate \(\tilde\theta\)
  2. Use \(\tilde\theta\) to see how noisy and correlated the moment conditions are
  3. Re-estimate using weights that downweight the noisier conditions

With the optimal weight matrix, GMM achieves the smallest possible variance among all choices of \(W\) — an efficiency result, the theme of the next section.

Example: Instrumental Variables Regression

Model: \(y = X\beta + \epsilon\), but \(\operatorname{E}[X'\epsilon] \neq 0\) — \(X\) is endogenous, so OLS is biased.

Fix: find instruments \(Z\) that are correlated with \(X\) but uncorrelated with \(\epsilon\): \[\operatorname{E}[Z'\epsilon] = 0 \quad \Longrightarrow \quad g(y,X,Z;\beta) = Z'(y - X\beta)\]

GMM / IV Estimator

\[\hat{\beta}_{GMM} = (X'ZW_nZ'X)^{-1}X'ZW_nZ'y\]

OLS Bias vs. IV Consistency, Visualized

A live simulation with \(x = 0.9z + u\) and \(y = 2x + u + \varepsilon\): the confounder \(u\) sits in both, so \(x\) is endogenous.

Both lines are noisy at small \(n\) and both settle down — but they settle on different numbers. More data does not cure a biased estimator; it only pins down the wrong answer more precisely.

GMM Shows Up All Over Economics

  • Instrumental variables & 2SLS: fixing endogeneity in regression (the last two slides)
  • Rational expectations models: Euler equations linking today’s consumption choices to future returns
  • Asset pricing (CAPM): relating an asset’s expected return to its exposure to market risk
  • In each case, economic theory supplies the moment conditions — GMM turns them into an estimator, without requiring a full distributional assumption

Model Testing and GMM’s Family Tree

J-Test for Over-Identification

When \(q > p\), the extra moment conditions can be used to test the model itself: \[J = n \cdot \bar{g}_n(\hat{\theta})'W\bar{g}_n(\hat{\theta}) \xrightarrow{d} \chi^2_{q-p}\] A large \(J\) suggests some moment condition — and so the model — may be wrong.

Tests get their own lecture next. The end of this deck shows why every test is also an interval.

When (Not) to Reach for GMM

Good fit when…

  • You have more moment conditions than parameters
  • There’s an endogeneity problem
  • The model is nonlinear
  • You don’t want to assume a full distribution

Watch out for…

  • GMM needs fairly large samples to behave well
  • Results can be sensitive to which moments you pick
  • Less efficient than a correct MLE: the price of assuming less

How Precise Is the Estimate?

That Variance Has a Name

Every sampling distribution in this deck had a width. For the sample mean it is \(\operatorname{Var}(\bar X) = \sigma^2/n\), and its square root is the number you report.

Standard deviation

\[\operatorname{SD}(X_i) = \sqrt{\operatorname{Var}(X_i)}\]

A property of the population. It does not shrink with \(n\); a larger sample only estimates it better.

Standard error

\[\operatorname{SE}(T) = \sqrt{\operatorname{\widehat{Var}}(T)}\]

A property of the estimator \(T = T(X_1,\dots,X_n)\) — the width of the sampling distribution we have been drawing all along.

Every interval from here is built from the second one

\(\bar x \pm 1.96\,s\) says where the data lie; \(\bar x \pm 1.96\,s/\sqrt{n}\) says where the mean lies. The rest of this deck is about the second: how small it can get, and how to report it.

What Drives Estimation Uncertainty

For the sample mean, \(\operatorname{SE} = \sigma/\sqrt n\). The same three levers set the standard error of every estimator in this deck:

  • Noise in the data, \(\sigma\): noisier outcomes, a wider sampling distribution
  • Amount of data, \(\sqrt n\): the SE shrinks with the square root of \(n\) — four times the data to halve it
  • The estimator itself: two estimators on the same data can have different SEs. How much of the data’s information an estimator puts to use is its efficiency

And the shape is (almost) always the same

In large samples the mean, least squares, MLE and GMM are all approximately normal around \(\theta\) — under their assumptions — with that SE as their spread. Every interval at the end of this deck is built on this one fact.

Efficiency: How Low Can the Variance Go?

Among all unbiased estimators, some are more precise than others. Efficiency asks: is \(\hat\theta\) the most precise unbiased estimator possible given some sample size?

It turns out there is a hard floor on how small the variance of an unbiased estimator can ever be — no amount of cleverness can beat it.

That floor is the Cramér-Rao Lower Bound.

The Cramér-Rao Lower Bound

Fisher Information

\[I(\theta) = -\operatorname{E}\left[\frac{\partial^2 \ln f(X;\theta)}{\partial \theta^2}\right]\] Roughly: how sharply peaked the likelihood is — how fast the log-likelihood in Likelihood, Live falls as you move away from the best fit. More information means the data pin down \(\theta\) more precisely.

Cramér-Rao Lower Bound (CRLB)

For any unbiased estimator \(\hat\theta\) based on \(n\) observations: \[\operatorname{Var}(\hat{\theta}) \geq \frac{1}{nI(\theta)}\] An unbiased estimator that achieves this bound is called efficient. The MLE reaches it as \(n\) grows.

Example: The Sample Mean Is Efficient

For \(X_i \sim N(\mu, \sigma^2)\), estimating \(\mu\):

  • Fisher information: \(I(\mu) = 1/\sigma^2\)
  • Cramér-Rao bound: \(\text{CRLB} = \sigma^2/n\)
  • Actual variance: \(\operatorname{Var}(\bar{X}) = \sigma^2/n\)

Conclusion

\(\operatorname{Var}(\bar{X})\) exactly meets the CRLB, so \(\bar{X}\) is an efficient estimator of \(\mu\) — no unbiased estimator can do better.

Efficiency, Visualized

\(\operatorname{Var}(\bar X)\) sits exactly on the CRLB floor; the shaded region below it is empty for every unbiased estimator, at every \(n\).

The median is unbiased here too — the normal is symmetric — but it ignores how far each point lies from the centre. In large samples its variance is \(\pi/2 \approx 1.57\) times the floor: it needs 57% more data for the same precision.

Mean or Median? Efficiency, Live

Each dot is one estimate from a fresh sample; the truth is 0. Most points come from \(N(0,1)\), a share of “wild” ones from a distribution ten times as wide. Both are centred on 0, so both estimators stay unbiased.

On clean data the mean wins, by roughly the \(\pi/2\) of the last slide. A few percent of wild points flips the ranking: the mean inherits their variance, the median barely notices. Efficiency is a property of an estimator and the data it meets.

Efficiency Is Bought with Assumptions

  • MLE reaches the Cramér-Rao floor as \(n\) grows — if the distribution you assumed is the true one
  • Least squares is the best linear unbiased estimator (Gauss-Markov), and fully efficient when errors are normal, because then it is the MLE
  • GMM makes the best use of the moments you gave it, and ignores everything else — so a correct MLE beats it
  • The median loses to the mean on normal data and beats it when tails are fat

The trade-off behind every choice in this deck

The more you assume, the more precise you can be — if you are right. Assume less, and you trade precision for robustness.

What is the variance for?

A variance nobody reports is useless. Turning the standard error into a statement on a page is the job of the confidence interval.

From Spread to Interval

From a Point to a Range

A point estimate says what is our best guess, never how good the guess is — the question a policy note has to survive.

Confidence interval

An interval estimator \([\hat\theta_L, \hat\theta_U]\) has confidence level \(1-\alpha\) if its coverage probability \(P_\theta\big(\hat\theta_L \le \theta \le \hat\theta_U\big)\) is \(1-\alpha\), whatever the true \(\theta\).

The workhorse

The estimator is approximately normal, so \(\hat\theta \pm 1.96 \times \operatorname{SE}(\hat\theta)\) covers \(\theta\) in about 95% of samples. A small-sample mean with \(\sigma\) estimated by \(s\) uses \(t_{n-1}\) instead of 1.96.

Read the probability carefully

\(\hat\theta_L\) and \(\hat\theta_U\) are the random variables. \(\theta\) is a fixed unknown constant. The probability is over intervals, not over \(\theta\).

Coverage, Live

Each horizontal line is one 95% interval from one sample — the same repeated sampling as the dot strips earlier, with a bar instead of a dot. The vertical rule is the true \(\mu\), which in real life you never see.

95% is a promise about the procedure, not about the interval on your screen. Break an assumption — a skewed population at small \(n\), a rare binary event, or \(z\) where \(t\) belongs — and the promise quietly stops being kept.

What the Interval Plot Shows

  • With the \(t\) interval on a normal population, the running coverage settles on the nominal level — the procedure keeps its promise
  • A skewed population at \(n = 10\) under-covers: the interval is symmetric, the sampling distribution is not
  • Bernoulli with \(p = 0.05\) is worse still, and the intervals visibly extend below zero
  • Substituting \(z\) for \(t\) at small \(n\) under-covers — which is precisely why \(t\) exists

The point

Nominal coverage is a property of the procedure under its assumptions. The label keeps saying 95% also if the assumptions are violated.

The scatter of the dots in Unbiasedness, Visualized and the length of these bars are the same quantity: one standard error, times a critical value.

What a Confidence Interval Is Not

Correct interpretation

  • 95% of intervals built this way contain \(\theta\)
  • The procedure fails 5% of the time, in the long run
  • Wider interval \(\Rightarrow\) less precise estimate

A confidence interval is NOT

  • \(P(\theta \in [\hat\theta_L, \hat\theta_U] \mid \text{data}) = 0.95\)
  • A range containing 95% of the data
  • A range containing 95% of future estimates
  • “They overlap, so the difference is not significant”

Once you have seen the data, the interval either contains \(\theta\) or it does not. Nothing random is left to attach a probability to.

Confidence Interval \(\neq\) Prediction Interval

Two intervals that look identical on a chart and answer different questions.

Confidence interval — for the mean

\[\bar x \pm z\,\frac{\sigma}{\sqrt n} \;\xrightarrow[n\to\infty]{}\; \{\mu\}\]

Width \(\to 0\). With enough data you know \(\mu\) exactly.

Prediction interval — for the next draw

\[\bar x \pm z\,\sigma\sqrt{1 + \tfrac1n} \;\xrightarrow[n\to\infty]{}\; \mu \pm z\sigma\]

Width \(\to 2z\sigma\). It stops.

The distinction economists get wrong

Forecast fan charts are prediction intervals. Reading one as a confidence interval understates forecast uncertainty by a factor that grows with \(\sqrt n\). The left-hand limit is the consistency picture from earlier; the right-hand one cannot collapse, because the next observation is not an estimator of anything.

p-Values and Intervals

What a p-Value Measures

Turn the question around. Instead of which values of \(\theta\) are plausible?, ask about one value \(\theta_0\) — say “the reform had no effect”, \(\theta_0 = 0\).

p-value, in words

If \(\theta\) really were \(\theta_0\), how often would a fresh sample give an estimate at least as far from \(\theta_0\) as the one we got? \[p(\theta_0) \;=\; P_{\theta_0}\big(\,\lvert\hat\theta - \theta_0\rvert \;\ge\; \lvert\hat\theta_{\text{obs}} - \theta_0\rvert\,\big)\] The sampling distribution again — this time centred at \(\theta_0\).

Small p

The data would be surprising if \(\theta\) were \(\theta_0\): evidence against it. Below a threshold \(\alpha\), often 5%, we reject \(\theta_0\).

Not

The probability that \(\theta_0\) is true — \(\theta_0\) is either true or not. The next lecture treats tests and p-values in full.

A Confidence Interval Is an Inverted Test

Hypothesis testing, the next lecture, builds rejection regions; this deck builds intervals. They are the same object, seen from two sides.

Duality

\[\text{CI}_{1-\alpha}(x) \;=\; \big\{\,\theta_0 \;:\; \text{the level-}\alpha\text{ test of } H_0{:}\ \theta = \theta_0 \text{ does not reject}\,\big\}\]

Equivalently, writing \(p(\theta_0)\) for the p-value of the test of \(H_0: \theta = \theta_0\),

\[\text{CI}_{1-\alpha}(x) \;=\; \{\,\theta_0 : p(\theta_0) > \alpha\,\}\]

Read that as a picture

The confidence interval is the horizontal slice of the p-value curve at height \(\alpha\).

The p-Value Curve, Live

Drag \(\theta_0\) and watch the test on the left agree with the interval on the right — every time.

Left: the null distribution centred at the current \(\theta_0\), with the observed \(\bar x\) and its tail area.

Right: \(p(\theta_0)\) for every \(\theta_0\). Where the curve crosses \(\alpha\) are exactly the interval endpoints.

What the Duality Buys You

  • You never need two sets of tables. “Is \(\theta_0\) in the interval?” and “is \(p < \alpha\)?” are one question
  • Lower \(\alpha\) and the slice moves down, so the interval widens — the same trade-off a rejection region draws
  • The interval says strictly more than the test: it also reports every value the data cannot rule out

Why this matters for reporting

“Not significant” collapses an interval to one bit. An interval of \([-0.01,\ 0.42]\) and an interval of \([-0.20,\ 0.21]\) tell very different stories, and the p-value is nearly the same for both.

When the Formula Runs Out

The Luxury You Only Have in a Lecture

Every simulation so far drew fresh samples from a known truth — a luxury you have exactly once, in a lecture.

You cannot do that with real data

One sample, no population to redraw from. The way out so far was a formula for the standard error; these have none:

  • The Gini coefficient, the 90–10 ratio, top income shares
  • Elasticities, ratios of estimates, differences of quantiles
  • Anything computed by a multi-step procedure

The bootstrap principle

The sample is to the population as a resample is to the sample. Draw \(B\) samples of size \(n\) with replacement from the data, recompute \(\hat\theta^*\) on each, and use the spread of \(\hat\theta^*\) as the sampling distribution of \(\hat\theta\).

The Bootstrap, Live

Left: the sample. Dot size is how often each observation was drawn in one replicate; greyed-out points were not drawn at all.

Choose Maximum to watch the method fail: a resample misses the largest observation with probability \((1-1/n)^n \to e^{-1}\).

What the Bootstrap Can and Cannot Do

  • For the mean, the bootstrap standard error reproduces \(s/\sqrt n\) to two decimals — which is what earns it trust
  • For the Gini, it produces an interval where no textbook formula exists at all
  • For the maximum it collapses: a resample misses the largest observation with probability \((1-1/n)^n \to e^{-1} \approx 0.368\), so roughly 63% of replicates return exactly the sample max

\(B\) is not \(n\)

Raising \(B\) shrinks Monte-Carlo noise in the endpoints. It does not shrink the interval, and it adds no information.

It cannot repair the sample

Resampling a biased sample gives biased resamples. And i.i.d. resampling destroys dependence — clustered or serial data need a block bootstrap.

Summary

Comparing the Estimation Methods

Least squares Median / LAD MLE GMM
Built from squared loss absolute loss a full distribution moment conditions
Needs a full distribution ❌ ❌ ✅ ❌
Robust to outliers ❌ ✅ depends on model depends on moments
Efficient when… errors normal errors Laplace the model is right best weights, given its moments

Rules of Thumb

  • Quick and simple: method of moments
  • The average outcome, no distribution needed: least squares
  • The typical outcome, or outliers in the data: median / LAD
  • You know the distribution: MLE
  • Endogeneity or extra moment conditions: GMM

Key Takeaways — The Estimator

  1. An estimator is judged by its sampling distribution: where it sits (bias), how wide it is (variance), and whether it closes in on \(\theta\) (consistency)
  2. MSE = variance + bias² — which is why a little bias can be worth buying
  3. Choose a loss, get an estimator: squared → mean, absolute → median, asymmetric → quantile
  4. MLE is loss minimization with the loss derived from a model; with normal errors it is least squares
  5. GMM needs only moment conditions — and every estimator above is a special case

The through-line

Every one of these is a statement about the same distribution — the one you never observe, because you only ever draw one sample from it.

Key Takeaways — Uncertainty

  1. The standard error is the width of the sampling distribution; it shrinks like \(1/\sqrt n\)
  2. Efficiency is how little of the data’s information an estimator wastes — bought with assumptions, lost when they fail
  3. A confidence interval is a statement about the procedure; coverage is kept only while its assumptions hold
  4. A p-value asks how surprising the data are under one \(\theta_0\); the interval is every \(\theta_0\) that is not surprising
  5. The bootstrap covers statistics with no formula — and fails visibly at the edge of the sample

Choosing an interval

  • Large sample — \(\hat\theta \pm 1.96\,\operatorname{SE}\)
  • Mean, small sample, unknown \(\sigma\) — \(t\) interval
  • Smooth function of estimates — delta method
  • No formula — bootstrap

Report the interval

It carries the effect size, the precision, and every value the data cannot rule out. A p-value carries one bit of that.

What This Deck Left Out

Bayesian methods

Priors and posteriors · credible intervals, the probability statement a CI is not · hierarchical models

Finite-sample optimality

Sufficiency · Rao-Blackwell · the UMVUE, the best unbiased estimator

Robust and modern estimation

M-estimators, the general form of the loss section · LASSO and Ridge · machine learning and causal inference

And the interval kind

Pivots and exact intervals — \(\chi^2\) for a variance, Wilson for a proportion · sample-size planning · the delta method · cluster bootstraps