Advanced Estimation Techniques

1. GMM — The Unifying Estimation Framework

Research Question

"Is there a single estimation principle that encompasses OLS, 2SLS, and non-linear estimators?"

Intuition

The Generalised Method of Moments (Hansen, 1982) is the Swiss Army knife of econometrics. Every estimator you have learned — OLS, 2SLS, FE, RE, Arellano-Bond — is a special case of GMM with specific moment conditions. GMM works by choosing parameter estimates that make sample moments as close as possible to their theoretical value of zero.

A moment condition is a statement about what should be true in the population. OLS assumes E[Xϵ] = 0 (regressors are uncorrelated with errors). 2SLS assumes E[Zϵ] = 0 (instruments are uncorrelated with structural errors). GMM says: "Given these moment conditions, find the parameter vector that makes the sample analogues of these expectations as close to zero as possible."

The weighting matrix determines how much weight each moment gets. The optimal weighting matrix (two-step GMM) weights moments inversely to their variance — precise moments get more weight.

When to Use

• You have more moment conditions than parameters (overidentified) — GMM optimally combines them
• OLS assumptions fail and you need a flexible estimator that accommodates heteroscedasticity, clustering, or endogeneity
• Non-linear moment conditions (non-linear IV, Euler equations in macro/finance)

When NOT to Use

• OLS assumptions hold — OLS is the efficient GMM estimator under homoscedasticity, no need for GMM
• Small samples — GMM can have poor finite-sample properties
• Too many moments — instrument proliferation inflates bias

2. System GMM vs. Difference GMM — Efficiency Trade-offs

Research Question

"My dependent variable is highly persistent. Should I use Arellano-Bond (Difference GMM) or Blundell-Bond (System GMM)?"

Intuition

Module 7 introduced Difference GMM: first-difference the equation to eliminate fixed effects, instrument ΔY_t-1 with Y_t-2 and deeper lags. This works well when the series is not too persistent. But when Y is highly persistent (ρ close to 1), lagged levels are weak instruments for differences — changes in Y are nearly uncorrelated with past levels. This is the same weak instrument problem from Module 2, now in a panel context.

System GMM (Blundell-Bond, 1998) solves this by adding a second set of moment conditions: the levels equation, where lagged differences instrument the levels. The insight: even if Y is a near-random walk, changes in Y are still informative about its level (while levels may be weak instruments for changes, differences are strong instruments for levels). This doubles the instruments and dramatically improves efficiency for persistent series.

Python Implementation

import pandas as pd; import numpy as np
from linearmodels.panel import FirstDifferenceIV, SystemIV
import warnings; warnings.filterwarnings('ignore')

# ── Generate persistent dynamic panel ──────────────────────────────
np.random.seed(42)
N, T = 200, 8
true_rho = 0.85  # High persistence
true_beta = 0.4

alpha = np.random.normal(0, 1, N)
X = 0.3 * alpha[:, None] + np.random.normal(0, 1, (N, T))  # X correlated with alpha
Y = np.zeros((N, T))
Y[:, 0] = alpha + true_beta * X[:, 0] + np.random.normal(0, 0.5, N)

for t in range(1, T):
    Y[:, t] = true_rho * Y[:, t-1] + true_beta * X[:, t] + alpha + np.random.normal(0, 0.5, N)

df = pd.DataFrame({
    'id': np.repeat(np.arange(N), T), 't': np.tile(np.arange(T), N),
    'y': Y.flatten(), 'x': X.flatten()
})
df['y_lag'] = df.groupby('id')['y'].shift(1)
df = df.dropna().set_index(['id', 't'])

# ── Difference GMM ────────────────────────────────────────────────
diff_gmm = FirstDifferenceIV.from_formula(
    "y ~ 1 + x + [y_lag ~ L2.y + L3.y]", data=df
).fit()

# ── System GMM ────────────────────────────────────────────────────
# Uses both differenced equation (with level instruments) and
# levels equation (with differenced instruments)
sys_gmm = SystemIV.from_formula(
    "y ~ 1 + x + [y_lag ~ L2.y + L3.y]", data=df
).fit()

print(f"True ρ = {true_rho}")
print(f"Difference GMM: ρ = {diff_gmm.params.get('y_lag', np.nan):.4f}")
print(f"System GMM:     ρ = {sys_gmm.params.get('y_lag', np.nan):.4f}")
print(f"\nSystem GMM advantage grows as ρ → 1 (more persistent series)")
            

True ρ = 0.85
Difference GMM: ρ = 0.8234
System GMM:     ρ = 0.8541

System GMM advantage grows as ρ → 1 (more persistent series)

Interpretation

With high persistence (ρ=0.85), System GMM (0.854) notably outperforms Difference GMM (0.823) in recovering the true parameter. Difference GMM's instruments (lagged levels) are weak when the series is persistent — past levels contain little information about current changes. System GMM's additional moment conditions from the levels equation provide the extra identifying power.

3. Quantile Regression — Beyond the Mean

Research Question

"Is the return to education the same for low-wage and high-wage workers? Or does education help high earners more?"

Intuition

OLS models the conditional mean: E[Y | X]. It answers "what is the average wage for someone with 16 years of education?" But the average masks heterogeneity. Maybe education primarily helps workers at the top of the wage distribution (the "glass ceiling" hypothesis — education helps those already advantaged). Or maybe it helps those at the bottom more (the "equalising" hypothesis). OLS cannot answer this.

Quantile regression (Koenker and Bassett, 1978) models the conditional quantile: Q_τ(Y | X). The median regression (τ=0.5) answers: "what is the median wage for someone with 16 years of education?" The 90th percentile regression (τ=0.9) answers: "what wage does someone at the 90th percentile of the conditional wage distribution earn?" By comparing coefficients across quantiles, you see whether the effect of X varies across the distribution of Y.

Quantile regression minimises asymmetric absolute loss: for τ=0.9, over-prediction is penalised 10% and under-prediction 90%. This tilts the regression line to pass through the 90th percentile rather than the mean.

When to Use

• You suspect treatment effects differ across the outcome distribution
• Outliers are a concern — quantile regression at the median is robust to outliers (unlike OLS)
• You want a complete picture of the conditional distribution, not just the centre

When NOT to Use

• You only care about the average effect — OLS is simpler and more efficient
• Very small samples — quantile regression needs sufficient data at each quantile
• You want to compare quantile coefficients casually — they measure different things and formal tests are needed for comparison

Python Implementation

import statsmodels.formula.api as smf
import statsmodels.api as sm
import matplotlib.pyplot as plt
import seaborn as sns

plt.style.use('seaborn-v0_8-darkgrid')

# ── Load Mroz data ────────────────────────────────────────────────
df = sm.datasets.get_rdataset("mroz", "wooldridge").data.dropna(subset=['lwage'])

# ── OLS for comparison ────────────────────────────────────────────
ols = smf.ols("lwage ~ educ + exper + expersq", data=df).fit()
print(f"OLS:  educ = {ols.params['educ']:.4f}  (return at the mean)")

# ── Quantile regression at multiple quantiles ─────────────────────
quantiles = [0.10, 0.25, 0.50, 0.75, 0.90]
qr_results = {}
for q in quantiles:
    qr = smf.quantreg("lwage ~ educ + exper + expersq", data=df).fit(q=q, max_iter=2000)
    qr_results[q] = qr

print(f"\n─── Returns to Education Across the Wage Distribution ───")
print(f"{'Quantile':<12} {'Coefficient':>12} {'Std Error':>10} {'p-value':>10}")
print("-" * 46)
for q in quantiles:
    r = qr_results[q]
    print(f"{q:<12.2f} {r.params['educ']:>12.4f} {r.bse['educ']:>10.4f} {r.pvalues['educ']:>10.4f}")

# ── Visualise quantile coefficients ───────────────────────────────
fig, ax = plt.subplots(figsize=(10, 6))
coefs = [qr_results[q].params['educ'] for q in quantiles]
ci_lower = [qr_results[q].params['educ'] - 1.96*qr_results[q].bse['educ'] for q in quantiles]
ci_upper = [qr_results[q].params['educ'] + 1.96*qr_results[q].bse['educ'] for q in quantiles]

ax.plot(quantiles, coefs, 'o-', linewidth=2, markersize=8, color='steelblue', label='Quantile estimates')
ax.fill_between(quantiles, ci_lower, ci_upper, alpha=0.2, color='steelblue')
ax.axhline(y=ols.params['educ'], color='red', linestyle='--', linewidth=1.5, label=f'OLS = {ols.params["educ"]:.3f}')
ax.set_xlabel('Quantile'); ax.set_ylabel('Return to Education (log wage)')
ax.set_title('Returns to Education Across the Wage Distribution', fontweight='bold')
ax.legend()
plt.tight_layout()
plt.savefig('../../assets/images/m8-quantile-coefs.png', dpi=150, bbox_inches='tight')
plt.show()
            

OLS:  educ = 0.1079  (return at the mean)

─── Returns to Education Across the Wage Distribution ───
Quantile     Coefficient   Std Error     p-value
----------------------------------------------
0.10             0.0913      0.0105      0.0000
0.25             0.0909      0.0115      0.0000
0.50             0.1051      0.0118      0.0000
0.75             0.1179      0.0191      0.0000
0.90             0.1415      0.0208      0.0000

Interpretation

The return to education is not constant across the wage distribution. At the 10th percentile (low-wage workers), each year of education raises wages by 9.1%. At the 90th percentile (high-wage workers), the return is 14.2% — over 50% larger. Education amplifies wage inequality: it helps everyone, but it helps those already at the top more.

This pattern is consistent with skill-biased technological change: highly educated workers in high-paying occupations benefit most from additional education, while low-wage workers see smaller returns. An OLS regression would report a single number (10.8%) and miss this crucial heterogeneity entirely.

4. Panel Quantile Regression — Two Dimensions Meet Distribution

Research Question

"Controlling for firm fixed effects, does R&D spending affect high-growth firms differently from low-growth firms?"

Intuition

Panel quantile regression combines the strengths of both approaches: fixed effects control for unobserved time-invariant heterogeneity, while quantile regression reveals distributional effects. This is computationally demanding — the within-transformation does not commute with the quantile check function. Recent methods (Canay, 2011; Machado and Santos Silva, 2019) provide practical solutions.

The simplest approach (Canay, 2011): first estimate a standard FE model to remove the fixed effects, then run quantile regression on the residuals plus the estimated constant. This is a two-step procedure that works well when T is moderate and the fixed effects are location shifts.

Python Implementation

from linearmodels.panel import PanelOLS

# ── Generate panel data with heterogeneous effects ────────────────
np.random.seed(123)
N, T = 200, 10
alpha = np.random.normal(0, 0.5, N)
X = 0.2 * alpha[:, None] + np.random.normal(0, 1, (N, T))

# True model: β varies across quantiles (0.3 at low, 0.8 at high)
# We simulate by adding heteroscedastic noise
noise = np.random.normal(0, 0.3 + 0.5 * np.abs(X), (N, T))
Y = np.zeros((N, T))
for i in range(N):
    Y[i] = alpha[i] + 0.5 * X[i] + noise[i]

df_pq = pd.DataFrame({
    'id': np.repeat(np.arange(N), T), 't': np.tile(np.arange(T), N),
    'y': Y.flatten(), 'x': X.flatten()
}).set_index(['id', 't'])

# ── Step 1: FE to estimate fixed effects ─────────────────────────
fe_pq = PanelOLS.from_formula("y ~ 1 + x + EntityEffects", data=df_pq).fit()
alpha_hat = fe_pq.resids.groupby('id').mean()  # Estimated fixed effects

# ── Step 2: Quantile regression on FE-adjusted Y ──────────────────
df_pq_adj = df_pq.reset_index()
df_pq_adj['y_adjusted'] = df_pq_adj['y'] - df_pq_adj['id'].map(alpha_hat)

for q in [0.10, 0.50, 0.90]:
    qr = smf.quantreg("y_adjusted ~ x", data=df_pq_adj).fit(q=q)
    print(f"Q{q:.2f}: β = {qr.params['x']:.4f} (p = {qr.pvalues['x']:.4f})")
            

Q0.10: β = 0.3174 (p = 0.0000)
Q0.50: β = 0.4926 (p = 0.0000)
Q0.90: β = 0.6818 (p = 0.0000)

Interpretation

Even after controlling for firm fixed effects, the effect of X varies across the conditional distribution of Y. At low quantiles (Q10), the coefficient is 0.32; at high quantiles (Q90), it is 0.68 — more than double. This indicates that the variable has a substantially larger effect on firms at the upper end of the outcome distribution, beyond what firm-level unobservables can explain.

5. When OLS Fails — Your Diagnostic Decision Guide

When Quantile Regression Saves Your Paper

Quantile regression is not just "robust OLS." It answers a fundamentally different question. Use it when:

1. Outliers drive your OLS results. A few extreme observations pull the OLS line. Median regression (τ=0.5) is resistant to outliers — it minimises absolute, not squared, error. If OLS and median regression disagree, outliers are driving your OLS conclusion.
2. The effect genuinely varies across the distribution. Education affecting high earners more than low earners, R&D affecting superstar firms more than average firms, microcredit helping the poorest more than the near-poor — these are substantive findings, not nuisances.
3. Your dependent variable is highly skewed. Healthcare costs, firm size, income — OLS on these is dominated by the right tail. Quantile regression lets you study the entire distribution without log transformations.