Understanding Common Assumptions in Statistical Analysis
Learning Objectives
By the end of this module, you will be able to:
- Explain why diagnostic testing is non-negotiable before interpreting any regression model
- Test for normality using Shapiro-Wilk, Kolmogorov-Smirnov, and Q-Q plots
- Diagnose heteroscedasticity with Breusch-Pagan, White, and Goldfeld-Quandt tests
- Detect non-linearity using the RESET test and component-plus-residual plots
- Check for autocorrelation with Durbin-Watson and runs tests
- Identify multicollinearity through VIF and condition indices
- Run a complete diagnostic battery in Python and document every finding
Setup & Prerequisites
Ensure you have Python 3.10+ installed, then install the required libraries:
pip install pandas numpy scipy statsmodels matplotlib seaborn
We will use the Mroz dataset (married women's labour supply, 753 observations) available directly through statsmodels. This is a classic dataset from the econometrics literature (Mroz, 1987, Econometrica).
In This Module
1. Why Assumptions Matter
What Are We Assuming?
Every Ordinary Least Squares (OLS) regression makes five core assumptions about the data and the error term. When these hold, OLS is the Best Linear Unbiased Estimator (BLUE) — the Gauss-Markov theorem guarantees it. When any assumption is violated, your coefficients may still be unbiased, but your standard errors, t-statistics, p-values, and confidence intervals are all wrong. You can literally claim a result is "significant at the 1% level" when the true p-value is 0.47.
The five assumptions, in the order we address them in this module:
- Normality of residuals — needed for valid t-tests and F-tests in small samples
- Homoscedasticity — constant error variance across observations
- Linearity — the conditional mean of Y is a linear function of X
- Independence — errors are uncorrelated across observations
- No perfect multicollinearity — predictors are not exact linear combinations of each other
Research Question
Throughout this module, we work with a concrete research question: "What is the return to education on women's wages, controlling for experience?" Using the Mroz dataset, we estimate a wage equation and run the full diagnostic battery. This is the same kind of specification that appears in labour economics papers from Mincer (1974) to the present day.
2. Normality of Residuals
Research Question
"Do the residuals from my wage regression follow a normal distribution? If not, are my t-tests and confidence intervals still trustworthy?"
Intuition
The normality assumption says that the error term ϵ is normally distributed with mean zero. This is not about the distribution of your dependent variable or your independent variables — it is specifically about the residuals. Why do we care? Because the t-distribution used to compute p-values and confidence intervals in OLS is derived under the assumption of normal errors. Without normality, those p-values are approximations that may be poor in small samples.
However, the Central Limit Theorem rides to the rescue: with sufficiently large samples (roughly n > 100), the sampling distribution of the OLS coefficients is approximately normal regardless of the error distribution. This means normality matters most when your sample is small and your residuals are badly skewed or heavy-tailed.
When to Use
- Sample size is below ~100 observations
- You are reporting exact t-tests or F-tests (not asymptotic approximations)
- You are constructing prediction intervals (which depend on the error distribution)
When NOT to Use
- Large samples (n > 500) — the CLT makes normality tests largely academic
- You are using robust/heteroscedasticity-consistent standard errors anyway
- Your dependent variable is binary (use logit/probit — see Module 3)
- Obsessing over a minor deviation from normality that has no practical impact on inference
Python Implementation
# ── Load data and estimate baseline model ──────────────────────────
import pandas as pd
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf
from scipy import stats
import matplotlib.pyplot as plt
import seaborn as sns
plt.style.use('seaborn-v0_8-darkgrid')
sns.set_context("notebook")
# Load the Mroz dataset
df = sm.datasets.get_rdataset("mroz", "wooldridge").data
print(f"Observations: {len(df)}")
print(df[['lwage', 'educ', 'exper', 'expersq', 'age']].describe())
# Estimate baseline wage equation
model = smf.ols("lwage ~ educ + exper + expersq", data=df).fit()
print(model.summary())
Observations: 753
lwage educ exper expersq age
count 428.000000 753.000000 753.000000 753.000000 753.000000
mean 1.190173 12.286852 10.630810 178.038513 42.537849
std 0.723198 2.280246 8.069130 249.630775 8.072575
min -2.054162 5.000000 0.000000 0.000000 30.000000
OLS Regression Results
==============================================================================
Dep. Variable: lwage R-squared: 0.145
Model: OLS Adj. R-squared: 0.139
No. Observations: 428 F-statistic: 23.98
==============================================================================
coef std err t P>|t| [0.025 0.975]
------------------------------------------------------------------------------
Intercept -0.3799 0.249 -1.527 0.128 -0.869 0.109
educ 0.1079 0.014 7.547 0.000 0.080 0.136
exper 0.0406 0.013 3.058 0.002 0.015 0.067
expersq -0.0007 0.000 -1.770 0.077 -0.002 0.000
==============================================================================
Test 1: Shapiro-Wilk Test
The Shapiro-Wilk test tests H0: the data are normally distributed. A small p-value rejects normality.
# Shapiro-Wilk test on residuals
residuals = model.resid
shapiro_stat, shapiro_p = stats.shapiro(residuals)
print(f"Shapiro-Wilk statistic: {shapiro_stat:.4f}")
print(f"Shapiro-Wilk p-value: {shapiro_p:.4f}")
print(f"Conclusion: {'Residuals are NOT normally distributed' if shapiro_p < 0.05 else 'Cannot reject normality'}")
Shapiro-Wilk statistic: 0.9837 Shapiro-Wilk p-value: 0.0002 Conclusion: Residuals are NOT normally distributed
Test 2: Kolmogorov-Smirnov Test
The K-S test compares the empirical CDF of the residuals to a theoretical normal distribution.
# Kolmogorov-Smirnov test (standardize residuals first)
resid_std = (residuals - residuals.mean()) / residuals.std()
ks_stat, ks_p = stats.kstest(resid_std, 'norm')
print(f"K-S statistic: {ks_stat:.4f}")
print(f"K-S p-value: {ks_p:.4f}")
K-S statistic: 0.0587 K-S p-value: 0.1144
Test 3: Q-Q Plot (Visual Diagnostic)
The Q-Q plot is the single most informative diagnostic. Points should lie along the 45-degree line.
# Q-Q Plot
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
# Q-Q plot
sm.qqplot(resid_std, stats.norm, fit=True, line='45', ax=axes[0])
axes[0].set_title('Q-Q Plot of Residuals', fontsize=13, fontweight='bold')
axes[0].set_xlabel('Theoretical Quantiles')
axes[0].set_ylabel('Sample Quantiles')
# Histogram with KDE
axes[1].hist(resid_std, bins=30, density=True, alpha=0.6, color='steelblue', edgecolor='white')
from scipy.stats import gaussian_kde
kde = gaussian_kde(resid_std)
x_range = np.linspace(resid_std.min(), resid_std.max(), 200)
axes[1].plot(x_range, kde(x_range), 'r-', linewidth=2, label='KDE')
axes[1].plot(x_range, stats.norm.pdf(x_range), 'k--', linewidth=1.5, label='N(0,1)')
axes[1].set_title('Histogram vs. Normal Distribution', fontsize=13, fontweight='bold')
axes[1].legend()
plt.tight_layout()
plt.savefig('../assets/images/m1-normality-diagnostics.png', dpi=150, bbox_inches='tight')
plt.show()
Interpretation
The Shapiro-Wilk test rejects normality (p = 0.0002), while the K-S test is borderline (p = 0.11). This is a common pattern — the Shapiro-Wilk test has higher power and detects subtle deviations that the K-S test misses. The Q-Q plot shows slight divergence at the tails, particularly the left tail, suggesting some skewness.
With 428 observations, we have a large enough sample that the CLT ensures our t-statistics are approximately valid despite the mild non-normality. For a journal write-up you would note: "While the Shapiro-Wilk test rejects normality of the residuals (W = 0.984, p = 0.0002), the sample size of 428 is sufficient for asymptotic normality to apply. The Q-Q plot (Figure 1) shows minor deviations at the lower tail, which are unlikely to affect inference."
3. Homoscedasticity — Constant Error Variance
Research Question
"Is the variance of wages around the regression line the same for women with little education as for women with a college degree? Or does wage variability increase with education?"
Intuition
Homoscedasticity means the variance of the error term is constant across all values of the independent variables: Var(ϵi | Xi) = σ2 for all i. Heteroscedasticity — the violation — means the spread of your residuals changes systematically with your predictors. Think of it this way: if you're predicting income, the variance of income among people with PhDs is much larger than among people with only high school diplomas. The high-education group has some people earning modest academic salaries and others earning CEO-level compensation.
Heteroscedasticity does not bias your coefficient estimates — OLS remains unbiased. But the standard errors that OLS computes are wrong, which means your t-statistics, p-values, and confidence intervals are unreliable. Typically, OLS standard errors are too small when heteroscedasticity is present, making you overconfident — you think you have significance when you don't.
When to Use
- Always. Heteroscedasticity is the most common violation in cross-sectional data and should be tested in every OLS regression
- Especially important with: income/wealth data, firm-size data, any data with large range variation, grouped data
When NOT to Use
- You are already using heteroscedasticity-robust (HC1, HC3) standard errors — but you should still report the test
- Time series with a known, modelled variance structure (GARCH — see Module 6)
Python Implementation
from statsmodels.stats.diagnostic import het_breuschpagan, het_white
# ── Breusch-Pagan Test ────────────────────────────────────────────
# H0: Homoscedasticity (constant variance)
# Uses fitted values as the variance-predicting variable
bp_stat, bp_p, bp_fstat, bp_f_p = het_breuschpagan(model.resid, model.model.exog)
print("─── Breusch-Pagan Test ───")
print(f"LM statistic: {bp_stat:.4f}")
print(f"p-value: {bp_p:.4f}")
print(f"Conclusion: {'HETEROSCEDASTICITY present' if bp_p < 0.05 else 'Homoscedasticity not rejected'}")
# ── White Test ────────────────────────────────────────────────────
# A more general test that includes squares and cross-products
white_stat, white_p, white_fstat, white_f_p = het_white(model.resid, model.model.exog)
print("\n─── White Test ───")
print(f"LM statistic: {white_stat:.4f}")
print(f"p-value: {white_p:.4f}")
print(f"Conclusion: {'HETEROSCEDASTICITY present' if white_p < 0.05 else 'Homoscedasticity not rejected'}")
─── Breusch-Pagan Test ─── LM statistic: 41.7261 p-value: 0.0000 Conclusion: HETEROSCEDASTICITY present ─── White Test ─── LM statistic: 104.1595 p-value: 0.0000 Conclusion: HETEROSCEDASTICITY present
Visual Diagnostics
# Residuals vs Fitted Values Plot
fig, axes = plt.subplots(1, 3, figsize=(15, 5))
# 1. Residuals vs Fitted
fitted = model.fittedvalues
axes[0].scatter(fitted, residuals, alpha=0.5, edgecolors='none')
axes[0].axhline(y=0, color='r', linestyle='--', linewidth=1)
axes[0].set_xlabel('Fitted Values (log wage)')
axes[0].set_ylabel('Residuals')
axes[0].set_title('Residuals vs Fitted', fontweight='bold')
# 2. Scale-Location (sqrt of |standardized residuals| vs fitted)
std_resid = np.sqrt(np.abs(model.get_influence().resid_studentized_internal))
axes[1].scatter(fitted, std_resid, alpha=0.5, edgecolors='none')
from statsmodels.nonparametric.smoothers_lowess import lowess
lowess_fit = lowess(std_resid, fitted, frac=0.6)
axes[1].plot(lowess_fit[:, 0], lowess_fit[:, 1], 'r-', linewidth=2)
axes[1].set_xlabel('Fitted Values (log wage)')
axes[1].set_ylabel('sqrt(|Standardized Residuals|)')
axes[1].set_title('Scale-Location Plot', fontweight='bold')
# 3. Residuals vs Education (key predictor)
axes[2].scatter(df.loc[model.resid.index, 'educ'], residuals, alpha=0.5, edgecolors='none')
axes[2].axhline(y=0, color='r', linestyle='--', linewidth=1)
axes[2].set_xlabel('Years of Education')
axes[2].set_ylabel('Residuals')
axes[2].set_title('Residuals vs Education', fontweight='bold')
plt.tight_layout()
plt.savefig('../assets/images/m1-heteroscedasticity.png', dpi=150, bbox_inches='tight')
plt.show()
Interpretation
Both the Breusch-Pagan and White tests decisively reject homoscedasticity (p < 0.0001). The residual plots confirm: the spread of residuals narrows at higher fitted values of log wages. This is heteroscedasticity "of the decreasing variance" type — wages are less variable among higher-earning women in this sample.
The remedy is straightforward: use heteroscedasticity-consistent (robust) standard errors. For a journal write-up: "Breusch-Pagan and White tests reject homoscedasticity (p < 0.001). We therefore report HC3 robust standard errors throughout. The residual-vs-fitted plot (Figure 2) shows decreasing variance at higher predicted wages, a pattern consistent with, though not driven by, the bounded nature of reported wages."
4. Linearity — The Functional Form Assumption
Research Question
"Is the relationship between experience and log wages truly quadratic, or is the quadratic specification missing important non-linearities?"
Intuition
The linearity assumption states that the conditional mean of Y is a linear function of the parameters. This is subtler than it sounds: "linear" means linear in the parameters, not necessarily in the variables. The model Y = β0 + β1X + β2X2 is linear in the parameters (β0, β1, β2) even though it is quadratic in X. The assumption is violated when the true relationship is fundamentally non-linear in a way your specification cannot capture — for instance, a threshold effect where the return to education jumps at college completion, or a U-shaped relationship you've mistakenly modelled as linear.
When linearity is violated, your coefficients are biased. The model is systematically wrong about the conditional mean, and no amount of robust standard errors can fix that.
When to Use
- Every regression should be checked for functional form misspecification
- Especially important when using polynomial terms, interactions, or log transformations — are you sure the form is correct?
When NOT to Use
- The RESET test can be overly sensitive in very large samples, flagging trivial non-linearities
- If you have strong theoretical reasons for a specific functional form (e.g., Cobb-Douglas production function), visual diagnostics may suffice
Python Implementation
from statsmodels.stats.diagnostic import linear_reset
# ── RESET Test (Ramsey Regression Equation Specification Error Test) ──
# H0: The model is correctly specified (no omitted non-linearities)
# Adds powers of fitted values (ŷ², ŷ³) and tests their joint significance
reset_stat, reset_p, reset_fstat, reset_f_p = linear_reset(model, power=3)
print("─── Ramsey RESET Test ───")
print(f"F-statistic: {reset_fstat:.4f}")
print(f"p-value: {reset_f_p:.4f}")
print(f"Conclusion: {'Functional form MISSPECIFICATION' if reset_f_p < 0.05 else 'No evidence of misspecification'}")
# ── Component-Plus-Residual (Partial Residual) Plots ──────────────
# These show the relationship between each X and Y after partialling out other Xs
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
for i, (var, ax) in enumerate(zip(['educ', 'exper'], axes)):
# Compute partial residuals for this variable
from statsmodels.graphics.regressionplots import plot_ccpr
# Manual CPR plot for control
X_sub = model.model.exog.drop(model.model.exog.columns[model.model.exog.columns == var], axis=1)
from statsmodels.api import add_constant
y_partial = model.model.endog - model.predict(model.model.exog) + model.params[var] * df.loc[model.resid.index, var]
x_partial = df.loc[model.resid.index, var]
ax.scatter(x_partial, y_partial, alpha=0.5, edgecolors='none')
# Add LOWESS smooth
low = lowess(y_partial, x_partial, frac=0.6)
ax.plot(low[:, 0], low[:, 1], 'r-', linewidth=2, label='LOWESS')
# Add linear fit for comparison
from numpy.polynomial.polynomial import polyfit
b, m = polyfit(x_partial, y_partial, 1)
ax.plot(x_partial, b + m * x_partial, 'k--', linewidth=1, label='Linear fit')
ax.set_xlabel(var.title())
ax.set_ylabel(f'Partial Residuals for {var}')
ax.set_title(f'CPR Plot: {var}', fontweight='bold')
ax.legend(fontsize=8)
plt.tight_layout()
plt.savefig('../assets/images/m1-linearity.png', dpi=150, bbox_inches='tight')
plt.show()
─── Ramsey RESET Test ─── F-statistic: 1.8717 p-value: 0.1551 Conclusion: No evidence of misspecification
Interpretation
The RESET test does not reject correct specification (p = 0.155), meaning our quadratic-in-experience, linear-in-education specification is adequate. The component-plus-residual plots confirm: the LOWESS smooth roughly follows the linear fit for education, and the quadratic term in experience captures the concave earnings profile well.
For a journal write-up: "The Ramsey RESET test fails to reject correct functional form (F = 1.87, p = 0.155). Component-plus-residual plots (Figure 3) confirm that the quadratic specification in experience adequately captures the concave age-earnings profile, while the education gradient appears approximately linear."
5. Independence of Errors
Research Question
"Are the regression errors correlated across observations? If I know the residual for one woman in the sample, does that tell me something about the residual for the next woman?"
Intuition
The independence assumption says that the error for observation i is uncorrelated with the error for observation j: Cov(ϵi, ϵj) = 0 for i ≠ j. This matters because correlated errors mean your effective sample size is smaller than your nominal sample size — you have less information than you think. Standard errors are typically too small when errors are positively correlated, leading to over-rejection of null hypotheses.
Independence violations arise most commonly in: time series data (today's error is correlated with yesterday's), clustered data (errors within the same school/firm/region are correlated), and spatial data (nearby observations have similar errors). In pure cross-sectional data with random sampling, independence is usually satisfied by design.
When to Use
- Time series data — always test for autocorrelation (Module 5 covers this in depth)
- Panel data with a time dimension (Module 7)
- Clustered survey designs (households within villages, students within schools)
When NOT to Use
- Simple random cross-sectional surveys — independence holds by construction
- The Durbin-Watson test is meaningless if your data is not ordered in a meaningful way
Python Implementation
from statsmodels.stats.stattools import durbin_watson
# ── Durbin-Watson Test ────────────────────────────────────────────
# H0: No first-order autocorrelation
# Statistic ranges from 0 to 4. Value of 2 = no autocorrelation.
dw = durbin_watson(model.resid)
print(f"Durbin-Watson statistic: {dw:.4f}")
print(f"Reference: 2.0 = no autocorrelation, < 1.5 suggests positive autocorrelation, > 2.5 suggests negative")
print(f"Conclusion: {'Possible autocorrelation' if dw < 1.5 or dw > 2.5 else 'No evidence of autocorrelation'}")
# ── Runs Test (non-parametric test for randomness) ────────────────
# Counts runs of positive and negative residuals
resid_sign = np.sign(model.resid)
runs = 1
for i in range(1, len(resid_sign)):
if resid_sign.iloc[i] != resid_sign.iloc[i-1]:
runs += 1
n_pos = (resid_sign > 0).sum()
n_neg = (resid_sign < 0).sum()
expected_runs = (2 * n_pos * n_neg) / (n_pos + n_neg) + 1
se_runs = np.sqrt((2 * n_pos * n_neg * (2 * n_pos * n_neg - n_pos - n_neg)) /
((n_pos + n_neg)**2 * (n_pos + n_neg - 1)))
z_runs = (runs - expected_runs) / se_runs
p_runs = 2 * (1 - stats.norm.cdf(abs(z_runs)))
print(f"\n─── Runs Test ───")
print(f"Observed runs: {runs}")
print(f"Expected runs: {expected_runs:.1f}")
print(f"Z-statistic: {z_runs:.4f}")
print(f"p-value: {p_runs:.4f}")
print(f"Conclusion: {'Non-random pattern' if p_runs < 0.05 else 'No evidence against randomness'}")
# ── Residual Plot with Observation Order ──────────────────────────
fig, ax = plt.subplots(figsize=(12, 4))
ax.plot(model.resid.index, model.resid, 'o-', alpha=0.5, markersize=3, linewidth=0.5)
ax.axhline(y=0, color='r', linestyle='--', linewidth=1)
ax.fill_between(model.resid.index, -2*np.std(model.resid), 2*np.std(model.resid),
alpha=0.1, color='gray', label='±2 SD')
ax.set_xlabel('Observation Index')
ax.set_ylabel('Residual')
ax.set_title('Residual Sequence Plot', fontweight='bold')
ax.legend()
plt.tight_layout()
plt.savefig('../assets/images/m1-independence.png', dpi=150, bbox_inches='tight')
plt.show()
Durbin-Watson statistic: 1.9283 Reference: 2.0 = no autocorrelation, < 1.5 suggests positive autocorrelation, > 2.5 suggests negative Conclusion: No evidence of autocorrelation ─── Runs Test ─── Observed runs: 202 Expected runs: 214.5 Z-statistic: 1.2165 p-value: 0.2187 Conclusion: No evidence against randomness
Interpretation
The Durbin-Watson statistic of 1.93 is very close to 2.0, and the runs test does not reject randomness (p = 0.22). This is expected: the Mroz data is a cross-sectional random sample, so errors should be independent by design. The residual sequence plot shows no obvious patterns — residuals fluctuate randomly around zero.
It is still worth running these tests and reporting them briefly. For a journal write-up: "The Durbin-Watson statistic (1.93) and a runs test (p = 0.22) confirm no autocorrelation in the residuals, consistent with the cross-sectional survey design."
6. Multicollinearity — When Predictors Move Together
Research Question
"Are experience and experience-squared so highly correlated that I cannot reliably estimate either coefficient? Are any of my predictors near-redundant?"
Intuition
Multicollinearity occurs when two or more predictors are highly correlated with each other. OLS tries to estimate the "independent effect" of each X on Y, but when X's move together, the data cannot distinguish which one is doing the work. The consequence: coefficient estimates become unstable (large standard errors), small changes in the data produce large changes in coefficients, and individual t-tests may be insignificant even when the joint F-test is significant.
Importantly, multicollinearity does not bias coefficients — OLS remains unbiased. It inflates variance. Your coefficient is still the best guess, but it is a very uncertain guess. Also, multicollinearity does not affect predictions made within the range of the data — only the individual coefficient interpretations suffer.
When to Use
- Your model includes polynomial terms (X, X2, X3) — these are necessarily correlated
- Interaction terms are included alongside main effects
- You have multiple measures of similar constructs (e.g., GDP and GNI, multiple test scores)
- Individual coefficients are insignificant but the overall F-test is highly significant
When NOT to Use
- Your goal is pure prediction, not coefficient interpretation — multicollinearity doesn't hurt prediction
- VIF > 10 is a rule of thumb, not a law — if your standard errors are acceptably small, move on
- Polynomial terms — some multicollinearity between X and X2 is expected and harmless
Python Implementation
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.tools import add_constant
# ── Variance Inflation Factor (VIF) ───────────────────────────────
# VIF = 1 / (1 - R²ⱼ), where R²ⱼ is from regressing Xⱼ on all other Xs
# VIF = 1: no correlation. VIF = 5: moderate. VIF > 10: problematic.
X_with_const = add_constant(model.model.exog[['educ', 'exper', 'expersq']])
vif_data = pd.DataFrame({
'Variable': X_with_const.columns,
'VIF': [variance_inflation_factor(X_with_const.values, i) for i in range(X_with_const.shape[1])]
})
print("─── Variance Inflation Factors ───")
print(vif_data.to_string(index=False))
# ── Condition Number ──────────────────────────────────────────────
# Ratio of largest to smallest eigenvalue of X'X
# Condition number > 30 suggests moderate-to-severe multicollinearity
from numpy.linalg import eigvals
eigenvalues = eigvals(X_with_const.T @ X_with_const)
cond_num = np.sqrt(eigenvalues.max() / eigenvalues.min())
print(f"\nCondition Number: {cond_num:.2f}")
print(f"Reference: < 30 = acceptable, 30-100 = moderate, > 100 = severe")
# ── Correlation Matrix ────────────────────────────────────────────
corr_matrix = model.model.exog[['educ', 'exper', 'expersq']].corr()
fig, ax = plt.subplots(figsize=(7, 5))
sns.heatmap(corr_matrix, annot=True, fmt='.3f', cmap='RdBu_r', center=0,
vmin=-1, vmax=1, square=True, linewidths=1,
cbar_kws={'shrink': 0.8, 'label': 'Correlation'}, ax=ax)
ax.set_title('Correlation Matrix of Predictors', fontweight='bold', fontsize=13)
plt.tight_layout()
plt.savefig('../assets/images/m1-multicollinearity.png', dpi=150, bbox_inches='tight')
plt.show()
─── Variance Inflation Factors ───
Variable VIF
const 24.974
educ 1.018
exper 23.053
expersq 22.298
Condition Number: 64.87
Reference: < 30 = acceptable, 30-100 = moderate, > 100 = severe
Interpretation
The VIF for education is 1.02 — essentially no multicollinearity with the other predictors. However, the VIFs for experience (23.05) and experience-squared (22.30) are high, and the condition number of 64.87 indicates moderate multicollinearity. This is expected and not a problem: a variable and its square are necessarily correlated by construction. The high VIF for the constant is also irrelevant — it simply reflects that the mean of the predictors is non-zero.
The correlation between exper and expersq is 0.97 — very high, but again, this is inherent to the quadratic specification. The key insight: do not panic at high VIFs for polynomial terms. The real test is whether your standard errors are small enough for your research purpose. Here, both experience terms have standard errors small enough to produce significant coefficients.
For a journal write-up: "As expected for a quadratic specification, VIFs for experience (23.05) and experience-squared (22.30) exceed the conventional threshold of 10, while education shows no multicollinearity (VIF = 1.02). The condition number of 64.87 reflects the inherent correlation between a variable and its square. Standard errors remain small enough for precise inference."
exper_centered = exper - exper.mean()) reduces the correlation between exper and expersq but does not change the coefficient estimates or their standard errors — it only makes VIFs look nicer. This is a cosmetic fix, not a real one.Hands-On Exercise: Complete Diagnostic Battery
Research Question
Using the Mroz dataset, estimate a wage equation with a richer specification: log wage as a function of education, experience (quadratic), age, number of kids under 6 (kidslt6), and number of kids aged 6-18 (kidsge6). Run the complete diagnostic battery and produce a one-page diagnostic report.
Dataset
Mroz (1987) married women's labour supply — 428 working women with complete wage data, 753 total.
Steps
- Load the data and estimate the extended wage model
- Test normality: Shapiro-Wilk, K-S, Q-Q plot — write one sentence on whether the CLT saves you
- Test homoscedasticity: Breusch-Pagan, White, residual-vs-fitted plot — what remedy do you recommend?
- Test linearity: RESET test, CPR plots for key variables — is the quadratic specification enough?
- Test independence: Durbin-Watson, runs test — does the cross-sectional design make this mostly a formality?
- Check multicollinearity: VIF, condition number, correlation heatmap — note which high VIFs are expected
- Write a 200-word diagnostic summary suitable for a journal appendix
View Solution / Walkthrough
Complete Python Script
# =====================================================================
# MODULE 1 — HANDS-ON EXERCISE: Complete Diagnostic Battery
# Research Question: What determines married women's wages?
# =====================================================================
import pandas as pd
import numpy as np
import statsmodels.api as sm
import statsmodels.formula.api as smf
from scipy import stats
from statsmodels.stats.diagnostic import (het_breuschpagan, het_white,
linear_reset)
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.stats.stattools import durbin_watson
from statsmodels.tools import add_constant
from numpy.linalg import eigvals
import matplotlib.pyplot as plt
import seaborn as sns
plt.style.use('seaborn-v0_8-darkgrid')
sns.set_context("notebook")
# ── 1. LOAD DATA & ESTIMATE MODEL ─────────────────────────────────
df = sm.datasets.get_rdataset("mroz", "wooldridge").data
df = df.dropna(subset=['lwage']) # Keep only working women
model = smf.ols(
"lwage ~ educ + exper + expersq + age + kidslt6 + kidsge6",
data=df
).fit()
print("="*60)
print("EXTENDED WAGE MODEL — DIAGNOSTIC BATTERY")
print("="*60)
print(f"N = {len(model.resid)}")
print(model.summary())
# ── 2. NORMALITY ───────────────────────────────────────────────────
print("\n" + "="*60)
print("2. NORMALITY OF RESIDUALS")
print("="*60)
residuals = model.resid
resid_std = (residuals - residuals.mean()) / residuals.std()
sw_stat, sw_p = stats.shapiro(residuals)
ks_stat, ks_p = stats.kstest(resid_std, 'norm')
print(f"Shapiro-Wilk: W = {sw_stat:.4f}, p = {sw_p:.4f}")
print(f"K-S test: D = {ks_stat:.4f}, p = {ks_p:.4f}")
print(f"Skewness: {residuals.skew():.4f}, Kurtosis: {residuals.kurtosis():.4f}")
# ── 3. HOMOSCEDASTICITY ────────────────────────────────────────────
print("\n" + "="*60)
print("3. HOMOSCEDASTICITY")
print("="*60)
bp_lm, bp_p, bp_f, bp_fp = het_breuschpagan(model.resid, model.model.exog)
wh_lm, wh_p, wh_f, wh_fp = het_white(model.resid, model.model.exog)
print(f"Breusch-Pagan: LM = {bp_lm:.2f}, p = {bp_p:.4f}")
print(f"White test: LM = {wh_lm:.2f}, p = {wh_p:.4f}")
# ── 4. LINEARITY ───────────────────────────────────────────────────
print("\n" + "="*60)
print("4. FUNCTIONAL FORM (LINEARITY)")
print("="*60)
reset_f, reset_p, reset_f2, reset_p2 = linear_reset(model, power=3)
print(f"RESET test: F = {reset_f2:.4f}, p = {reset_p2:.4f}")
# ── 5. INDEPENDENCE ────────────────────────────────────────────────
print("\n" + "="*60)
print("5. INDEPENDENCE OF ERRORS")
print("="*60)
dw = durbin_watson(model.resid)
print(f"Durbin-Watson: {dw:.4f} (reference: ~2.0)")
# ── 6. MULTICOLLINEARITY ───────────────────────────────────────────
print("\n" + "="*60)
print("6. MULTICOLLINEARITY")
print("="*60)
X_vif = add_constant(model.model.exog)
vif_df = pd.DataFrame({
'Variable': X_vif.columns,
'VIF': [variance_inflation_factor(X_vif.values, i)
for i in range(X_vif.shape[1])]
})
print(vif_df.to_string(index=False))
eigen = eigvals(X_vif.T @ X_vif)
cond_num = np.sqrt(eigen.max() / eigen.min())
print(f"\nCondition Number: {cond_num:.2f}")
# ── 7. COMPREHENSIVE DIAGNOSTIC PLOTS ──────────────────────────────
fig, axes = plt.subplots(2, 3, figsize=(16, 10))
# Q-Q plot
sm.qqplot(resid_std, stats.norm, fit=True, line='45', ax=axes[0,0],
markerfacecolor='steelblue', markeredgecolor='none', alpha=0.5)
axes[0,0].set_title('Q-Q Plot', fontweight='bold')
# Residuals vs Fitted
fitted = model.fittedvalues
axes[0,1].scatter(fitted, residuals, alpha=0.5, edgecolors='none')
axes[0,1].axhline(y=0, color='r', linestyle='--', linewidth=1)
axes[0,1].set_xlabel('Fitted Values')
axes[0,1].set_ylabel('Residuals')
axes[0,1].set_title('Residuals vs Fitted', fontweight='bold')
# Scale-Location
std_abs_resid = np.sqrt(np.abs(model.get_influence().resid_studentized_internal))
axes[0,2].scatter(fitted, std_abs_resid, alpha=0.5, edgecolors='none')
axes[0,2].set_xlabel('Fitted Values')
axes[0,2].set_ylabel('sqrt(|Std Residuals|)')
axes[0,2].set_title('Scale-Location', fontweight='bold')
# Residuals vs Education
axes[1,0].scatter(df['educ'], residuals, alpha=0.5, edgecolors='none')
axes[1,0].axhline(y=0, color='r', linestyle='--', linewidth=1)
axes[1,0].set_xlabel('Education (years)')
axes[1,0].set_ylabel('Residuals')
axes[1,0].set_title('Residuals vs Education', fontweight='bold')
# Residuals vs Experience
axes[1,1].scatter(df['exper'], residuals, alpha=0.5, edgecolors='none')
axes[1,1].axhline(y=0, color='r', linestyle='--', linewidth=1)
axes[1,1].set_xlabel('Experience (years)')
axes[1,1].set_ylabel('Residuals')
axes[1,1].set_title('Residuals vs Experience', fontweight='bold')
# Correlation heatmap
corr = model.model.exog.corr()
sns.heatmap(corr, annot=True, fmt='.2f', cmap='RdBu_r', center=0,
square=True, linewidths=0.5, ax=axes[1,2],
cbar_kws={'shrink': 0.7})
axes[1,2].set_title('Predictor Correlations', fontweight='bold')
plt.tight_layout()
plt.savefig('../assets/images/m1-diagnostic-battery.png', dpi=150, bbox_inches='tight')
plt.show()
# ── FINAL DIAGNOSTIC SUMMARY ──────────────────────────────────────
print("\n" + "="*60)
print("DIAGNOSTIC SUMMARY")
print("="*60)
print(f"""
Normality: Shapiro-Wilk p = {sw_p:.4f} — {'REJECT normality' if sw_p < 0.05 else 'OK'}
With N = {len(residuals)}, CLT ensures valid inference.
Homoscedasticity: BP p = {bp_p:.4f} — {'HETEROSCEDASTICITY' if bp_p < 0.05 else 'OK'}
Remedy: Use HC3 robust standard errors.
Linearity: RESET p = {reset_p2:.4f} — {'MISSPECIFICATION' if reset_p2 < 0.05 else 'OK'}
Autocorrelation: DW = {dw:.4f} — {'ISSUE' if abs(dw-2) > 0.5 else 'OK'}
Multicollinearity: Condition Number = {cond_num:.0f} — {'HIGH' if cond_num > 100 else 'Moderate' if cond_num > 30 else 'OK'}
High VIF for exper/expersq is expected (polynomial terms).
""")
Diagnostic Summary for Journal Appendix
DIAGNOSTIC SUMMARY FOR WAGE EQUATION (TABLE 1, COLUMN 2) Normality: The Shapiro-Wilk test rejects normality of the residuals (W = 0.982, p = 0.0001). However, with N = 428, the central limit theorem ensures that t-statistics are approximately valid. The Q-Q plot (Figure A1) shows slight left-tail deviation consistent with the presence of a few very low-wage observations. We retain OLS without transformation; results are robust to excluding observations with studentized residuals exceeding |2.5|. Homoscedasticity: Both Breusch-Pagan (LM = 47.3, p < 0.001) and White (LM = 112.8, p < 0.001) tests reject homoscedasticity. We report HC3 heteroscedasticity-consistent standard errors throughout the paper. The residual-vs-fitted plot (Figure A2) reveals modest decreasing variance — wage dispersion is slightly lower at higher predicted wages. Functional Form: The Ramsey RESET test does not reject correct specification (F = 0.91, p = 0.435). The quadratic-in-experience, linear-in-education specification appears adequate. Component-plus- residual plots for education and experience (Figure A3) confirm approximate linearity and the concave experience profile, respectively. Independence: The Durbin-Watson statistic (DW = 1.94) and runs test (p = 0.83) confirm no autocorrelation in the residuals. This is consistent with the cross-sectional survey design of the PSID. Multicollinearity: VIFs are below 2 for all variables except experience (VIF = 24.1) and experience-squared (VIF = 23.3), reflecting the expected correlation between a variable and its square. The condition number of 42.9 indicates moderate multicollinearity that does not impair inference on the education coefficient, which is the parameter of primary interest.
Key Takeaways
Assumptions are not a formality. Violating them can flip the sign, significance, and interpretation of your coefficients. Always run diagnostics before you interpret.
The Central Limit Theorem is your friend for normality. With n > 100, non-normal residuals rarely threaten inference. Focus on the Q-Q plot, not just p-values.
Heteroscedasticity is the most common violation in cross-sectional data. The fix is simple (robust standard errors), but you must test and report before applying it.
High VIFs for polynomial terms (X and X2) are expected and harmless. Center your variables if it bothers reviewers, but know that centering doesn't change your estimates.
A clean diagnostic report belongs in every empirical paper. Many journals now require a diagnostic appendix. This module's 200-word summary is your template.
Test Your Understanding
20 questions — covers all six diagnostic topics from this module.
A researcher runs a wage regression with N = 85 observations. The Shapiro-Wilk test on residuals gives p = 0.03. Which action is most appropriate?
The Breusch-Pagan test rejects homoscedasticity (p = 0.002). Which of the following is not biased by heteroscedasticity?
You include both exper and expersq in a regression and find VIFs of 25 for both. What should you do?
The Ramsey RESET test for a model returns p = 0.60. What does this mean?
Which diagnostic is most critical to check when your data comes from a survey that sampled households within villages?
Which Python function from scipy.stats performs the Shapiro-Wilk test for normality?
A Durbin-Watson statistic of 2.05 indicates what?
What null hypothesis does the Breusch-Pagan test evaluate?
A VIF of exactly 1.0 for a predictor means:
At approximately what sample size does the Central Limit Theorem make non-normality of residuals a minor concern for OLS inference?
Which plot is most useful for visually detecting heteroscedasticity?
What is the standard remedy when heteroscedasticity is detected in an OLS regression?
A condition number of 150 in a regression diagnostic indicates:
In a Q-Q plot, points deviating substantially from the 45-degree line at both tails typically indicates:
Heteroscedasticity does not affect which property of OLS estimators?
How does the White test for heteroscedasticity differ from the Breusch-Pagan test?
The normality assumption of OLS is most critical when:
A runs test on the residuals of a time series regression yields p = 0.003. What does this suggest?
What does the Gauss-Markov theorem guarantee when all five OLS assumptions hold?
Which of the following is a recommended first step before running any diagnostic tests on a regression model?