Python OLS from Scratch

ECON 3209 · Week 8, Lecture 3 · Kerala Agricultural University

Department of Development Economics, KAU

Autumn 2026

Learning Outcomes

  1. Build a simple regression dataset in NumPy and pandas.
  2. Compute OLS manually using matrix algebra.
  3. Calculate fitted values, residuals, and \(R^2\) by hand.
  4. Estimate the same model with statsmodels and compare results.
  5. Understand how theory maps into practical Python code.

The matrix form of OLS

\[\hat{\beta} = (X'X)^{-1}X'Y\]

  • Here \(X\) is the design matrix containing a column of ones and the regressor values.
  • \(Y\) is the vector of observed outcomes.
  • Matrix notation scales easily to multiple regression.
  • Today we compute each step directly in Python.

Generating a simple dataset

Building the design matrix

  • The first column of \(X\) is a column of ones for the intercept.
  • The second column contains the regressor fertilizer.
  • In simple regression, \[X = \begin{bmatrix}1 & X_1 \\ 1 & X_2 \\ \vdots & \vdots \\ 1 & X_n\end{bmatrix}\]
  • Once \(X\) and \(Y\) are defined, matrix OLS is mechanical.

Manual OLS with NumPy

Fitted values and residuals

  • Fitted values are \[\hat{Y} = X\hat{\beta}\]
  • Residuals are \[\hat{u} = Y - \hat{Y}\]
  • These are the building blocks for SSR and \(R^2\).
  • Good coding practice makes each object explicit.

Computing fit statistics manually

Estimating the same model with statsmodels

  • statsmodels.formula.api lets us write models in a readable formula style.
  • It also provides standard errors, tests, and confidence intervals automatically.
  • Manual OLS helps intuition; statsmodels helps practical analysis.
  • We should understand both.

OLS with formula API

Comparing manual and package results

  • The manual slope and intercept should match the statsmodels estimates up to rounding.
  • The same is true for fitted values and residuals.
  • Matching results confirm that the matrix formula and package implementation are consistent.
  • This is a good debugging habit when learning econometrics.

Plotting data and fitted line

Why this matters for econometrics students

  • Manual computation shows where coefficients really come from.
  • Matrix notation prepares you for multiple regression later.
  • statsmodels output becomes easier to interpret when you know the mechanics.
  • Coding OLS from scratch connects theory, algebra, and software.
  • This is one of the best ways to internalise regression logic.

Exercise

Generate 20 observations for income and repayment_rate. Compute the OLS coefficients manually using \(\hat{\beta} = (X'X)^{-1}X'Y\), then fit the same model with statsmodels and compare the two slopes.

Summary

  • ✅ Matrix OLS uses \(\hat{\beta} = (X'X)^{-1}X'Y\).
  • ✅ The design matrix includes a column of ones for the intercept.
  • ✅ Fitted values, residuals, SSR, and \(R^2\) can all be computed manually.
  • ✅ statsmodels reproduces the same coefficient estimates.
  • ✅ Manual coding improves intuition and debugging ability.
  • ✅ Python allows us to move smoothly from algebra to empirical analysis.

Next Lecture

  • We begin Inference in OLS.
  • The next topic is hypothesis testing: nulls, alternatives, test statistics, and p-values.