Simple OLS Derivation

ECON 3209 · Week 7, Lecture 3 · Kerala Agricultural University

Department of Development Economics, KAU

Autumn 2026

Learning Outcomes

1. Distinguish between the population regression function and the sample regression function.
2. Define residuals and the sum of squared residuals.
3. Derive \(\hat{\beta}_0\) and \(\hat{\beta}_1\) by minimising SSR.
4. Compute the simple OLS line by hand and in Python.
5. Interpret slope and intercept in an applied Kerala context.

The population regression function (PRF)

\[Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i\]

• \(Y_i\) is the dependent variable.
• \(X_i\) is the explanatory variable.
• \(\beta_0\) and \(\beta_1\) are unknown population parameters.
• \(\varepsilon_i\) collects all other influences on \(Y_i\).
• Example: rice yield as a function of fertilizer use.

The sample regression function (SRF)

\[\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i\]

• The SRF is estimated from sample data.
• It gives fitted values, not actual observations.
• Residuals are the gaps between actual and fitted outcomes.
• OLS chooses the line that fits the sample best in a specific sense.
• That sense is minimising squared residuals.

PRF versus SRF: population and sample

\[\text{PRF: } Y_i = \beta_0 + \beta_1 X_i + \mu_i\]

\[\text{SRF: } \hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_i\]

\[Y_i = \hat{Y}_i + \hat{u}_i\]

• The PRF describes the population relationship with unknown parameters and an unobserved disturbance \(\mu_i\).
• The SRF is the fitted sample line built from estimated coefficients \(\hat{\beta}_0\) and \(\hat{\beta}_1\).
• Residuals \(\hat{u}_i\) are sample estimates of unexplained variation.

Residuals and the objective function

• Residual for observation \(i\) is \[\hat{u}_i = Y_i - \hat{Y}_i = Y_i - \hat{\beta}_0 - \hat{\beta}_1 X_i\]
• OLS minimises the sum of squared residuals: \[SSR = \sum_{i=1}^n (Y_i - \hat{\beta}_0 - \hat{\beta}_1 X_i)^2\]
• Squaring penalises large errors heavily.
• It also prevents positive and negative residuals from cancelling out.

Minimising SSR: first derivative

Take the derivative of SSR with respect to \(\hat{\beta}_0\):

\[\frac{\partial SSR}{\partial \hat{\beta}_0} = -2\sum_{i=1}^n (Y_i - \hat{\beta}_0 - \hat{\beta}_1 X_i) = 0\]

This implies the first normal equation:

\[\sum_{i=1}^n Y_i = n\hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^n X_i\]

• The fitted line passes through the sample means.
• This gives \(\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{X}\).

Minimising SSR: second derivative

Take the derivative of SSR with respect to \(\hat{\beta}_1\):

\[\frac{\partial SSR}{\partial \hat{\beta}_1} = -2\sum_{i=1}^n X_i(Y_i - \hat{\beta}_0 - \hat{\beta}_1 X_i) = 0\]

This gives the second normal equation:

\[\sum_{i=1}^n X_iY_i = \hat{\beta}_0\sum_{i=1}^n X_i + \hat{\beta}_1\sum_{i=1}^n X_i^2\]

• Solving the two equations together gives the OLS estimators.

Closed-form OLS estimators

\[\hat{\beta}_1 = \frac{\sum (X_i - \bar{X})(Y_i - \bar{Y})}{\sum (X_i - \bar{X})^2}\]

\[\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{X}\]

• The slope depends on sample covariance divided by sample variance.
• The intercept ensures the line goes through \((\bar{X}, \bar{Y})\).

RSS minimisation and first-order conditions

\[RSS = \sum_{i=1}^{n}\hat{u}_i^2 = \sum_{i=1}^{n}(Y_i - \hat{\beta}_0 - \hat{\beta}_1 X_i)^2 \rightarrow \min\]

\[\frac{\partial RSS}{\partial \hat{\beta}_0} = 0 \qquad \frac{\partial RSS}{\partial \hat{\beta}_1} = 0\]

• OLS chooses the coefficient values that make the residual sum of squares as small as possible.
• These first-order conditions generate the normal equations used to solve for \(\hat{\beta}_0\) and \(\hat{\beta}_1\).

Manual OLS calculation with five observations

Let \(\bar{x}=4\) and \(\bar{y}=3\).

\(i\)	\(y_i\)	\(x_i\)	\((x_i-\bar{x})^2\)	\((x_i-\bar{x})(y_i-\bar{y})\)
1	1	2	4	4
2	2	3	1	1
3	3	5	1	0
4	4	4	0	0
5	5	6	4	4
Sum	15	20	10	9

• Therefore, \[\hat{\beta}_1 = \frac{9}{10} = 0.9\]
• And \[\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x} = 3 - 0.9(4) = 0.6\]

Verifying the manual calculation in Python

Matrix form of OLS

\[\hat{\beta} = (X'X)^{-1}X'Y\]

• In simple regression, the first column of \(X\) is a column of ones and the second column is the regressor values.
• This compact matrix formula extends naturally to multiple regression.

Computing OLS by hand in Python

Visualising the fitted regression line

Geometric interpretation of OLS

• OLS chooses the line with the smallest total squared vertical distance from the data points.
• In simple regression, the fitted line summarises average linear association.
• The slope tells us how \(Y\) changes when \(X\) increases by one unit.
• The intercept is the predicted value when \(X = 0\).
• Interpretation must always respect the economic context and units.

Estimating the same model with statsmodels

Residuals and fit quality

Interpreting coefficients in context

• If \(\hat{\beta}_1 = 0.26\), then one more unit of fertilizer is associated with about 0.26 more units of yield.
• If fertilizer is measured in kg/ha and yield in quintals/ha, interpret exactly in those units.
• The intercept may be outside the realistic data range, so it is not always economically meaningful.
• The slope is usually the main object of interest in simple regression.
• Regression is a tool for disciplined interpretation, not just equation fitting.

Exercise

Use the arrays x = [10, 20, 30, 40] and y = [15, 18, 23, 27] to compute \(\hat{\beta}_1\) and \(\hat{\beta}_0\) manually. Then print the fitted values.

Summary

• ✅ The PRF describes the population relationship; the SRF is estimated from sample data.
• ✅ OLS minimises the sum of squared residuals.
• ✅ Differentiating SSR gives the normal equations.
• ✅ The resulting estimators are \(\hat{\beta}_0\) and \(\hat{\beta}_1\).
• ✅ We can compute OLS manually and verify it with statsmodels.
• ✅ Coefficients must be interpreted with units and economic context.

Next Lecture

• We move to OLS theory and properties.
• The next lecture introduces the Gauss-Markov assumptions and the meaning of BLUE.