Forecast Evaluation

ECON 3209 · Week 20, Lecture 2 · Kerala Agricultural University

Department of Development Economics, KAU

Autumn 2026

Learning Outcomes

By the end of this lecture, you should be able to:

  1. compute MAE, RMSE, and MAPE for forecast comparison
  2. distinguish in-sample fit from out-of-sample forecast performance
  3. set up train-test splits and rolling-window evaluation in Python
  4. compare naïve and ARIMA forecasts on a Kerala-style series
  5. choose a forecasting model using evidence rather than fit alone

Why Forecast Evaluation Matters

  • A model can fit historical data well but forecast poorly.
  • Forecasting focuses on future accuracy, not just in-sample explanation.
  • We therefore evaluate models on observations not used in estimation.

Common Accuracy Measures

  • MAE: average absolute error.
  • RMSE: square root of mean squared error; penalises large mistakes more heavily.
  • MAPE: average absolute percentage error.

Lower values indicate better forecast accuracy, subject to context and scale.

Train-Test and Rolling Windows

  • Train-test split: estimate on earlier data, evaluate on later data.
  • Rolling window: move through time, re-estimating or updating as new data arrive.
  • Rolling evaluation mimics real forecasting practice more closely.

Set Up a Hold-Out Sample

Compare Naive and ARIMA Forecasts

Compute MAE, RMSE, and MAPE

Kerala Forecasting Context

A cooperative department may forecast monthly procurement, paddy arrivals, or loan demand.

The best forecasting model is the one that predicts future observations reliably, not necessarily the one with the best-looking in-sample fit.

Rolling Window Example

Visual Forecast Comparison

Choosing the Best Forecast

  • Start with a sensible benchmark such as the naïve forecast.
  • Compare models on the same hold-out period.
  • Report multiple metrics when possible.
  • Simpler models often perform surprisingly well, so do not assume complexity always wins.

Exercise

Use the starter code to compare a naïve forecast with ARIMA on the last five observations. Which model wins on RMSE, and would your answer change if you cared more about percentage errors?

Summary

  • ✅ Forecast evaluation must be out-of-sample whenever possible.
  • ✅ MAE, RMSE, and MAPE capture different aspects of forecast accuracy.
  • ✅ Rolling-window evaluation mimics real forecasting decisions over time.
  • ✅ The preferred forecasting model is the one that performs best on future data, not merely past data.

Next Lecture

  • We conclude the course with a full review of all 20 weeks.
  • The final class summarises core formulas, workflows, and research applications.
  • Be ready for a mini-project that brings the course together.