Real Agricultural Data

ECON 3209 · Week 4, Lecture 3 · Kerala Agricultural University

Department of Development Economics, KAU

Autumn 2026

Learning Outcomes

By the end of this lecture, you will be able to:

  1. Simulate loading agricultural data from CSV-style text
  2. Detect and handle missing values in pandas
  3. Clean inconsistent codes and data types
  4. Build a small Kerala agricultural case-study table
  5. Explain why data quality checks matter before econometric analysis

From Raw Data to Analysis-Ready Data

  • Real data is often incomplete, inconsistent, and untidy
  • Missing rainfall, crop yield, or bank values are common
  • We must inspect, clean, and document each transformation
  • pandas gives practical tools for these tasks
  • Good cleaning protects the credibility of later results

Simulating CSV Import with StringIO

  • StringIO lets us treat text like a CSV file
  • This is useful for classroom demonstrations without external files
  • Missing entries often become NaN
  • Text codes like NA may need explicit cleaning

Inspecting Missing Values

  • .isna() identifies missing values cell by cell
  • Summing the boolean table shows how many values are missing per column
  • Missingness itself can be informative in field data
  • Always inspect before deciding how to treat gaps

Replacing Special Codes

  • Some datasets use codes like -999 or 99 for missing data
  • These are not true values and must be converted
  • If left unchanged, they distort averages and regressions
  • Always read codebooks when available

Filling and Dropping Missing Values

  • fillna() replaces missing values using a chosen rule
  • dropna() removes rows or columns with missing values
  • The best choice depends on the context and research design
  • Never hide missing-value decisions from your audience

Fixing Data Types

  • Imported columns often arrive as text instead of numbers
  • pd.to_numeric() converts them safely
  • Wrong types can break calculations and sorting
  • Data types should be checked before modelling

Agricultural Case Study: District Crop Table

  • This type of integrated district table is common in applied project work
  • Each row is a district-crop observation
  • Columns mix agricultural and financial variables
  • Such tables are ideal for descriptive analysis and simple models

Deriving Useful Indicators

  • Derived indicators often communicate better than raw totals
  • Unit conversion should be explicit and documented
  • Ratios can help compare districts of different sizes when the units are labelled explicitly
  • Good data cleaning leads naturally into useful feature creation

A Basic Quality Checklist

  • Are all variables in the right units?
  • Are missing values coded consistently?
  • Are district and year labels standardised?
  • Are impossible values removed or investigated?
  • Is every cleaning decision reproducible in code?

In econometrics, data cleaning is not “boring preparation” — it is part of the research design.

Loading Excel Files in Practice

  • pandas usually imports Excel with pd.read_excel("file.xlsx")
  • In real projects, you may specify sheet_name= and skiprows=
  • Excel files often hide merged cells, extra headers, and formatting issues
  • The cleaning ideas from this lecture still apply after import
  • Always save cleaned outputs as reproducible code, not only as edited spreadsheets

Exercise

Create a DataFrame with columns district, yield_kg, and deposit_crore.

  1. Put one missing value in yield_kg.
  2. Use fillna() with the column mean.
  3. Create a new column yield_tonnes.
  4. Print the cleaned DataFrame.

Summary

  • ✅ Real datasets often include missing values and messy codes
  • ✅ StringIO is a convenient way to simulate CSV loading in class
  • ✅ .isna(), fillna(), and dropna() support missing-data workflows
  • ✅ Type conversion is essential before numerical analysis
  • ✅ Inline Kerala agricultural tables can illustrate realistic cleaning problems
  • ✅ Reproducible cleaning code is part of sound econometric practice

Next Lecture

Matplotlib Foundations

  • We now turn cleaned data into clear visual evidence
  • You will learn the basic grammar of figures, axes, and chart types
  • Strong visualisation makes your descriptive analysis persuasive