Import, Export & APIs

ECON 3209 · Week 6, Lecture 1 · Kerala Agricultural University

Department of Development Economics, KAU

Autumn 2026

Learning Outcomes

  1. Use pd.read_csv(), pd.read_excel(), and pd.read_json() to bring data into Python.
  2. Export cleaned data with to_csv() for future analysis and reporting.
  3. Understand the logic of API calls and JSON responses in applied economics.
  4. Recognise common import problems such as bad delimiters, headers, and data types.
  5. Build a simple data workflow using Kerala agriculture and India-level public data examples.
  6. Navigate MOSPI Unit Data and eSankhyiki portal to access Indian micro and macro data.

Why data import matters

  • Econometrics begins with data, not with estimation.
  • Real-world datasets arrive as CSV files, Excel sheets, JSON files, and API responses.
  • Agricultural economists often combine farm surveys, market prices, rainfall, and banking records.
  • A careful import step prevents errors from spreading into regression analysis.
  • Good data management saves time when assignments become larger.

Common file formats in practice

  • CSV: simple, portable, and common for field surveys.
  • Excel: frequent in offices, banks, and administrative records.
  • JSON: common when data come from websites and APIs.
  • Export files let us share reproducible datasets after cleaning.
  • In Kerala, we may merge crop yield data, district rainfall, and cooperative loan records.

Reading a CSV file with pandas

Important import options

  • sep= changes the delimiter when a file uses ; instead of ,.
  • header= tells pandas where variable names are located.
  • na_values= helps identify missing values such as NA, -, or missing.
  • usecols= keeps only needed variables.
  • parse_dates= is useful for time-series data such as monthly rubber prices.

Excel, JSON, and exporting data

  • pd.read_excel() is common for cooperative bank branch records.
  • pd.read_json() is useful when data arrive in nested web formats.
  • df.to_csv() stores a clean version for later econometric work.
  • Exporting cleaned data improves reproducibility and transparency.
  • Always check whether row labels should be saved with index=False.

Reading JSON and exporting to CSV

What is an API?

  • An API is a structured way to request data from a server.
  • Instead of downloading a file manually, Python can ask for current information.
  • API responses often arrive in JSON format.
  • Economists use APIs for prices, inflation, rainfall, or macro indicators.
  • The workflow is: send request \(\rightarrow\) receive response \(\rightarrow\) convert to DataFrame.

API call pattern in Python

A simple import-clean-export workflow

  • Step 1: import raw data from CSV, Excel, JSON, or API.
  • Step 2: inspect the first rows using head().
  • Step 3: check variable types with info().
  • Step 4: clean names, missing values, and formats.
  • Step 5: export a stable analysis file with to_csv().

A good rule: never run a regression on data you have not inspected.

Building a small workflow in code

Common problems when importing data

  • Numbers may be stored as text because of commas or symbols.
  • Headers may be repeated or shifted down by one row.
  • Missing values may appear as blank cells, -, or n.a..
  • JSON data may be nested and need flattening.
  • API calls may fail because of bad URLs, rate limits, or internet problems.

Applied example: Kerala data sources

  • Crop data may come from district agriculture offices.
  • Cooperative credit figures may come from annual reports in Excel.
  • India-level macro indicators may come from RBI or World Bank APIs.
  • A research project often combines several sources in one master dataset.
  • Careful import choices affect every later econometric conclusion.

MOSPI: Two Python Packages for Indian Data

Package Install What it gives you
mospi-unitdata pip install mospi-unitdata Unit-level microdata — download raw NSS, PLFS, ASI datasets from microdata.gov.in
mospi-esankhyiki pip install mospi-esankhyiki Aggregate indicators — 500+ statistical series across 22 datasets (PLFS, CPI, IIP, NAS, WPI …)

Both packages are official MOSPI/NSO tools (MIT licence). mospi-unitdata requires a free API key; mospi-esankhyiki is fully open.

Run these in a local Python environment (Jupyter / terminal) — they call live government APIs that are not available inside the browser sandbox.

mospi-unitdata: Downloading Microdata

# pip install mospi-unitdata
from MospiUnitdata import list_datasets, list_files, download_file

API_KEY = "YOUR_KEY"   # get free key at microdata.gov.in → Profile

# 1. Search available surveys
datasets = list_datasets(API_KEY, query="labour force")
for d in datasets:
    print(f"{d['idno']}: {d['title']}")
# DDI-IND-NSO-PLFS-2023-24: Periodic Labour Force Survey 2023-24
# DDI-IND-NSO-PLFS-2022-23: Periodic Labour Force Survey 2022-23

# 2. See files in a dataset
files = list_files("DDI-IND-NSO-PLFS-2023-24", API_KEY)
for f in files:
    print(f['name'], f.get('size', '?'))

# 3. Download to your project
download_file("DDI-IND-NSO-PLFS-2023-24",
              "PLFS_2023_24_Visit1_CSV.zip", "./data", API_KEY)

Once downloaded: pd.read_csv("./data/PLFS_2023_24_Visit1_CSV.zip") — one row per household member.

mospi-esankhyiki: Statistical Indicators

# pip install mospi-esankhyiki
import esankhyiki

# Step 1 – discover the 22 datasets
datasets_df = esankhyiki.list_datasets(format="df")

# Step 2 – see indicators for PLFS (Periodic Labour Force Survey)
indicators = esankhyiki.get_indicators("PLFS")

# Step 3 – find valid filter codes (years, states, sectors …)
meta = esankhyiki.get_metadata("PLFS", indicator_code=3, frequency_code=1)

# Step 4 – fetch as DataFrame
df = esankhyiki.get_data("PLFS", {
    "indicator_code": 3,   # Unemployment Rate
    "frequency_code": 1,   # Annual
    "year": "2023-24",
    "state_code": 99,      # All India
    "gender_code": 3,      # Persons (all)
    "age_code": 1,
    "sector_code": 3,      # Rural + Urban
}, format="df")
print(df)

Other useful datasets: "CPI", "NAS" (GDP), "IIP", "HCES" (consumption/poverty), "ASI", "WPI"

Working with esankhyiki output in Python

Once you call esankhyiki.get_data(..., format="df") locally, the DataFrame has this same structure — ready for regression or visualisation.

Exercise

A cooperative bank officer gives you a semicolon-separated file with columns branch, loans, and recovery_rate. Write code to import the data, convert loans to numeric, and export a clean CSV preview without row numbers.

Summary

  • ✅ CSV, Excel, JSON, and APIs are the main gateways into econometric work.
  • ✅ pd.read_csv(), pd.read_excel(), and pd.read_json() are core import tools.
  • ✅ to_csv(index=False) helps save a clean analysis-ready dataset.
  • ✅ APIs return structured data that can be turned into pandas DataFrames.
  • ✅ Import mistakes in headers, delimiters, or types can distort later results.
  • ✅ mospi-unitdata — downloads raw NSS/PLFS/ASI unit-level microdata from microdata.gov.in via Python API.
  • ✅ mospi-esankhyiki — fetches 500+ aggregate indicators (PLFS, CPI, NAS, IIP …) directly as DataFrames in 4 steps.
  • ✅ A reproducible import-clean-export workflow is essential in applied economics.

Next Lecture

  • We move from importing data to exploratory data analysis (EDA).
  • We will summarise variables, inspect distributions, and study relationships visually.
  • The goal is to understand the data before formal econometric modelling begins.