LLMSkillHub

atharvhogade12 / india-open-data-cleaner

Cleans messy Indian government open-data CSV and XLSX files (data.gov.in, state portals, ministry, census, rainfall, or health tables). Use when the user asks to "clean this dataset", "fix Indian numbers", "prepare data.gov.in file", or shares tabular data with title rows above the header, lakh-style numbers (1,23,456), NA/--/N.A. placeholders, Orissa/Uttaranchal names, dd/mm/yyyy dates, or Total rows mixed in the data. Not for general CSV analysis, charts, or non-Indian data.

dataindian-dataopen-datadata-cleaningdata-gov-incsv
0
SKILL.md
2.9 KB

India Open-Data Cleaner

Never read or print the raw data file directly into the context window; reading large datasets wastes tokens and risks truncation. Instead, run the cleaning script:

python scripts/clean.py FILE --out cleaned

Read only the concise console summary printed by the script. If the output displays REVIEW: lines, open cleaned/changes.md to review the flagged items and ask the user before taking action. Otherwise, report the execution summary and the generated output files to the user.

Output Files

The script writes all results to the designated output directory (./cleaned by default):

  • cleaned.csv: Analysis-ready dataset with cleaned numbers, ISO dates, snake_case headers, and standardized state names.
  • removed_rows.csv: Contains moved Total, duplicate, and entirely blank rows with an explicit reason column (never silently deleted).
  • changes.md: Comprehensive audit log recording every normalization with row counts and sample values.

Core Rules

  1. No Silent Imputation: Never impute, guess, or fill missing values with statistical means. Placeholders become empty cells. Offer imputation only if the user explicitly requests it, naming the method.
  2. Data Preservation: Never delete rows silently. All removed summary or total rows are segregated into removed_rows.csv.
  3. Number & Date Formats: Indian number formats (1,23,456) and currency/percentages (₹, Rs., %) are parsed to standard numbers. Ambiguous dates are parsed day-first (Indian convention) and flagged under REVIEW:.
  4. Entity Renaming: Historical state/UT names (Orissa, Uttaranchal) map to current names (Odisha, Uttarakhand), keeping the original values in <col>_original.
  5. Unit Preservation: Do not convert units (lakh, crore, hectare). Read references/india-quirks.md if an unfamiliar unit appears.
  6. Error Handling: On script error, report the error line once; do not retry blindly more than once.

Command flags supported: --sheet N (select XLSX sheet), --monthfirst (override date order), --keep-totals (keep Total rows in place).