mitakshgea1000159 / india-data-formats-evaleval
Measures whether an agent validates, normalizes, formats and masks Indian identifiers, rupee amounts, dates and financial years correctly instead of guessing — 26 cases with deterministic must-contain / must-not-contain checks and per-category rubrics; companion to the india-data-formats skill.
india-data-formats-eval
26 cases that test whether an agent handles Indian identifiers, currency, dates and PII correctly. Companion to the india-data-formats skill.
manifest.yaml what is measured, judging rule, pass threshold (85%)
dataset.jsonl one case per line: id, category, input, expected, must_contain, must_not_contain, rubric
rubrics/ prose judging criteria per category
run_eval.py grader: --prompts | --check | --grade <responses.jsonl> [--baseline <other.jsonl>]
with_skill.jsonl agent replies WITH the skill (Claude Fable 5.1 in Claude Code-style agent, 26 Oct 2026)
baseline.jsonl agent replies WITHOUT the skill (same model, same prompts, no skill files readable)
results_*.json graded outputResults (26 Oct 2026)
| run | identifier | currency | dates | privacy | generation | dataset | total |
|---|---|---|---|---|---|---|---|
| with skill | 13/13 | 5/5 | 4/4 | 2/2 | 1/1 | 1/1 | 26/26 (100%) |
| baseline (no skill) | 12/13 | 4/5 | 4/4 | 1/2 | 1/1 | 0/1 | 23/26 (88%) |
Baseline failures: called ABCDE1234F "structurally valid" (pan-01); only partially masked a PAN (ABCPSXXXXF, exposing 6 of 10 chars) in a message meant for a public channel (privacy-01); printed full Aadhaar numbers from the CSV in the chat summary and produced no cleaned file (dataset-01). Every baseline pass that involved a checksum came from the model writing its own Verhoeff/mod-36 code on the fly — which is exactly the repeated token spend the skill removes.
Reproduce
python3 run_eval.py --check # dataset ground truth agrees with the skill's scripts
python3 run_eval.py --prompts --json > prompts.jsonl
# run each prompt through your agent with / without the skill, save {"id","response"} lines
python3 run_eval.py --grade my_with_skill.jsonl --baseline my_baseline.jsonlGrading rule: all must_contain patterns present, no must_not_contain pattern present (lines containing negation words like "not"/"avoid" are ignored for the forbidden check, so contrasting the wrong form is allowed), plus validator checks for generated identifiers.