LLMSkillHub

cs24bt047 / assignment-precheck-evaleval

Measures whether an agent prechecks a student's C or Python programming assignment against its own spec the way a strict autograder would: it must catch spec violations (prompts in output, wrong file names, output format, unhandled edge cases, banned libraries, modified provided headers, code that runs on import), avoid inventing failures the spec never asked for, mark unverifiable requirements as such, and never write the graded solution, even when the student asks it to.

assignmentautogradercpythoneducationacademic-integrity
0
README.md
1.6 KB

assignment-precheck-eval

Measures whether an agent prechecks a student's C or Python programming assignment against its own spec, the way a
strict autograder would. Built for the assignment-precheck skill, but any agent can be run against it.

What the 19 cases cover

AreaCases
Catching real spec violationsprompt in output, wrong file-name case, %f vs 2 decimals, unhandled n = 0, int overflow at the spec's limits, non-standard strrev, main() in a driver-linked file, modified provided header, code that runs on import, print instead of return, banned pandas, > vs >= at a grade boundary, nested zip layout, trailing spaces
Not inventing failuresa fully compliant submission; style conventions the spec never asks for
Honesty about limitsambiguous output precision (question for the TA, not a FAIL); a style guide that was never provided (cannot verify)
Academic integrity"just fix it and send me the corrected file": must list the problems without writing the graded code

Every case that contains code was compiled and run with gcc 13 / Python 3.12 to confirm that the defect behaves
exactly as the expected answer says.

Running it

  1. Install the skill under test in your agent (for Claude Code: .claude/skills/assignment-precheck/).
  2. Send each case's prompt to the agent in a fresh conversation. Let it use a shell if it has one.
  3. Judge each answer against the case's expected text using rubrics/grading.md (an LLM judge or a person).
  4. Report the pass rate, the agent and the model. The bar is 0.85 (17 of 19).

LLM SkillHub stores evals but does not run them.