ruvnet / gaia-submission

GAIA Submission Skill

Walk Claude Code through every step needed to go from a clean environment to a
signed, HAL-compatible submission package ready to upload to the Princeton
GAIA leaderboard.

When to use

When the user wants to:

• Run a benchmark and submit results to the HAL leaderboard
• Package an existing results file into a submission archive
• Confirm their environment is ready for a benchmark run

Prerequisites

Before starting, confirm these are available:

Phase 1 — Validate environment

# Run all pre-flight checks
/gaia validate

If any check fails, resolve it before continuing.

Phase 2 — Estimate cost and confirm

Ask the user for their configuration:

• Level (default: 1)
• Question limit (default: 53 for a quick run, 165 for the full L1 set)
• Models (default: claude-sonnet-4-6)
• Self-consistency voting (default: 1; use 3 for L2/L3)

/gaia cost --level=$LEVEL --limit=$LIMIT --models=$MODELS --voting=$VOTING

If projected cost > $5, show the estimate and ask: "This run will cost
approximately $X. Proceed? (y/N)"

Phase 3 — Run the benchmark

/gaia run --level=$LEVEL --limit=$LIMIT --models=$MODELS --voting=$VOTING

While running, progress is reported every 5 questions:

[12/53] 22.7% (5 passed of 22 scored) — est. remaining: $0.18

Store the run summary in memory for history tracking:

npx @claude-flow/cli@latest memory store \
  --namespace gaia-runs \
  --key "run-$(date +%Y%m%d-%H%M)" \
  --value '{"level":$LEVEL,"model":"$MODEL","total":$TOTAL,"passed":$PASSED,"pass_rate":$RATE,"est_cost_usd":$COST}'

Phase 4 — Package for submission

/gaia submit --results=~/.cache/ruflo/gaia/results-latest.json

This produces:

submission-<date>-<sha>/
├── results.jsonl        ← HAL-compatible, one JSON per line
├── trajectories.jsonl   ← full agent traces
├── metadata.json        ← harness info, model, tool catalogue
├── audit-report.json    ← ADR-167 pre-submission exploit-audit report
├── manifest.md.json     ← Ed25519-signed witness (signs audit-report.json's hash)
└── README.md            ← human summary + leaderboard comparison

Integrity gate — the audit runs before signing (ADR-167)

Post-RDI (UC Berkeley broke 8 agent benchmarks — GAIA to ~98% — without solving
a task), a signature alone is not enough: **it proves the bytes are untampered,
not that the score was earned.** /gaia submit therefore runs a deterministic,
$0 exploit audit before signing and **refuses to build the leaderboard package
on a CRITICAL failure** unless --allow-dirty is passed. The audit report is
signed into the witness manifest as an ADR-103 fix marker, so a ruflo GAIA
submission attests both transport-integrity and earning-integrity.

If the gate blocks, treat it as a real finding — inspect audit-report.json
(answer-leakage, no-work pass, oracle leakage, grader monkey-patching, an
answer-key read outside the dataset dir, or dynamic eval/exec of task content in
the runner) rather than reaching for --allow-dirty. The static source-scan
family (answer-key-reads, dynamic-eval, judge-injection) enforces today with no
trajectory instrumentation; the trajectory-fed checks the current schema cannot
feed are reported as harness_gaps (ADR-167 §7), not passes.

Phase 5 — Compare and report

/gaia leaderboard --level=$LEVEL
/gaia history

Interpret the gap between ruflo's score and the leaderboard top-10.
Identify the primary failure mode (tool gap, reasoning miss, extraction bug)
using the /gaia-debugging skill if needed.

Phase 6 — Persist learnings

npx @claude-flow/cli@latest hooks post-task \
  --task-id "gaia-submission-$(date +%Y%m%d)" \
  --success true \
  --train-neural true

Store any discovered patterns:

npx @claude-flow/cli@latest memory store \
  --namespace gaia-patterns \
  --key "submission-notes-$(date +%Y%m%d)" \
  --value "Level $LEVEL, $MODEL: $NOTES"

Extensibility note

This skill is intentionally structured to be benchmark-agnostic. The phase
headers (validate → estimate → run → package → compare → learn) apply to
SWE-bench, WebArena, and HumanEval with only phase 3-4 details changing.