ai4s-research / traceability-review

Traceability Review

Audit a workspace document (report, manuscript, or notebook) with three checks.
You verify traceability — that claims trace to sources, data, and code —
not truth. Never state or imply that the document is error-free.

PDF manuscripts — extract first, never guess

If the document is a PDF, do not read the raw bytes or infer its contents.
Run the bundled extractor first — it pulls the text plus the concrete citation
identifiers and quantitative claims deterministically, so you audit real
identifiers, not ones recalled from memory:

python "$XDG_CONFIG_HOME/opencode/skills/traceability-review/pdf_extract.py" MANUSCRIPT.pdf

It prints JSON: `{backend, pages, chars, citations:{dois,arxiv,pmids},
claims:[{kind,text,context}], text}. Usecitations` as the input to Check 1,
claims as the input to Check 2, and text to locate figure references for
Check 3. If it returns {"error": …} (no PDF backend installed), say so plainly
and fall back to whatever text you can read — do not fabricate identifiers.

Check 1 · Citation audit

1. Extract every citation identifier from the document: DOI (10.xxxx/…),
   arXiv id, PMID, or title + year when no identifier is given.

2. Resolve each against a public registry (no API key needed):

   • DOI: curl -s "https://api.crossref.org/works/<doi>"
   • arXiv: curl -s "http://export.arxiv.org/api/query?id_list=<id>"
   • PMID: curl -s "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=<pmid>&retmode=json"

3. Findings:

   • error — the identifier does not resolve (HTTP 404 / empty result).

   • warn — it resolves, but the registry's title/authors/year clearly
     disagree with how the document cites it.

   • warn — network unavailable: report "could not verify (offline)" rather
     than skipping silently.

Check 2 · Untraceable numbers

1. List the document's quantitative claims: statistics, percentages, sample
   sizes, effect sizes, p-values, model scores.

2. For each, look for its source inside the workspace: a data file, a code or
   notebook output, or an execution log that produces that value.

3. Finding: warn for any number with no traceable source. Quote the exact
   sentence in the evidence.

Check 3 · Figure ↔ code consistency

1. Read .openscience/provenance.jsonl in the workspace — one JSON record per
   line: {path, version, ts, tool, content, …}; ts is epoch seconds. It
   records every file version the agent wrote. The directory is hidden: read
   the file directly (cat .openscience/provenance.jsonl) instead of relying
   on ls. Fall back to file mtimes only when the file is truly absent.

2. For each figure the document references:

   • Latest record ts for the figure file (fall back to file mtime when the
     figure has no record).

   • Latest record ts of the script/notebook that generates it — match by
     scanning record content and workspace code for the figure's filename.

3. Findings:

   • warn — the generating code has a newer version than the figure:
     "figure may be stale — regenerate it from the current code".

   • warn — a referenced figure has no provenance record and no matching
     workspace file.

Output contract

End the reply with exactly one fenced block (the app renders it as reviewer
cards; keep it as the LAST thing in the message):

{"findings":[{"level":"error","check":"citation","title":"DOI does not resolve","evidence":"10.9999/fake.2026 → Crossref 404"}],"note":"Traceability review — verified what could be traced. Absence of findings is not a guarantee of correctness."}

• level: error | warn | ok · check: citation | number | figure.

• One finding per issue; ok findings are allowed for confirmed traceable
  items worth stating explicitly.

• Evidence: the exact identifier / quoted sentence / file paths, plus what you
  observed.

• The note must never claim the document has no errors.