wshobson / trace-to-training-data

Trace To Training Data

This skill assumes eval-harness-first
already graded the traces being
converted here — goldens, graders,
and runs/<run-id>/results.json
all exist before conversion
starts. This is the flywheel edge
that skill names in its own flow:
"the same labeled traces become
the training set." Conversion
happens here; grading already
happened upstream.

Input: graded traces —
eval/goldens.jsonl plus
runs/<run-id>/results.json, each
row carrying a task_id, a
verdict from the grader, and a
reward when the task supports a
scalar score (judge score,
execution partial-credit, or an
RLVR verifier):

{"task_id": "t-042", "trace_id": "t-042-a3",
 "messages": [{"role": "user", "content": "..."}],
 "verdict": "pass", "reward": 0.91,
 "grader": "exact_match"}

Output format: rows shaped
exactly like dataset-curation's
Format Selection table — SFT
messages rows or DPO
prompt/chosen/rejected
pairs — so this skill's output is
that skill's input with no
reshaping step in between.

The Principle

The eval harness already did the
labeling work: every trace in
results.json carries a verdict,
and often a reward, before this
skill ever touches it. Converting
a graded trace into a training
row is mechanical — pick a shape
from dataset-curation's table,
map fields, write JSONL.
**Curation is the work that
remains** — which traces clear a
quality bar, which pairs are
informative, and which rows must
never enter the training set at
all.

Treat any conversion step that
requires re-judging a trace as a
sign the harness is missing a
grader, not a gap this skill
should paper over. A trace with
no verdict or reward isn't
convertible yet — route it back
to eval-harness-first first,
don't hand-label it here to
unblock conversion.

SFT From Traces

• **Keep the top-reward fraction
  of successful trajectories**,
  not every passing one. Rank
  passing traces by reward and
  take a fraction (the
  Agent-lightning pattern) rather
  than every trace that merely
  cleared the pass bar — a trace
  that barely passed is a weaker
  SFT signal than one that scored
  well above threshold.

• **Expert-corrected failures
  become gold SFT examples
  directly** (the Langfuse
  pattern) — when a human edits a
  failing trace's output into a
  correct one, that correction
  needs no reward threshold; a
  human already validated it.
  Route corrections straight into
  the SFT set.

• **Step-level masking beats
  whole-trajectory discard for
  multi-step traces.** When only
  some steps in a multi-step
  trajectory are bad, mask the
  loss on the bad steps and keep
  the good ones, rather than
  discarding the whole trajectory.
  SRFT reports 32.2% vs. 30.9% on
  SWE-bench for step-level critic
  masking over trajectory discard
  — a real, if modest, gap from
  the finer-grained cut.

Preference Pairs From Traces

• **Build pairs from
  passing-vs-failing trajectories
  on the SAME task**, never from
  unrelated best- and
  worst-scoring traces pulled
  across different tasks —
  cross-task pairs teach the
  model to prefer one task over
  another, not one response over
  another.

• **Select the rejected member at
  μ−2σ of the reward distribution
  for that task, never the
  absolute minimum.**
  preference-optimization's
  Pair Construction section owns
  the full selection formula;
  this skill supplies the graded
  trajectories it consumes.

• **Judge-scored delta selection
  cuts pair volume without
  cutting signal.** Score each
  candidate pair by
  chosen-minus-rejected judge
  delta and keep only the
  highest-delta subset — the top
  5k of a 16.5k candidate pool
  matched the full pool's
  downstream result. Build the
  full candidate set first, then
  filter by delta; don't cap
  generation at 5k up front.

Hygiene

• **Scan for secrets and PII before any row ships,
  and redact what's found.** Traces sourced from
  production logs can carry credentials, API keys,
  tokens, or customer data — run a secret/PII scan
  over every SFT and DPO row and redact matches;
  conversion fails closed (the row is dropped, not
  shipped with the raw content) if sensitive fields
  remain after redaction. Never commit secrets.

• **Eval goldens must never leak
  into training data.** Hold
  every eval/goldens.jsonl ID
  out of every converted SFT and
  DPO set — a trace that also
  appears as a golden trains on
  the exact item the checkpoint
  gets graded against later,
  silently inflating every
  subsequent eval run.

• **Dedup against the training
  set**, not just within the
  newly converted rows —
  exact-match or
  embedding-similarity, matching
  dataset-curation's dedup
  method field, run against
  whatever training data already
  exists before this batch merges
  in.

• **Provenance goes into the
  dataset card.** Every converted
  row must trace back to its
  source run_id and trace_id
  — dataset-curation's
  Provenance field checks for
  exactly this link back to
  trace-to-training-data
  output; a row with no traceable
  source isn't ready to merge.

Related Skills

• eval-harness-first — produces
  the graded traces this skill
  converts; a trace with no
  verdict or reward isn't
  convertible yet, route it back
  there before conversion.

• dataset-curation — owns the
  target formats and the dataset
  card this skill's provenance
  data feeds; converted rows must
  match its Format Selection
  table field names exactly, not
  an approximation of them.

• preference-optimization —
  consumes the DPO pairs this
  skill builds and owns the full
  μ−2σ rejection-selection
  formula referenced above.

Worked JSONL-to-JSONL conversions
— graded trace to SFT row, trace
pair to DPO pair, correction to
SFT row, the rejection-sampling
loop, and the goldens-holdout
check — live in
references/conversion-recipes.md.