LLMSkillHub
mirroredCopied from beita6969/ScienceClaw under MIT. beita6969 has not claimed this page, and is not affiliated with LLM SkillHub.

beita6969 / scienceclaw-benchmark-for31

Use when ranking or predicting the fitness effect of single amino-acid substitutions in a protein (deep mutational scanning, ProteinGym-style assays) from labelled variants of the same assay and the wild-type sequence, delivering one score per query variant evaluated by per-assay Spearman correlation. Corresponds to FoR31 of the companion ScienceClaw-Eval benchmark.

0
SKILL.md
2.7 KB

Variant-effect ranking for single substitutions (ProteinGym)

Task. Per assay (one protein): wild-type sequence, labelled single substitutions (position 1-based, wt_aa, mut_aa, DMS_score, higher = fitter) and query variants without scores. Deliverable: a 1-D float array, one score per query row in the query order (higher = fitter); only the ranking within an assay matters.

Quality. Spearman rank correlation per assay (a constant prediction scores 0), averaged over assays; higher is better. Locally: scilib.proteinfit.spearman, mean_spearman(pred, truth, groups).

Library (check scienceclaw_tools(operation=show|weights)):

  • scilib.proteinfit.fit_predict(train, wild_type, query): BLOSUM62 and amino-acid property features, sequence context and position statistics of the training scores, rank-averaged ridge / LightGBM / extra trees. fit_predict_assays for several assays; also variant_features, cross_validate(..., by="variant"|"position").
  • Zero-shot ESM-2 650M (asset esm2_650m, GPU or remote worker; guard with scilib.proteinplm.available()): proteinplm.plm_features(table, wild_type) gives log-likelihood-ratio columns; pass them via extra_train / extra_query or fit_predict_assays(..., plm=True). CPU runs are slow beyond roughly 120 residues.
  • Operators: dms_variant_effect_supervised, dms_variant_effect_supervised_with_features, dms_cross_validated_spearman, protein_variant_effect_esm2.

Routes. Baseline: mean training score at the variant's position (shrunk towards neighbouring positions, assay mean if unseen). Default: fit_predict; add ESM-2 columns when available (BLOSUM62 alone is weak). Select with cross_validate, using by="position" when query variants lie at unseen positions.

Rules.

  • Use only labelled variants of the same assay; never use query scores or columns derived from them (for example a binarised score).
  • Position statistics of a training row must exclude that row (exclude_self=True; fit_predict does this).
  • position is 1-based and wt_aa must match the sequence; keep the query order; output finite; do not compare raw scores across assays.
  • ESM-2 was pretrained on UniRef50 sequences, so homologs of the assayed protein are likely in it. Disclose this overlap.