LLMSkillHub
mirroredCopied from hoangsonww/Claude-Code-Agent-Monitor under MIT. hoangsonww has not claimed this page, and is not affiliated with LLM SkillHub.

hoangsonww / regression-watch

Detect quality and efficiency regressions over time using Agent Monitor data — rising error rate (APIError events), falling cache hit rate, growing compaction frequency, and climbing cost-per-session. Splits history into an earlier baseline window and a recent window and reports which metrics are getting worse, by how much, and where. Use when checking whether things are degrading or trending in the wrong direction.

0
SKILL.md
4.0 KB

Regression Watch

Detect whether Claude Code sessions are getting worse over time across quality and
efficiency metrics, using Agent Monitor data.

Input

The user provides: $ARGUMENTS

This may be:

  • empty or "all" — check every regression metric (default)
  • "errors" — error-rate regression only
  • "cache" — cache hit-rate regression only
  • "compaction" — compaction-frequency regression only
  • "cost" — cost-per-session regression only
  • A window like "last 30d" or "30 vs 90" — set the recent vs baseline window sizes

Data Sources

EndpointReturns
GET /api/analyticsdaily_events (365d), daily_sessions (365d), event_types, tokens (totalinput, totaloutput, totalcacheread, totalcachewrite — baselines pre-summed), avg_events_per_session
GET /api/events?session_id=XEvent stream incl. APIError, Compaction, PreToolUse/PostToolUse — used to localize regressions to specific sessions
GET /api/pricing/cost{ total_cost, breakdown[...] } — total cost to derive cost-per-session
GET /api/pricing/cost/{sessionId}Per-session cost — used to compare recent vs baseline session cost
GET /api/workflows/{sessionId}compaction (impact), errorPropagation (by depth), effectiveness — per-session quality signals
GET /api/sessions?limit=NSessions with started_at, cost, metadata — to bucket sessions into time windows

Report Sections

1. Windowing

Split history into a baseline window (older) and a recent window (newer).
Default: recent = last 30 days, baseline = the 30–90 day range before it. Use
daily_events/daily_sessions for series metrics and GET /api/sessions?limit=N
to assign sessions to each window by started_at.

2. Error Rate Regression

  • Recent error rate = APIError count / total events in the recent window
    (from event_types and daily_events, or per-session GET /api/events).

  • Compare to the baseline rate. Flag if recent is higher.

  • Report the absolute and relative change and which sessions contributed most
    APIError events.

3. Cache Hit Rate Regression

  • Cache hit rate = total_cache_read / (total_cache_read + total_input).

  • Compute for each window (per-window input/cache_read from session metadata or
    the pricing breakdown). Flag a falling hit rate — that means more
    uncached input tokens and higher cost.

4. Compaction Frequency Regression

  • Compaction frequency = Compaction events / session per window (from
    event_types / daily_events, confirmed via per-session
    GET /api/workflows/{id} compaction). Flag a rising rate — context is
    overflowing more often.

5. Cost-per-Session Regression

  • Cost-per-session = window total cost / window session count, using
    GET /api/pricing/cost overall and GET /api/pricing/cost/{id} for the
    sessions in each window. Flag a climbing value.

6. Verdict

Roll up which metrics regressed, rank by relative worsening, and name the most
likely driver (e.g., cache hit rate fell → cost per session climbed).

Output

  • A Markdown table: metric | baseline | recent | Δ | direction (▲ worse / ▼ better) | verdict.
  • Tag each regressed metric 🔴 (clear regression), 🟡 (mild/within noise), or 🟢 (improved).
  • Currency in USD to 4 decimals; rates as percentages to 2 decimals.
  • List the specific session IDs that contributed most to any regression.
  • End with the single highest-priority regression to address and a concrete next step.
  • Read-only: only report what the API returns; never fabricate baselines.