sp-suriyaprakash21 / ci-failure-triage

CI Failure Triage

Purpose

Diagnose GitHub Actions failures in Python and Node.js projects. Produce a concise, evidence-based diagnosis rather than guessing or suggesting unrelated changes.

This release focuses on triage and recommendations. Do not modify files, commit changes, push code, change repository settings, or access external systems unless the user explicitly requests and authorizes those actions.

Scope

Support these failure areas:

• dependency installation and lockfile problems;
• test discovery and test execution failures;
• linting and formatting failures;
• runtime or language-version mismatches;
• missing environment variables and configuration;
• GitHub Actions workflow structure, paths, permissions, and job dependencies.

Prioritize GitHub Actions logs and workflows for Python and Node.js projects. If the project uses another platform or language, state that it is outside the primary scope and provide only cautious general guidance.

Inputs

Use whatever evidence is available:

• CI log or error output;
• .github/workflows/*.yml or .yaml files;
• pyproject.toml, requirements.txt, poetry.lock, or Pipfile.lock;
• package.json, package-lock.json, yarn.lock, or pnpm-lock.yaml;
• runtime-version files such as .python-version, .nvmrc, or workflow setup steps;
• relevant source files and tests;
• the command that failed and its exit code.

If a required artifact is missing, continue only when the remaining evidence is sufficient. Otherwise, ask for the smallest missing artifact and explain why it is needed.

Operating rules

1. Read before diagnosing. Inspect the complete relevant error section, not only the final summary line.
2. Separate symptom from cause. Distinguish the command that failed from the earlier event that caused the failure.
3. Cite evidence. Quote short, relevant log lines or identify exact file and line locations when available.
4. Use calibrated confidence. Mark a diagnosis high, medium, or low confidence.
5. Keep hypotheses separate. State the primary hypothesis first and list alternatives only when they are plausible.
6. Recommend the smallest safe fix. Do not suggest disabling tests, deleting lockfiles, exposing secrets, or hiding failures with continue-on-error as a default fix.
7. Distinguish verified from unverified. Never claim that a fix works unless it was actually tested.
8. Protect secrets. Redact tokens, passwords, API keys, private keys, credential values, and sensitive environment-variable values from the report.
9. Avoid overreach. Do not infer repository permissions, secret values, external service health, or production impact without evidence.

Workflow

1. Establish the failure boundary

Classify where the failure occurred:

• dependency installation;
• build or compile;
• test collection/discovery;
• test execution;
• linting or formatting;
• service/container startup;
• deployment or release;
• workflow configuration or permissions.

If more than one phase failed, identify the earliest meaningful failure and treat later errors as possible cascading symptoms.

2. Inspect the workflow context

When available, inspect the relevant workflow for:

• operating system and runner;
• Python or Node.js version;
• dependency-installation command;
• cache configuration;
• working directory;
• environment-variable names, without revealing values;
• job dependencies and conditional expressions;
• permissions;
• the exact command that failed.

Do not assume that a workflow setting caused the failure merely because it is nearby in the file.

2a. Run deterministic pattern detection

When a log file is available, run the bundled detector before forming the final diagnosis:

python scripts/detect_failure_patterns.py --log <path-to-ci-log>

The script returns JSON containing likely categories, language hints, redacted evidence lines, and suggested checks. Treat this output as supporting evidence, not as a final diagnosis. Confirm the signals against the workflow and repository context.

The detector uses only Python's standard library and does not make network requests. If it cannot read the file or finds no known pattern, continue with manual inspection and report that limitation.

3. Form a diagnosis

For each important finding, connect:

Observed evidence -> likely mechanism -> practical impact

Prefer a direct root cause when the evidence supports one. If evidence is incomplete, state what is known, what is uncertain, and which check would distinguish the hypotheses.

4. Recommend remediation

Give:

• the smallest likely fix;
• the file or workflow step to inspect;
• a concrete example command or configuration change when safe;
• risks or compatibility considerations;
• an alternative if the primary fix is not appropriate.

Do not provide a destructive command without explaining its consequences. Do not recommend changing production configuration when a local or CI-only fix is sufficient.

5. Define verification

Provide the narrowest verification command that tests the diagnosis, such as:

• rerunning the dependency installation;
• running the failed test file or test selection;
• running the same lint command locally;
• checking the runtime version used by CI;
• validating the workflow YAML;
• rerunning the affected workflow after a reviewed change.

Mark verification as not run when execution is unavailable.

Required output

Use this report structure:

CI Failure Triage Report

Executive summary

State the failure phase, primary diagnosis, confidence, and whether the diagnosis was verified.

Failure classification

• Phase:
• Project/runtime:
• Workflow/job/step:
• First meaningful failure:

Primary diagnosis

• Root cause:
• Confidence: high | medium | low
• Evidence: Include concise log excerpts and/or file locations.
• Impact: Explain what failed and why.

Recommended fix

1. State the smallest safe change.
2. Identify the file or workflow step.
3. Include a command or short patch example when appropriate.

Verification plan

• Command/check:
• Expected result:
• Status: verified | not run | blocked

Alternative hypotheses

Include only plausible alternatives. For each, state the evidence needed to confirm or reject it. Omit this section when the diagnosis is clear.

Limitations and safety notes

Mention missing artifacts, unavailable execution, redacted values, or external systems that prevent complete verification.

Quality bar

Before finalizing the report, check that:

• the diagnosis is tied to evidence;
• the earliest meaningful failure was considered;
• confidence matches the evidence;
• recommendations are actionable and minimal;
• verification is explicit;
• secrets are not reproduced;
• no unrun command is described as successful.

Bundled scripts

scripts/detect_failure_patterns.py

Use this script to quickly classify common Python and Node.js CI log signals. It is intentionally conservative and deterministic. It does not execute the failed command, inspect a repository, or prove the root cause.

Supported signal groups include:

• dependency and lockfile failures;
• Python and Node.js runtime mismatches;
• test assertion and test-discovery failures;
• missing environment variables;
• lint and formatting failures;
• GitHub Actions permission and workflow errors;
• common network and service-startup failures.

The script accepts a file or standard input and emits JSON by default:

python scripts/detect_failure_patterns.py --log failure.log
cat failure.log | python scripts/detect_failure_patterns.py --stdin

For a human-readable demonstration, request text output:

python scripts/detect_failure_patterns.py --log failure.log --format text

The text report includes the primary category, numeric confidence, language hints, matched evidence, suggested checks, and a plain-language confidence explanation. The explanation describes whether the match is strong, broad, ambiguous, or unknown; it is not a statistical probability.

Never treat a detected category or confidence value as proof. Use the extracted evidence to guide the investigation.

Automated tests

Run the v1.0.0 test suite from the skill root:

python -m unittest discover -s tests -v

The tests cover representative Python and Node.js logs, primary-category selection, language hints, confidence explanations, secret redaction, unknown-log fallback, both output formats, and the command-line interface. Add a fixture and a focused test whenever a new detector signal is introduced.

End-to-end demonstration

For the competition demo, follow demo/end-to-end-demo.md. The architecture workflow is in demo/architecture.mmd.

Examples

Use the examples in the examples/ directory as style and reasoning references:

• examples/dependency-failure.md — peer-dependency conflict;
• examples/test-failure.md — assertion failure with uncertain business correctness;
• examples/missing-environment-variable.md — safe handling of configuration values;
• examples/ambiguous-failure.md — calibrated uncertainty for incomplete evidence.

Do not copy an example's diagnosis without checking the actual log and repository context.