lyonzin / rag-troubleshoot

rag-troubleshoot — RAG-first debugging

When to use this skill

Trigger the moment the user reports:

• A specific error message, stack trace, or exception
• "It broke", "it fails", "not working", "returns null when it shouldn't"
• Unexpected behavior in a component
• Alert / incident triage
• "Why does X do Y" where Y is wrong
• A pasted log line
• A failed CI/CD run

The core insight: many bugs are already solved somewhere in your corpus — runbook, postmortem, incident report, prior fix commit, ADR, chat thread indexed via add_from_url. Search first.

What this skill commits to

Before proposing a fix, the agent searches for:

1. The error signature itself — exception name, first line of stack, unique error code
2. The affected component / function — module name, service name, feature
3. Prior incidents with similar symptoms — even if the error message differs

Only after those three come back empty does the agent apply general debugging techniques.

Steps

1. Extract error signatures from the user's message:

   • Exception class name (ValueError, ConnectionError, TimeoutError, etc.)
   • Error code (ERR_INVALID_TOKEN, E42_INDEX_MISS, HTTP status)
   • First distinctive line of the stack trace
   • Unique keywords ("segfault at ...", "cannot connect to ...")

2. First search — exact error signature:
   `
   searchknowledge(query="<exception name> <error keyword>", maxresults=5, min_score=0.15)
   `
   Example: search_knowledge(query="ConnectionError refused chromadb")

3. Second search — affected component + symptom:
   `
   searchknowledge(query="<component name> <what happens>", maxresults=5)
   `
   Example: search_knowledge(query="reindex fails halfway large corpus")

4. Third search — historical incidents / postmortems:
   `
   searchknowledge(query="incident postmortem <component>", maxresults=5)
   `
   Or with a category filter if your corpus categorizes by document type:
   `
   search_knowledge(query="<component> failure", category="incidents")
   `

5. Evaluate:

   Result	Action
   Prior fix exists	Apply it, cite the source, verify it still applies
   Related runbook exists	Follow the runbook, cite it
   Related incident, different fix	Read the incident, adapt
   Nothing found	Do general-purpose debugging, but flag that "this may be a new failure mode"

6. After solving, if the fix is non-obvious, chain to rag-index-decisions to add a short note back to the corpus so the next occurrence is one search away.

Examples

Example 1 — Prior fix

User: "Getting EmbeddingModelLoadError: sticky failure when starting knowledge-rag. What's happening?"

Agent:

search_knowledge(query="EmbeddingModelLoadError sticky failure") → 3 hits, top 0.91

Reply: "This is documented in [CHANGELOG.md#v381]. The `sticky _load_failed` flag was
added in v3.8.1 as a hotfix — when the ONNX model fails to load, subsequent calls
re-raise immediately instead of looping through HF Hub download attempts. The fix is
in [providers/embeddings/fastembed.py:145]. Root cause is usually one of:
  1. HF Hub unreachable at first startup (offline / DNS)
  2. Corrupt cached model in models_cache/
  3. Disk full during model download

Try: (a) check network to huggingface.co, (b) rm -rf models_cache/ and restart,
(c) df -h. If none apply, this may be a new failure mode."

Example 2 — Runbook

User: "Our search results are empty after nuclear rebuild. What now?"

Agent:

search_knowledge(query="empty results nuclear rebuild") → 4 hits
1st hit: docs/reindex-operations.md (0.87)

get_document(filepath="docs/reindex-operations.md")
→ Full runbook including "post-rebuild sanity check" section

Reply: "Per [docs/reindex-operations.md#post-rebuild-sanity-check], after a nuclear
rebuild you should:
1. Confirm `get_index_stats().chunks_count` > 0
2. Run one `search_knowledge('any indexed keyword')` — should return hits
3. If empty, check for stale FTS5 marker (v4.8.2+ specific) — delete
   `data/fts5_migration.state` and restart.
The most common cause is exactly the FTS5 stale marker case, added in v4.8.3
[CHANGELOG.md#v483]."

Example 3 — Nothing found, general debug

User: "My Rust binding for the MCP server segfaults on second call. Ideas?"

Agent:

search_knowledge(query="rust binding segfault MCP") → 0 hits
search_knowledge(query="rust FFI second call crash") → 0 hits
search_knowledge(query="rust bindings segfault") → 0 hits

Reply: "Nothing in the local corpus about Rust bindings (searched 'rust binding
segfault MCP' + 2 paraphrases, 0 hits). This appears to be a new failure mode.
General-purpose debugging directions:
  1. Rebuild with `RUST_BACKTRACE=full` to get the panic location
  2. Check FFI lifetime — is a Python object dropped while Rust still holds a ref?
  3. Try with `--single-instance` mode to rule out concurrency
  4. Reduce to a minimal repro: 1 call, 2 calls, does it always fail on N=2?

Once you find the root cause, worth indexing back — see rag-index-decisions."

Edge cases

• Very generic error ("KeyError") — 1st search will have low precision. Add the component name early or skip the exact-error search and start with component + symptom.
• Multi-line stack traces — extract the deepest custom frame (not stdlib) as the search seed.
• Error is intermittent / rare — still search, but weight the "prior incidents" query more heavily.
• User pasted only the symptom, not the error — ask for the error message + a stack trace before searching. Guessing wastes search calls.

Related skills

• rag-check-first — the parent skill (troubleshooting is a specialized variant).
• rag-cite-sources — when you propose a fix, cite the source that documented it.
• rag-index-decisions — after solving a novel bug, index the postmortem for next time.
• rag-web-fallback — for truly novel errors, escalate to GitHub / StackOverflow after RAG comes back empty.