ancoleman / debugging-techniques

Debugging Techniques

Purpose

Provides systematic debugging workflows for local, remote, container, and production environments across Python, Go, Rust, and Node.js. Covers interactive debuggers, container debugging with ephemeral containers, and production-safe techniques using correlation IDs and distributed tracing.

When to Use This Skill

Trigger this skill for:

• Setting breakpoints in Python, Go, Rust, or Node.js code
• Debugging running containers or Kubernetes pods
• Setting up remote debugging connections
• Safely debugging production issues
• Inspecting goroutines, threads, or async tasks
• Analyzing core dumps or stack traces
• Choosing the right debugging tool for a scenario

Quick Reference by Language

Python Debugging

Built-in: pdb

# Python 3.7+
def buggy_function(x, y):
    breakpoint()  # Stops execution here
    return x / y

# Older Python
import pdb
pdb.set_trace()

Essential pdb commands:

• list (l) - Show code around current line
• next (n) - Execute current line, step over functions
• step (s) - Execute current line, step into functions
• continue (c) - Continue until next breakpoint
• print var (p) - Print variable value
• where (w) - Show stack trace
• quit (q) - Exit debugger

Enhanced tools:

• ipdb - Enhanced pdb with tab completion, syntax highlighting (pip install ipdb)
• pudb - Terminal GUI debugger (pip install pudb)
• debugpy - VS Code integration (included in Python extension)

Debugging tests:

pytest --pdb  # Drop into debugger on test failure

For detailed Python debugging patterns, see references/python-debugging.md.

Go Debugging

Delve - Official Go debugger

Installation:

go install github.com/go-delve/delve/cmd/dlv@latest

Basic usage:

dlv debug main.go              # Debug main package
dlv test github.com/me/pkg     # Debug test suite
dlv attach <pid>               # Attach to running process
dlv debug -- --config prod.yaml  # Pass arguments

Essential commands:

• break main.main (b) - Set breakpoint at function
• break file.go:10 (b) - Set breakpoint at line
• continue (c) - Continue execution
• next (n) - Step over
• step (s) - Step into
• print x (p) - Print variable
• goroutine (gr) - Show current goroutine
• goroutines (grs) - List all goroutines
• goroutines -t - Show goroutine stacktraces
• stack (bt) - Show stack trace

Goroutine debugging:

(dlv) goroutines                 # List all goroutines
(dlv) goroutines -t              # Show stacktraces
(dlv) goroutines -with user      # Filter user goroutines
(dlv) goroutine 5                # Switch to goroutine 5

For detailed Go debugging patterns, see references/go-debugging.md.

Rust Debugging

LLDB - Default Rust debugger

Compilation:

cargo build  # Debug build includes symbols by default

Usage:

rust-lldb target/debug/myapp   # LLDB wrapper for Rust
rust-gdb target/debug/myapp    # GDB wrapper (alternative)

Essential LLDB commands:

• breakpoint set -f main.rs -l 10 - Set breakpoint at line
• breakpoint set -n main - Set breakpoint at function
• run (r) - Start program
• continue (c) - Continue execution
• next (n) - Step over
• step (s) - Step into
• print variable (p) - Print variable
• frame variable (fr v) - Show local variables
• backtrace (bt) - Show stack trace
• thread list - List all threads

VS Code integration:

• Install CodeLLDB extension (vadimcn.vscode-lldb)
• Configure launch.json for Rust projects

For detailed Rust debugging patterns, see references/rust-debugging.md.

Node.js Debugging

Built-in: node --inspect

Basic usage:

node --inspect-brk app.js       # Start and pause immediately
node --inspect app.js           # Start and run
node --inspect=0.0.0.0:9229 app.js  # Specify host/port

Chrome DevTools:

1. Open chrome://inspect
2. Click "Open dedicated DevTools for Node"
3. Set breakpoints, inspect variables

VS Code integration:
Configure launch.json:

{
  "type": "node",
  "request": "launch",
  "name": "Launch Program",
  "program": "${workspaceFolder}/app.js"
}

Docker debugging:

EXPOSE 9229
CMD ["node", "--inspect=0.0.0.0:9229", "app.js"]

For detailed Node.js debugging patterns, see references/nodejs-debugging.md.

Container & Kubernetes Debugging

kubectl debug with Ephemeral Containers

When to use:

• Container has crashed (kubectl exec won't work)
• Using distroless/minimal image (no shell, no tools)
• Need debugging tools without rebuilding image
• Debugging network issues

Basic usage:

# Add ephemeral debugging container
kubectl debug -it <pod-name> --image=nicolaka/netshoot

# Share process namespace (see other container processes)
kubectl debug -it <pod-name> --image=busybox --share-processes

# Target specific container
kubectl debug -it <pod-name> --image=busybox --target=app

Recommended debugging images:

• nicolaka/netshoot (~380MB) - Network debugging (curl, dig, tcpdump, netstat)
• busybox (~1MB) - Minimal shell and utilities
• alpine (~5MB) - Lightweight with package manager
• ubuntu (~70MB) - Full environment

Node debugging:

kubectl debug node/<node-name> -it --image=ubuntu

Docker container debugging:

docker exec -it <container-id> sh

# If no shell available
docker run -it --pid=container:<container-id> \
           --net=container:<container-id> \
           busybox sh

For detailed container debugging patterns, see references/container-debugging.md.

Production Debugging

Production Debugging Principles

Golden rules:

1. Minimal performance impact - Profile overhead, limit scope
2. No blocking operations - Use non-breaking techniques
3. Security-aware - Avoid logging secrets, PII
4. Reversible - Can roll back quickly (feature flags, Git)
5. Observable - Structured logging, correlation IDs, tracing

Safe Production Techniques

1. Structured Logging

import logging
import json

logger = logging.getLogger(__name__)
logger.info(json.dumps({
    "event": "user_login_failed",
    "user_id": user_id,
    "error": str(e),
    "correlation_id": request_id
}))

2. Correlation IDs (Request Tracing)

func handleRequest(w http.ResponseWriter, r *http.Request) {
    correlationID := r.Header.Get("X-Correlation-ID")
    if correlationID == "" {
        correlationID = generateUUID()
    }
    ctx := context.WithValue(r.Context(), "correlationID", correlationID)
    log.Printf("[%s] Processing request", correlationID)
}

3. Distributed Tracing (OpenTelemetry)

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def process_order(order_id):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)
        span.add_event("Order validated")

4. Error Tracking Platforms

• Sentry - Exception tracking with context
• New Relic - APM with error tracking
• Datadog - Logs, metrics, traces
• Rollbar - Error monitoring

Production debugging workflow:

1. Detect - Error tracking alert, log spike, metric anomaly
2. Locate - Find correlation ID, search logs, view distributed trace
3. Reproduce - Try to reproduce in staging with production data (sanitized)
4. Fix - Create feature flag, deploy to canary first
5. Verify - Check error rates, review logs, monitor traces

For detailed production debugging patterns, see references/production-debugging.md.

Decision Framework

Which Debugger for Which Language?

Which Technique for Which Scenario?

Production Debugging Safety Checklist

Before debugging in production:

• [ ] Will this impact performance? (Profile overhead)
• [ ] Will this block users? (Use non-breaking techniques)
• [ ] Could this expose secrets? (Avoid variable dumps)
• [ ] Is there a rollback plan? (Git branch, feature flag)
• [ ] Have we tried logs first? (Less invasive)
• [ ] Do we have correlation IDs? (Trace requests)
• [ ] Is error tracking enabled? (Sentry, New Relic)
• [ ] Can we reproduce in staging? (Safer environment)

Common Debugging Workflows

Workflow 1: Local Development Bug

1. Insert breakpoint in code (language-specific)
2. Start debugger (dlv debug, rust-lldb, node --inspect-brk)
3. Execute to breakpoint (run, continue)
4. Inspect variables (print, frame variable)
5. Step through code (next, step, finish)
6. Identify issue and fix

Workflow 2: Test Failure Debugging

Python:

pytest --pdb  # Drops into pdb on failure

Go:

dlv test github.com/user/project/pkg
(dlv) break TestMyFunction
(dlv) continue

Rust:

cargo test --no-run
rust-lldb target/debug/deps/myapp-<hash>
(lldb) breakpoint set -n test_name
(lldb) run test_name

Workflow 3: Kubernetes Pod Debugging

Scenario: Pod with distroless image, network issue

# Step 1: Check pod status
kubectl get pod my-app-pod -o wide

# Step 2: Check logs first
kubectl logs my-app-pod

# Step 3: Add ephemeral container if logs insufficient
kubectl debug -it my-app-pod --image=nicolaka/netshoot

# Step 4: Inside debug container, investigate
curl localhost:8080
netstat -tuln
nslookup api.example.com

Workflow 4: Production Error Investigation

Scenario: API returning 500 errors

# Step 1: Check error tracking (Sentry)
# - Find error details, stack trace
# - Copy correlation ID from error report

# Step 2: Search logs for correlation ID
# In log aggregation tool (ELK, Splunk):
# correlation_id:"abc-123-def"

# Step 3: View distributed trace
# In tracing tool (Jaeger, Datadog):
# Search by correlation ID, review span timeline

# Step 4: Reproduce in staging
# Use production data (sanitized) if needed
# Add additional logging if needed

# Step 5: Fix and deploy
# Create feature flag for gradual rollout
# Deploy to canary environment first
# Monitor error rates closely

Additional Resources

For language-specific deep dives:

• references/python-debugging.md - pdb, ipdb, pudb, debugpy detailed guide
• references/go-debugging.md - Delve CLI, goroutine debugging, conditional breakpoints
• references/rust-debugging.md - LLDB vs GDB, ownership debugging, macro debugging
• references/nodejs-debugging.md - node --inspect, Chrome DevTools, Docker debugging

For environment-specific patterns:

• references/container-debugging.md - kubectl debug, ephemeral containers, node debugging
• references/production-debugging.md - Structured logging, correlation IDs, OpenTelemetry, error tracking

For decision support:

• references/decision-trees.md - Expanded debugging decision frameworks

For hands-on examples:

• examples/ - Step-by-step debugging sessions for each language

Related Skills

For authentication patterns, see the auth-security skill.
For performance profiling (complementary to debugging), see the performance-engineering skill.
For Kubernetes operations (kubectl debug is part of), see the kubernetes-operations skill.
For test debugging strategies, see the testing-strategies skill.
For observability setup (logging, tracing), see the observability skill.