Debugging Production Issues: Logging, Tracing, and Reproduction Strategies

Editorial Team · · 3 min read

The 3 AM Page: Where Debugging Really Starts

There's a specific kind of dread that comes with a production alert at 3 AM. You're bleary-eyed, the dashboard is red, and users are complaining on Twitter. The code that's failing was written six months ago by someone who's since left the company. You've got maybe 30 minutes before the VP of Engineering starts asking questions.

This is where your debugging setup either saves you or fails you. And by "setup," I don't mean your IDE — I mean the logging, tracing, and observability infrastructure you built (or didn't build) months ago.

Structured Logging That Actually Helps

The single most impactful thing you can do for production debugging is switch from unstructured to structured logging. Instead of this:

logger.info(f"Processing order {order_id} for user {user_id}")
logger.error(f"Failed to charge card: {str(e)}")

Do this:

logger.info("processing_order", extra={
    "order_id": order_id,
    "user_id": user_id,
    "amount_cents": order.total,
    "payment_method": order.payment_type,
})

logger.error("charge_failed", extra={
    "order_id": order_id,
    "error_type": type(e).__name__,
    "error_message": str(e),
    "payment_provider": "stripe",
    "idempotency_key": idem_key,
})

Structured logs (JSON format) let you query specific fields in your log aggregator. "Show me all charge_failed events where payment_provider is stripe and amount_cents > 10000 in the last hour" — that's a query you can run in Datadog, Loki, or CloudWatch Logs Insights. You can't do that with grep on freeform text.

What to Log

Every request should carry a correlation ID (often called request_id or trace_id) that propagates through all service calls. When something breaks, you search for that one ID and get the complete picture across every service the request touched.

Log at service boundaries:

  • Incoming requests (method, path, relevant params — not passwords or tokens)
  • Outgoing calls to other services or databases (duration, status)
  • Decisions made by business logic ("applied discount X because condition Y")
  • Errors with full context (what was the input? what was the state?)

Don't log inside tight loops. Don't log entire request bodies (PII risk and storage cost). Don't log at DEBUG level in production unless you have dynamic log level control.

Distributed Tracing

When a single user action fans out to five microservices, logs alone aren't enough. You need traces.

OpenTelemetry is the standard now. It replaced OpenTracing and OpenCensus, and it's supported by every major observability vendor. The basic setup with Python:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

@tracer.start_as_current_span("process_payment")
def process_payment(order):
    span = trace.get_current_span()
    span.set_attribute("order.id", order.id)
    span.set_attribute("order.amount", order.total)
    # ... your code here

The key insight with tracing: instrument your service boundaries and slow operations first. Database queries, HTTP calls to other services, cache lookups, queue publish/consume. You don't need to trace every function — just the ones that talk to something external.

Reproduction Strategies

The hardest bugs to fix are the ones you can't reproduce. Here are strategies that work:

Request replay. If you log full request details (sanitized), you can replay the exact request that caused the failure against a staging environment. Some teams keep a "suspicious request" queue that automatically captures requests that resulted in 5xx errors.

Feature flags as debugging tools. If a bug appears after a deploy, toggle off recent feature flags one by one to isolate which change caused it. This is faster than git bisect on a running system.

Database snapshots. Many production bugs depend on specific data states. Having the ability to snapshot a subset of production data (anonymized) and load it into a staging database is incredibly valuable. I've seen teams spend days trying to reproduce a bug locally that was trivially reproducible with the right data.

Chaos engineering records. If you run chaos experiments (injecting latency, killing processes), keep records of what you tested and what broke. When a similar failure happens in production, you've already got a runbook.

The Debugging Checklist

When you get that 3 AM page, run through this:

  1. What changed? Check recent deploys, config changes, and infrastructure events. 80% of production incidents are caused by a recent change.
  2. What's the blast radius? Is it all users or a subset? One region or all regions? This tells you where to look.
  3. Can you mitigate before you diagnose? Sometimes rolling back or toggling a feature flag buys you time to investigate properly.
  4. Get the trace ID from an affected request and follow it through your systems.
  5. Check the boring stuff: disk space, memory, connection pool exhaustion, certificate expiry. These cause a surprising number of outages.

The goal isn't to fix the bug at 3 AM. The goal is to mitigate the impact and collect enough information to fix it properly during business hours.