Observability Stack: OpenTelemetry, Grafana, and Structured Logging Practices

Editorial Team · · 4 min read

Why "Just Add Logging" Isn't Observability

There's a pattern I see in almost every team that's "doing observability" — they've got a pile of logs in Elasticsearch, some Prometheus metrics they set up once and forgot about, and a Grafana dashboard with 47 panels nobody reads. That's monitoring infrastructure. It's not observability.

Observability means you can ask arbitrary questions about your system's internal state using external outputs. Logs, metrics, and traces are the raw materials. The tooling that ties them together — and the practices that make them useful — that's where it gets interesting.

OpenTelemetry: The Instrumentation Standard

Before OpenTelemetry (OTel), every observability vendor had their own SDK. Switching from Datadog to Jaeger meant re-instrumenting your entire codebase. OTel standardizes the collection layer — you instrument once, then send data wherever you want.

The architecture has three pieces:

  • API — defines what can be measured (spans, metrics, logs). Your application code depends only on this.
  • SDK — implements the API with sampling, batching, and export logic.
  • Collector — a standalone process that receives, processes, and exports telemetry data. This is where you route data to multiple backends.

For a Python service, basic tracing setup looks like:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("my-service")

The auto-instrumentation libraries are where OTel really shines. For Python, opentelemetry-instrument automatically patches Flask, Django, SQLAlchemy, requests, and dozens of other libraries. You get distributed traces across HTTP calls and database queries without touching application code.

The OTel Collector Pipeline

Running the Collector as a sidecar or gateway gives you a processing layer between your apps and backends. A typical config:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://mimir:9009/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

The memory_limiter processor is critical in production. Without it, a traffic spike can cause the Collector to OOM. I've seen this crash entire observability pipelines during the exact moments you need them most.

Structured Logging That Actually Helps Debug

Unstructured logs are nearly useless at scale. logger.info("Processing order " + order_id) becomes impossible to query when you've got 50,000 log lines per second.

Structured logging means every log entry is a machine-parseable event with typed fields:

import structlog

logger = structlog.get_logger()

logger.info("order.processed",
    order_id="ord-12345",
    customer_id="cust-789",
    total_amount=149.99,
    items_count=3,
    processing_time_ms=47
)

This produces JSON that's trivially filterable:

{"event": "order.processed", "order_id": "ord-12345", "customer_id": "cust-789", "total_amount": 149.99, "items_count": 3, "processing_time_ms": 47, "timestamp": "2026-03-15T10:23:45Z"}

A few hard-won lessons on structured logging:

  • Establish naming conventions early. user_id vs userId vs user.id — pick one and enforce it. Inconsistent field names make cross-service queries painful.
  • Include trace IDs and span IDs in every log line. This is how you correlate logs with traces — click a trace span in Grafana Tempo, see exactly what was logged during that operation.
  • Don't log request/response bodies by default. It's tempting for debugging but creates massive storage costs and potential PII exposure. Log them at debug level, behind a feature flag.
  • Set severity levels consistently across services. If one team's WARN is another team's ERROR, your alerting breaks.

Grafana as the Unified Frontend

Grafana's strength isn't any single visualization — it's that it can query Prometheus, Loki, Tempo, Elasticsearch, PostgreSQL, and dozens more from one interface. You build a dashboard with a latency graph from Prometheus, an error log panel from Loki, and a trace detail from Tempo, all correlated by the same time range.

The Grafana LGTM stack (Loki for logs, Grafana for visualization, Tempo for traces, Mimir for metrics) is the open-source answer to Datadog. It's significantly cheaper at scale — we're running it for ~$800/month on infrastructure that would cost $15,000/month on Datadog for equivalent data volume.

Dashboard Design That People Use

Most dashboards fail because they're built bottom-up — "here's everything we can measure." Useful dashboards are built top-down from questions operators actually ask during incidents:

  • Is the service healthy right now? (RED metrics: Rate, Errors, Duration)
  • Which endpoint is causing the problem? (per-route breakdown)
  • What changed? (deployment markers, config change annotations)
  • What's the blast radius? (affected users, regions, customer tiers)

I'd argue the single most valuable panel is p99 latency with deployment annotations overlaid. When latency spikes and you can see it started exactly when version 2.14.3 rolled out, you've cut your investigation time from hours to seconds.

Connecting the Three Pillars

The magic of observability isn't logs, metrics, or traces individually. It's the correlation between them. You see a latency spike in a metric, drill into exemplars to find specific slow traces, then jump to the logs for those trace IDs to see what went wrong.

OTel makes this correlation automatic when you propagate context correctly. Every HTTP call carries trace context in headers (traceparent in W3C format), so a request that touches five services produces a single connected trace. With structured logs emitting the same trace ID, the entire request lifecycle is one click away.

Getting this right takes discipline more than technology. Make sure every service uses the OTel SDK, every log includes trace context, and every dashboard links to trace exploration. The tooling supports it — the challenge is organizational consistency.