Set up observability: logs, metrics, traces

Picking the backend first prevents the worst pattern: instrument everything, then realize the vendor's pricing model penalizes the cardinality you committed to. Logs go in Loki / Datadog Logs / Sentry. Metrics go in Prometheus / Datadog. Traces go in Tempo / Honeycomb / Datadog APM. Pick a stack that covers all three; resist the urge to mix-and-match across vendors on day one.

Plain text logs are uncorrelatable. Structured JSON logs with a known schema are queryable, alertable, and aggregatable. Adopt a logging library that emits JSON by default, define 5-10 standard fields (timestamp, level, request_id, user_id, route, latency_ms), and prohibit free-form messages from carrying load-bearing data.

Read the codebase. Find every log call (console.log, logger.info, log.Info, etc.).

For each, propose a refactor that:
1. Uses the team's structured-logging library (ask the user which one).
2. Carries the standard fields (timestamp, level, request_id, user_id, route, event_name).
3. Replaces string templates with structured fields ("user logged in" becomes event: "login" + user_id).

Output as a list of file:line references with proposed replacements. Flag any log that contains PII (email, full name, IP) for review under your data classification.

Latency, traffic, errors, saturation. Every service needs them. Most observability tools give them to you for free if you instrument correctly: latency from request duration histograms, traffic from request counters, errors from status code counters, saturation from CPU / memory / queue depth. Build a dashboard with these four panels per service and call it the home base.

Tracing is the single biggest investigative tool you can add. A trace shows the path of one request through every service it touches, with timing per hop. OpenTelemetry's auto-instrumentation libraries cover most popular frameworks; you usually only need to add 5-10 manual spans for the important business logic.

Read the codebase and propose 10 manual span instrumentations.

For each:
1. The function or block to wrap.
2. The span name (use convention: "service.operation", e.g. "checkout.charge_card").
3. The attributes to set on the span (user_id, order_id, amount, etc.).

Prioritize: payment flows, signup, third-party API calls, slow DB queries, background jobs. Skip pure-CPU functions.

Output as a list with file:line references and a code snippet showing the instrumentation in the team's language.

An SLO is a target for how often the service does the right thing. The classic shape: '99.9% of /checkout requests complete in under 500ms over a rolling 30 days.' That gives you an error budget (0.1% over 30 days = ~43 minutes). When you burn through the budget, you slow feature velocity. When you have lots of budget, you ship aggressively. SLOs only work if the team treats them as real.

Threshold alerts ('latency over 500ms') page constantly on noise. Burn-rate alerts ('we'll exhaust this month's error budget in 6 hours at the current rate') page when something is actually breaking. Multi-window burn rate (fast: 1 hour, slow: 6 hours) catches both fast outages and slow degradations.

Runbooks turn a 2 AM page into a 15-minute fix instead of a 2-hour panic. Write four for the most common failure modes: high latency, error rate spike, saturation (out-of-memory / connection pool exhausted), third-party dependency down. Each is a numbered list of 'first 5 minutes' steps.

Postmortems compound. The first one feels like overhead; the tenth one is your team's most valuable institutional document. Write the postmortem within 48 hours of the incident, blameless tone, focused on the system gaps not the individual mistakes.

Observability is not done; it drifts. Set a weekly 30-min slot to walk through the dashboards, check SLO health, prune alerts that paged-but-weren't-real, add metrics for the new things you shipped. The team that does this is 10x better at responding to incidents in 12 months than the team that doesn't.