When an SRE asks "what did the agent see in Datadog last night, and who owns it now," the answer should not be a Slack scroll. Datadog and PagerDuty keep the raw signal and the page. Dock keeps the agent's read, the proposed remediation, the named human reviewer, and the timestamp the page was acknowledged. One row per anomaly, one owner per row, one audit trail across the whole incident.
Datadog stays the system of record for metrics, traces, and logs. PagerDuty stays the system of record for the page, the on-call rotation, and the incident timeline. Dock is the system of record for what the agent interprets from that telemetry. Each Dock row carries a datadog_monitor_id and pagerduty_incident_id pointer back to the platform record, plus agent identity, the anomaly read, the proposed remediation, the reviewer, and timestamps. The agent re-fetches Datadog query results and PagerDuty incident state via fresh API reads whenever it needs current state, because cached interpretations age into lies.
The remediation queue table
| dock_row_id | datadog_monitor_id | pagerduty_incident_id | anomaly_read | proposed_remediation | agent | reviewer | status |
|---|---|---|---|---|---|---|---|
| rq_8821 | mon_44213 | PD-9931 | p99 latency on checkout-api breached 800ms for 12m; correlated with RDS connection saturation at 94% | scale rds-checkout read replica from 2 to 4; revert in 4h if p99 recovers | argus | govind | approved, executing |
| rq_8822 | mon_44890 | PD-9933 | error rate on auth-service jumped to 4.1% after 03:12 deploy; stack trace points to expired JWT signing key | roll back auth-service to sha 8f2a1c | argus | sarah | pending two-key |
| rq_8823 | mon_45102 | PD-9940 | log volume on api-gateway down 78% vs 7d baseline; likely silent ingestion failure, not a real quiet period | open Datadog ticket, page data-platform on-call, do not auto-remediate | argus | govind | escalated |
How the workflow runs
Argus, the observability agent, polls Datadog monitors every 60 seconds. A monitor fires. Argus pulls the alert, fetches the underlying query, correlates against the last six hours of related metrics, and writes a new row to the remediation queue with the anomaly read and a proposed action. If the proposed action is read-only or reversible inside four hours, a single named reviewer approves and Argus executes. If the action touches production state in a way that cannot be reversed cheaply, the row blocks behind a two-key handshake and inherits the dangerous-ops contract. Once the action runs, Argus posts the Dock row link back into the PagerDuty incident and marks the page acknowledged with the reviewer's name.
Why this matters
Datadog is excellent at telling you that something is wrong. It is not designed to tell you which agent looked at it, what that agent concluded, and which human signed off on the fix. That gap is where post-incident reviews get expensive. With a Dock row per anomaly, the incident retro reads itself: the monitor, the agent's interpretation, the reviewer, the remediation, the time to acknowledge, and the rollback path are all in one queryable surface.
It also closes the audit loop. Every agent-driven remediation has a named human reviewer attached before execution, which is the baseline expectation under agent audit and compliance. No anonymous fixes. No "the bot did it" in the timeline.
And it composes with the rest of the DevOps surface: the same agent identity, the same reviewer pattern, the same pointer-back discipline that already governs deploys now governs incident response.
Build the remediation queue inside Dock for IT operations and route every Datadog-sourced anomaly through a named reviewer before remediation runs.
FAQ
Does Dock replace Datadog or PagerDuty? No. Datadog keeps the telemetry. PagerDuty keeps the page and rotation. Dock holds the agent's interpretation, the remediation proposal, the reviewer, and the audit trail. The three systems point at each other by ID.
What stops the agent from acting on stale Datadog data? Argus re-queries Datadog at the moment of decision and writes the query timestamp into the Dock row. If the row has been sitting in review for longer than the freshness window, the agent re-fetches before executing. Google's SRE practice frames this as symptom-based alerting on current state, which only works if the read is current.
How does this interact with the four golden signals? The remediation queue is monitor-agnostic. Whether the firing monitor is latency, traffic, errors, or saturation, the row shape is the same: anomaly read, proposed remediation, reviewer, pointer back. The golden-signal framing lives in Datadog where it belongs.
Can a human override the agent's read? Yes. The reviewer can reject the proposed remediation, write a counter-read into the row, and either remediate manually or hand back to the on-call. The override is logged with the reviewer's identity and timestamp.
