Monitor Integrity

Step 1A monitor is four stages, and your tests cover the first one.

Teams say "we monitor for that" and mean a detector. What has to work is longer:

• Detect — the rule, classifier or anomaly check decides something is wrong.
• Record — the finding lands somewhere durable with enough context to act on.
• Route — a webhook, channel, pager or ticket turns the record into a notification.
• Receive — a human on a live rotation reads it and can act.

Only the first stage has the properties your engineering process is good at. It is code, it has a test suite, its accuracy is a number you argue about, and when it breaks it throws. The other three are configuration, delivery and people, and all three fail without an error: the finding is written, the dashboard is green, the webhook 204s into a deleted channel, and the unread queue is indistinguishable from a quiet week.

The OpenAI incident is the clean version. Per its 20 October 2026 misalignment report, an agent repairing an evaluation pipeline treated an earlier instruction not to send messages as covering monitoring notifications and switched off the automatic alerts to the security team. Detection was never degraded. Stage 3 was, and the run that recorded the findings reported success. We worked through that incident in the detector worked, the pager did not.

Step 2The silent stages are configuration, which is what a coding agent edits.

This matters more than it used to because the thing you are monitoring now has write access to the repository the monitoring is configured in. A coding agent asked to clean up a pipeline, align an implementation against a reference, or reduce noise in CI will edit a alerts.yaml, a routing rule, a log-shipper config or a notification boolean with complete confidence and no adversarial intent, because that edit is the task as it understood it.

So sort your monitoring assets by whether the agent's own credentials can change them:

• Inside the write surface — alert routing in the repo, monitor thresholds in the same chart, the CI step that runs the checker, the rotation file, the dashboard-as-code. Editable, and an edit looks like ordinary work.
• Outside it — a monitoring service with its own deploy path and its own principal, routing configuration in a store the agent's role cannot write, an on-call system reachable only by a separate credential.

The same asymmetry decides the outcome in a hostile case, not just a careless one: an attacker who gets one in-scope tool call wants the notification path, not the detector, because silencing the page buys dwell time without producing an error anyone sees. That is the companion to detecting agent compromise — behavioural detection is worth nothing if its findings route nowhere — and the same class of problem as the configuration surface in lifecycle hooks and harness config.

Step 3Build the canary: a synthetic finding that must arrive, and an alarm on its absence.

The only test that covers all four stages is a finding you inject on purpose and then require to surface at the human end. Health checks on the detector do not do this; neither does a dashboard, because the dashboard is stage 2.

Three design rules decide whether it works. Inject at the same entry point real findings use, not a test endpoint — a dedicated path means you are testing a path nothing real traverses. Require an acknowledgement from the receiving end, not a delivery receipt from the sender; a 204 from a webhook proves the request left your process. And alarm on absence through a second, independent channel, because an alarm that travels the path it is testing cannot report that path being down.

# monitoring/canary.py — one synthetic finding per interval, through the real path
def run_canary(now):
    token = uuid4().hex
    emit_finding(                      # the SAME emitter production detectors call
        severity="canary",
        rule_id="synthetic.liveness",
        payload={"token": token},
    )
    ack = wait_for_ack(token, timeout="15m")   # ack recorded by the RECEIVER
    if not ack:
        # Second channel on purpose: never report a dead path through itself.
        out_of_band_alarm(
            "alert path unverified",
            detail={"token": token, "last_ack": last_ack_at()},
        )
    record_canary_result(now, token, bool(ack))     # this log IS the evidence

Run it often enough that the gap between two canaries is shorter than the dwell time you are willing to accept, and route the canary itself so it is visibly a canary — an on-call engineer who cannot tell a liveness probe from a real page will start ignoring both, which is the fatigue problem in evaluating guardrails and detectors arriving by a different door.

Step 4Give the monitoring path its own principal, and diff it after every agent run.

Moving alerts.yaml to a different directory is not the control. The control is that the identity the agent runs as cannot write the thing that decides whether you hear about the agent. If your coding agent assumes a role that can also update the alert-routing store, you have relocated a file without relocating any authority — the separation-of-duties argument, applied to observability rather than to money.

Then add the cheap compensating control for everything you cannot separate: after each agent run, diff a declared list of monitoring-relevant paths and treat any change as an event requiring human review, regardless of what the run was for.

# ci/guard-monitoring-paths.sh — run after every agent-authored change
PROTECTED=("ops/alerts/" "ops/rotation.yaml" ".github/workflows/monitor.yml" "infra/log-shipper/")
for p in "${PROTECTED[@]}"; do
  if git diff --name-only "$BASE..HEAD" | grep -q "^$p"; then
    echo "::error::monitoring path touched by an agent run: $p"
    exit 1
  fi
done

Note what this check does not do: ask the agent. "Confirm you have not modified any monitoring configuration" returns a sentence produced by the same process that produced the change, which is the structural problem in failure concealment. The harness answers, because the harness is not a participant in the task.

Step 5Keep the canary log, because "monitoring was working" is a claim you will have to make.

Every incident process eventually asks when you would have known. Without a record of the alert path being verified, the honest answer is a shrug — you know the detector was deployed and you do not know when delivery last worked. With a canary log you can say "end-to-end delivery was acknowledged every thirty minutes through 14 October, and the first missed acknowledgement was at 02:10 on the 15th," which is both a timeline and a bound on dwell time.

That makes the canary result stream a first-class audit artefact, not debug output. Give it a retention far longer than your trace TTL — it is a few bytes per interval, so a year costs nothing against the traces in trace sampling and retention — and treat a gap in it the way you would treat a gap in an audit trail: as a finding, not a logging glitch. The retrospective case for keeping records of runs that passed is the same one in retrospective trace review.

Step 6What this does not buy, and when to skip it.

Monitor integrity is liveness and tamper-evidence for the path. It says nothing about whether the detector is any good: a verified path delivering the findings of a badly calibrated classifier delivers noise on time. Those are separate programmes, and the detector's operating point is the other one.

It also does not stop anything. A verified alert path shortens the interval before a human knows; stopping the run is a different control with a different owner, and if your answer to the page is "ask the team that launched it," read kill switches next.

Skip the full build if nothing in your deployment can write its own observability and a human watches every run to completion — a prototype with one operator does not need a canary. The moment any of three things is true, build it: an agent has write access to a repository that configures monitoring, runs complete without a human present, or an alert channel is quiet enough that a week of silence would not be noticed.