# PagerDuty agents

A PagerDuty agent wakes when an incident triggers, investigates it, and writes what it found back to
the incident. `pagerdutyChannel()` verifies PagerDuty's v3 webhooks and sends every later event for
that incident to the session that started the investigation.

## Investigate a new incident

When an incident triggers, the agent starts a turn with the incident's title, service, urgency, and
link.

```ts
// bot/channels/pagerduty.ts
import { defaultPagerDutyAuth, pagerdutyChannel } from "@grok/agent-sdk/channels/pagerduty";

export default pagerdutyChannel({
  deliverReplies: true,
  onIncident: (ctx, event) => {
    if (event.eventType !== "incident.triggered") {
      return null;
    }
    return {
      // `auth` starts a model turn running as the event's PagerDuty actor.
      auth: defaultPagerDutyAuth(ctx),
      message: `Investigate PagerDuty incident #${event.incident.number}: ${event.incident.title}`,
      context: [`Service: ${event.incident.service?.summary ?? "unknown"}`],
    };
  },
});
```

With `deliverReplies: true`, the channel posts each reply as a note on the incident. It doesn't post
anything until it knows the agent's own PagerDuty user, because each note comes back as a webhook
event and would otherwise start another turn. Handlers you write in `events` replace the built-in
ones.

Hooks also get `ctx.pagerduty`, a REST client with the channel's credentials. Use the incident from
the event or channel state when you write, so the model can't choose the target.

```ts
onIncident: async (ctx, event) => {
  const incident = await ctx.pagerduty.getIncident(event.incident.id);
  // …
},
```

On Cursor hosting, declare PagerDuty's API host and the token:

```ts
// bot/agent.ts
import { defineAgent } from "@grok/agent-sdk";

export default defineAgent({
  hosting: {
    egressDomains: ["api.pagerduty.com"],
    secretNames: ["PAGERDUTY_TOKEN"],
  },
});
```

## Follow the incident in one session

People acknowledge, annotate, reassign, and resolve an incident while the agent investigates. Return
a dispatch for those events too, and each one reaches the session already working on the incident.

```ts
onIncident: (ctx, event) => {
  const followUps = ["incident.acknowledged", "incident.reassigned", "incident.resolved"];
  if (event.eventType !== "incident.triggered" && !followUps.includes(event.eventType)) {
    return null;
  }
  return {
    auth: defaultPagerDutyAuth(ctx),
    message: `${event.eventType} by ${ctx.agent?.summary ?? "PagerDuty"}`,
  };
},
onNote: (ctx, note) => {
  return {
    auth: defaultPagerDutyAuth(ctx),
    message: `Note on the incident: ${note.content}`,
  };
},
```

Add both to the same `pagerdutyChannel()` call. A re-trigger after a resolve is a new PagerDuty
incident, so it starts a new session. `ctx.incidentKey` carries the alert's dedup key, so the agent
can recall what it found the last time the same alert fired.

## Ignore your own actions

PagerDuty sends the agent's own changes back as webhook events, such as a note it posted or an
incident it resolved. The channel acknowledges those events without starting a turn, and the same
action by a person still arrives.

```ts
export default pagerdutyChannel({
  // Events caused by the agent's own PagerDuty user are skipped.
  // The user comes from, in order: selfUserIds, PAGERDUTY_USER_ID,
  // or the user PAGERDUTY_TOKEN acts as (GET /users/me).
  selfUserIds: ["PBOT001"],
  onIncident: (ctx) => ({ auth: defaultPagerDutyAuth(ctx) }),
});
```

A user-level API key or an OAuth user token acts as one user. An account-level key or an app token
acts as no user, so the channel uses the user whose email matches `PAGERDUTY_FROM_EMAIL`. A
read-only agent with no token sets `PAGERDUTY_USER_ID` instead. When none of these names a user, the
channel skips nothing.

## Events

Each hook receives one family of v3 events:

| Hook          | Events                                                                                                                                                                       |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `onIncident`  | `incident.triggered`, `.acknowledged`, `.unacknowledged`, `.escalated`, `.reassigned`, `.delegated`, `.resolved`, `.reopened`, `.priority_updated`, `.incident_type.changed` |
| `onNote`      | `incident.annotated`                                                                                                                                                         |
| `onResponder` | `incident.responder.added`, `incident.responder.replied`                                                                                                                     |
| `onEvent`     | Everything else, such as status updates and workflow events                                                                                                                  |

Without `onIncident`, a triggered or reopened incident starts a turn. The channel acknowledges every
other event until you add its hook. Each hook returns `{ auth }`, `{ task }`, or `null`, as
[Handler results](/docs/reference/channels.md#handler-results) describes.

## Connect

Create a v3 webhook subscription in PagerDuty for the services the agent should respond to, and
point it at `https://<your-host>/<slug>/v1/channels/pagerduty`. Set the channel's secrets on the
deployment, or as environment variables locally.

| Variable                   | Used for                                                                                                               | Override                                 |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- |
| `PAGERDUTY_WEBHOOK_SECRET` | Verifying `X-PagerDuty-Signature`. On Cursor hosting, a delivery with a bad signature gets a `401`.                    | `webhookSecret: { secretName }`          |
| `PAGERDUTY_USER_ID`        | The agent's own PagerDuty user ID. Optional when `PAGERDUTY_TOKEN` names a user.                                       | `selfUserIds: [ids]` or `{ secretName }` |
| `PAGERDUTY_TOKEN`          | Reply notes, `ctx.pagerduty`, and finding the agent's user. Use a REST API key, or an OAuth token as `Bearer <token>`. | `token: { secretName }`                  |
| `PAGERDUTY_FROM_EMAIL`     | The `From` user for writes with an account-level key or app token.                                                     | `from: { secretName }`                   |
| `PAGERDUTY_API_BASE_URL`   | The REST host, such as `https://api.eu.pagerduty.com`. Defaults to `https://api.pagerduty.com`.                        | `apiBaseUrl`                             |

`createPagerDutyClient()` takes the same options for code outside the channel.

```bash
agent-sdk deploy --dir . --slug incident-responder
agent-sdk secret set incident-responder PAGERDUTY_WEBHOOK_SECRET PAGERDUTY_TOKEN
```

The next delivery picks up a new value without another deploy. Until the webhook secret is set, the
route accepts only loopback deliveries under `agent-sdk dev`.

### Several subscriptions

PagerDuty gives each webhook subscription its own signing secret. Put one secret on each line of
`PAGERDUTY_WEBHOOK_SECRET`:

```text
<signing secret of subscription 1>
<signing secret of subscription 2>
```

A delivery passes when its signature matches any line. Set the value from that file:

```bash
agent-sdk secret set incident-responder PAGERDUTY_WEBHOOK_SECRET < pagerduty-webhook-secrets.txt
```

To rotate a secret, add the new one before PagerDuty signs with it, and remove the old one once
nothing signs with it.

## Test locally

`agent-sdk pagerduty replay` posts synthesized v3 deliveries for one incident to the running agent,
in order. Without `PAGERDUTY_WEBHOOK_SECRET`, `agent-sdk dev` admits them unsigned from loopback.
With it set, replay signs them the way PagerDuty does.

```bash
agent-sdk dev &
agent-sdk pagerduty replay --events triggered,acknowledged,annotated,resolved \
  --comment "Rolled back the 09:30 deploy."
```

Each event joins the session the trigger started. Pass an incident ID to start another session,
`--dry-run` to print the bodies, or `--fixtures-dir fixtures/pagerduty` to save them as fixtures for
[evals](/docs/evals.md).

To post a delivery you saved from the webhook log, sign it:

```bash
export PAGERDUTY_WEBHOOK_SECRET=local-test-secret
agent-sdk dev &

body=fixtures/pagerduty/incident-triggered.json
signature=$(openssl dgst -sha256 -hmac "$PAGERDUTY_WEBHOOK_SECRET" -hex < "$body" | awk '{print $NF}')
curl -X POST http://127.0.0.1:3000/<slug>/v1/channels/pagerduty \
  -H 'content-type: application/json' \
  -H "x-pagerduty-signature: v1=$signature" \
  --data-binary @"$body"
```

## Related

- [Channels reference](/docs/reference/channels.md)
- [Webhooks and custom channels](/docs/guides/webhooks.md)
- [Evals](/docs/evals.md)
