Outage alerts

Batchwatch watches every model it measures and opens an outage when the queue for one of them degrades against its own baseline. This page is how you get told about it.

Three routes, all keyed. A subscription belongs to the API key that created it: your key sees your subscriptions and nobody else's.

What triggers an alert

An outage is not a slow job. It is a sustained deviation confirmed by several independent contributors, held for a minimum duration before anything is sent:

• The current window is compared to the model's own recent baseline, not to a fixed threshold. A model whose median is 40 minutes is not "down" for being slow — it is down when it is slow for itself.
• At least three established contributors must see it. A contributor earns a vote by measuring on at least three separate days, so neither volume nor new accounts can manufacture an outage.
• The deviation must persist for 20 minutes before an outage_started event fires, and recovery must hold for 30 minutes before outage_ended.

The result is that you get few alerts and they mean something. A single slow batch will never page you.

POST /v1/subscriptions

curl -X POST https://batchwatch.dev/v1/subscriptions \
  -H "authorization: Bearer $BATCHWATCH_KEY" \
  -H 'content-type: application/json' \
  -d '{
        "channel": "webhook",
        "target": "https://example.com/hooks/batchwatch",
        "providers": ["openai", "anthropic"],
        "min_severity": "degraded"
      }'

Body

Response — 201

{
  "id": 1,
  "channel": "webhook",
  "target": "https://example.com/hooks/batchwatch",
  "providers": "openai,anthropic",
  "models": null,
  "min_severity": "degraded",
  "created_at": 1787727392,
  "has_secret": true,
  "secret": "SwQ4nhynWnKPBF8v99Ou91Xs-i7MoIH9YjPEoN6vBG8",
  "secret_note": "gemmes krypteret; vises kun her og aldrig igen"
}

secret appears once and never again. Copy it now. It is stored encrypted (AES-256-GCM under a key that is not in the database), so a dump of the database alone does not yield it — and neither can we read it back out to you later. If you lose it, delete the subscription and create a new one.

If you supplied your own secret, it is not echoed back. We do not return a secret you already have.

Other responses

GET /v1/subscriptions

{
  "subscriptions": [
    {
      "id": 1,
      "channel": "webhook",
      "target": "https://example.com/hooks/batchwatch",
      "providers": ["openai", "anthropic"],
      "models": null,
      "min_severity": "degraded",
      "created_at": 1787727392,
      "last_ok_at": 1787730000,
      "fail_streak": 0,
      "has_secret": true
    }
  ]
}

The secret is never in this response — has_secret tells you whether one exists, nothing more.

fail_streak is how many consecutive deliveries have failed, and last_ok_at is the last one that succeeded. A failing subscription is never disabled automatically. A channel that switches itself off in silence cannot be told apart from an alert that never had to fire, which is the one failure mode an alerting system must not have. Watch fail_streak instead.

Note that providers comes back as a list, and POST accepts a list — so fetch, edit, and send back works.

DELETE /v1/subscriptions/{id}

curl -X DELETE https://batchwatch.dev/v1/subscriptions/1 \
  -H "authorization: Bearer $BATCHWATCH_KEY"

200 with {"id": 1, "revoked": true} on success.

404 if the id is unknown or belongs to another key. The two are deliberately indistinguishable: otherwise one key could confirm the existence of another key's subscription by probing ids.

Receiving a webhook

POST to your target, content-type: application/json, with two headers:

Verifying the signature

Compute the HMAC over the exact bytes you received, before any JSON parsing.

import hmac, hashlib

def verify(raw_body: bytes, header: str, secret: str) -> bool:
    expected = "sha256=" + hmac.new(
        secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header)

import { createHmac, timingSafeEqual } from 'node:crypto';

function verify(rawBody, header, secret) {
  const expected = 'sha256=' + createHmac('sha256', secret)
    .update(rawBody).digest('hex');
  const a = Buffer.from(expected);
  const b = Buffer.from(header || '');
  return a.length === b.length && timingSafeEqual(a, b);
}

Use a constant-time comparison (compare_digest / timingSafeEqual), not ==. A plain string compare leaks how much of the signature was correct.

If the signature header is missing

Treat the message as unverified and drop it. We send unsigned rather than wrongly signed if we cannot read your secret — a wrong signature would look like an attack at your end, which is worse than an obvious absence. The delivery is logged as signed: false on our side either way.

Slack subscriptions carry no signature. The webhook URL is the secret.

Delivery behaviour

• Sequential, with a cap. Deliveries go out one at a time up to a per-event limit. A hundred parallel requests to a hundred dead endpoints from a worker on a tight budget would take the measurement run down with it, and the dataset matters more than the message.
• Five-second timeout per delivery.
• Failure is open. A dead endpoint, a DNS error, a timeout — none of it can break the measurement run that produced the alert.
• Every attempt is logged, success or failure, with the HTTP status and whether it was signed.

Public alternatives, no key needed

If you would rather poll than subscribe:

• GET /v1/outages — current and recent outages, as JSON.
• GET /v1/outages.atom — the same as an Atom feed.

Both are open, and neither is subject to the freshness delay that applies to the measurement routes. An outage feed that is fifteen minutes late is not a feed.