Outage alerts

Batchwatch watches every model it measures and opens an outage when the queue for one of them degrades against its own baseline. This page is how you get told about it.

Three routes, all keyed. A subscription belongs to the API key that created it: your key sees your subscriptions and nobody else's.

What triggers an alert

An outage is not a slow job. It is a sustained deviation confirmed by several independent contributors, held for a minimum duration before anything is sent:

• The current window is compared to the model's own recent baseline, not to a fixed threshold. A model whose median is 40 minutes is not "down" for being slow — it is down when it is slow for itself.
• At least three established contributors must see it. A contributor earns a vote by measuring on at least three separate days, so neither volume nor new accounts can manufacture an outage.
• The deviation must persist for 20 minutes before an outage_started event fires, and recovery must hold for 30 minutes before outage_ended.

The result is that you get few alerts and they mean something. A single slow batch will never page you.

POST /v1/subscriptions

curl -X POST https://batchwatch.dev/v1/subscriptions \
  -H "authorization: Bearer $BATCHWATCH_KEY" \
  -H 'content-type: application/json' \
  -d '{
        "channel": "webhook",
        "target": "https://example.com/hooks/batchwatch",
        "providers": ["openai", "anthropic"],
        "min_severity": "degraded"
      }'

Body

Response — 201

{
  "id": 1,
  "channel": "webhook",
  "target": "https://example.com/hooks/batchwatch",
  "providers": "openai,anthropic",
  "models": null,
  "min_severity": "degraded",
  "created_at": 1787727392,
  "has_secret": true,
  "secret": "SwQ4nhynWnKPBF8v99Ou91Xs-i7MoIH9YjPEoN6vBG8",
  "secret_note": "stored encrypted; shown here once and never again"
}

secret appears once and never again. Copy it now. It is stored encrypted (AES-256-GCM under a key that is not in the database), so a dump of the database alone does not yield it — and neither can we read it back out to you later. If you lose it, delete the subscription and create a new one.

If you supplied your own secret, it is not echoed back. We do not return a secret you already have.

Other responses

Self-monitoring alerts (operators)

batchwatch also watches itself. Three self-alerts today:

• rollup_stalled — the rollup write path stops publishing fresh numbers while the pages keep answering. Alerts at ~20 minutes, long before /v1/status flips to 503.
• homepage_unreachable (#307) — a synthetic probe of the real public site (/, /pricing, /docs) returns a non-200, a 200 body under its byte floor, or a 200 missing its content marker, for N consecutive probes (default 3, ~10–15 min). This is the alert for the #320 gap: the homepage 503'd for 84 minutes while every data-pipeline check stayed green. The message quotes the observed status and route, and the alert fires once per outage episode, not every tick.
• cron_stalled (#311) — the whole cron stopped: no tick has run, so no rollup has advanced past the dead-cron threshold (~30 min / 3 missed 10-min ticks). Because an in-cron watchdog is fail-silent when the cron itself is dead, this alert is measured on the request path instead — a sampled fraction of ordinary requests and every /v1/status poll — so it fires even when the cron cannot fire it for itself. One page per stall episode; it re-arms after the cron recovers.

These self-alerts also email the operator directly. Whenever a self-alert fires and the deployment has RESEND_API_KEY configured, batchwatch emails it to the fixed operator address hello@batchwatch.dev (src/self_alert_email.js, #330) — keyed off configuration, not off a subscription row, so an operator is reached even when outage_sub is empty. This is the fix for the #330 "empty room": a self-alert that matched no subscriber used to reach no one.

To also receive these on a webhook or Slack channel, subscribe with the sentinel provider batchwatch in providers:

{ "channel": "webhook", "target": "https://ops.example/hook",
  "providers": "batchwatch" }

Only a subscription that explicitly names batchwatch gets them — a subscription that omits providers (all provider outages) does not, so a customer watching real-provider outages is never sent our internal plumbing alerts. (The operator email leg is separate from the subscription table entirely.) The body is a distinct batchwatch.self_alert.v1 schema (a type, a human message, and the measured detail — the newest-rollup age, or the route/status/reason), not the outage schema, so a receiver never mistakes "batchwatch is down" for "a provider had an outage".

Three types reach this schema, because the pipeline can fail three ways:

The distinction matters: an in-cron watchdog is fail-silent when the whole cron stops, because the tick that would alert never runs either. cron_stalled closes that hole — it is measured on a sampled fraction of ordinary requests and on every /v1/status poll, both of which keep flowing when the cron is dead, so the alert fires even when the cron cannot fire it for itself. You receive at most one cron_stalled page per stall episode; it re-arms after the cron recovers.

GET /v1/subscriptions

{
  "subscriptions": [
    {
      "id": 1,
      "channel": "webhook",
      "target": "https://example.com/hooks/batchwatch",
      "providers": ["openai", "anthropic"],
      "models": null,
      "min_severity": "degraded",
      "created_at": 1787727392,
      "last_ok_at": 1787730000,
      "fail_streak": 0,
      "has_secret": true
    }
  ]
}

The secret is never in this response — has_secret tells you whether one exists, nothing more.

fail_streak is how many consecutive deliveries have failed, and last_ok_at is the last one that succeeded. A failing subscription is never disabled automatically. A channel that switches itself off in silence cannot be told apart from an alert that never had to fire, which is the one failure mode an alerting system must not have. Watch fail_streak instead.

Note that providers comes back as a list, and POST accepts a list — so fetch, edit, and send back works.

DELETE /v1/subscriptions/{id}

curl -X DELETE https://batchwatch.dev/v1/subscriptions/1 \
  -H "authorization: Bearer $BATCHWATCH_KEY"

200 with {"id": 1, "revoked": true} on success.

404 if the id is unknown or belongs to another key. The two are deliberately indistinguishable: otherwise one key could confirm the existence of another key's subscription by probing ids.

Receiving a webhook

POST to your target, content-type: application/json, with two headers:

Verifying the signature

Compute the HMAC over the exact bytes you received, before any JSON parsing.

import hmac, hashlib

def verify(raw_body: bytes, header: str, secret: str) -> bool:
    expected = "sha256=" + hmac.new(
        secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header)

import { createHmac, timingSafeEqual } from 'node:crypto';

function verify(rawBody, header, secret) {
  const expected = 'sha256=' + createHmac('sha256', secret)
    .update(rawBody).digest('hex');
  const a = Buffer.from(expected);
  const b = Buffer.from(header || '');
  return a.length === b.length && timingSafeEqual(a, b);
}

Use a constant-time comparison (compare_digest / timingSafeEqual), not ==. A plain string compare leaks how much of the signature was correct.

If the signature header is missing

Treat the message as unverified and drop it. We send unsigned rather than wrongly signed if we cannot read your secret — a wrong signature would look like an attack at your end, which is worse than an obvious absence. The delivery is logged as signed: false on our side either way.

Slack subscriptions carry no signature. The webhook URL is the secret.

Delivery behaviour

• Sequential, with a cap. Deliveries go out one at a time up to a per-event limit. A hundred parallel requests to a hundred dead endpoints from a worker on a tight budget would take the measurement run down with it, and the dataset matters more than the message.
• Five-second timeout per delivery.
• Failure is open. A dead endpoint, a DNS error, a timeout — none of it can break the measurement run that produced the alert.
• Every attempt is logged, success or failure, with the HTTP status and whether it was signed.

Public alternatives, no key needed

If you would rather poll than subscribe:

• GET /v1/outages — current and recent outages, as JSON.
• GET /v1/outages.atom — the same as an Atom feed.

Both are open, and neither is subject to the freshness delay that applies to the measurement routes. An outage feed that is fifteen minutes late is not a feed.