batchwatch
Live dashboard Use the API

batchwatch › How we measure

How we measure

Everyone else guesses how long the batch API takes. We measured it - and here is exactly how, so you can decide for yourself whether the number is worth trusting.

What a probe does

A probe is one of our own measurement jobs. On a schedule we submit a small batch to each provider, exactly the way a customer's job would go in, and we record when it started and when the result came back - timed on our server, not on a client clock that can drift. Probe rows are marked as ours, so a dataset that is really one account can never masquerade as the world.

Crowdsourced contributions

Probes get us off the ground; contributions are what make a number represent everyone. Anyone can report a measurement - when their own batch started and finished - with or without an API key. A keyed contribution earns a vote in the published percentile, because a figure that many independent accounts agree on is worth far more than one large account's median. The more contributors behind a model, the higher its confidence grade climbs.

The percentile method

We draw a distribution once a model has at least 2 measurements - two points are the mathematical minimum for a curve. We only write a headline percentile, such as "nine in ten finished within X", once a model has at least 8 measurements. Below that threshold the points are shown and the reader can count them, but the claim is not made: a 90th percentile computed from three measurements is literally just the slowest of the three, and we will not dress that up as a percentile. Show the data; do not infer too much from it.

Where a model is still thin, we fall back to the provider's own published completion window - typically 24 hours - and label it as the contract it is, not a measurement.

Delay and freshness

Public figures are precomputed and delayed by 15 minutes. That delay is the product: it is exactly what a paying customer buys out of. A contributor who verifies an email sees the numbers at 10 minutes, and a paying customer sees them live. Every page writes an absolute timestamp for the last measurement - a fixed instant, never a relative age like "moments ago" - so a cached copy says the same true thing an hour later.

Confidence grades

Every number carries a grade that blends how much we measured, how many independent contributors stand behind it, how fresh it is, and how wide the range is. A dataset from a single contributor is capped low however large it is - it measures that account's setup, not the queue.

The numbers we did not measure

The 50% batch discount is the providers' own published price - OpenAI, Anthropic and Google each charge half the synchronous rate for batch on both input and output tokens - and we quote it as theirs. When we estimate savings, The 30% batchable share is an assumption, not a measurement, and we label it as the weakest number in the model. Our queue times are measured; that share is not - and the difference between the two is the whole product. We tell you which numbers we observed and which we assumed, because that line is the one everyone else blurs.

What we do not collect

No prompts. No keys. No content.

A measurement is metadata only: provider, model, token counts, timestamps, status and region. There is no field anywhere in the API for prompt text or model output, so we could not store your content even if we wanted to. We never receive your provider API keys either - your client reports what its own job did; it does not route the job through us. On the free trial we identify a caller by a salted hash, never a raw IP address.

Questions about the method

Where do the queue times come from?
Every figure is an observed completion of a real batch job. A client opens a measurement when it submits a batch and closes it when the provider returns the result; the wait is the difference, timed on our server rather than the client's clock. Nothing is predicted and nothing is simulated.
What is a probe?
A probe is one of batchwatch's own measurement jobs: we submit a small batch to a provider on a schedule and record how long it takes, exactly as a customer's job would. Probe rows are marked as ours and can be filtered out, so a dataset that is really one account says so instead of hiding it.
How does crowdsourcing work?
Anyone can contribute a measurement, with or without an API key, by reporting when their own batch started and finished. Contributions are what make a percentile represent the world rather than one account, and only a keyed contribution earns a vote in the number.
How do you turn measurements into a percentile?
We publish a curve once a model has at least 2 measurements, and a headline percentile such as "nine in ten finished within X" only once it has at least 8. Below that the points are shown but the claim is not written, because a 90th percentile drawn from three points is just the slowest of the three.
Why are the public numbers delayed?
Public figures are precomputed and delayed by 15 minutes; that delay is what paying customers buy out of. A contributor who verifies an email sees them at 10 minutes, and a paying customer sees them live. The page writes an absolute timestamp for the last measurement, never a relative age, so a cached copy stays true.
What does a confidence grade mean?
Each number carries a grade - high, medium, low or very low - that blends how many measurements there are, how many independent contributors, how fresh the newest one is, and how wide the range is. A dataset from a single contributor is capped low however large it is, because it measures that account, not the queue.
What is the 50% batch discount?
The 50% batch discount is the providers' OWN published price - OpenAI, Anthropic and Google all charge half the synchronous rate for batch on both input and output tokens. We quote it as theirs, never as a batchwatch measurement.
Do you collect our prompts or API keys?
No. A measurement is metadata only: provider, model, token counts, timestamps, status and region. There is no field for prompt text or model output, and we never receive your provider API keys - a client reports what its own job did, it does not route the job through us.

See it for yourself

All of it is public.

The live dashboard shows every model with its distribution, its n and its confidence grade. /v1/coverage is the same data as JSON, no key required, and the questions answer the searches people actually type. Contribute measurements and you help build the number you are reading.