Where do the queue times come from?

Every figure is an observed completion of a real batch job. A client opens a measurement when it submits a batch and closes it when the provider returns the result; the wait is the difference, timed on our server rather than the client's clock. Nothing is predicted and nothing is simulated.

A probe is one of batchwatch's own measurement jobs: we submit a small batch to a provider on a schedule and record how long it takes, exactly as a customer's job would. Probe rows are marked as ours and can be filtered out, so a dataset that is really one account says so instead of hiding it.

How does crowdsourcing work?

Anyone can contribute a measurement, with or without an API key, by reporting when their own batch started and finished. Contributions are what make a percentile represent the world rather than one account, and only a keyed contribution earns a vote in the number.

How do you turn measurements into a percentile?

We publish a curve once a model has at least 2 measurements, and a headline percentile such as "nine in ten finished within X" only once it has at least 8. Below that the points are shown but the claim is not written, because a 90th percentile drawn from three points is just the slowest of the three.

How is the p90 actually computed?

We build a survival curve with the Kaplan-Meier estimator using Weibull plotting positions with linear interpolation, then read the 90th percentile off it. That convention has a property we can check - it places each measurement at its expected rank - so the p90 is a real percentile, not the largest of a handful of points dressed up as one. The full derivation is public in our estimator design of record.

What happens to a batch that is still running?

It is kept, not thrown away. A job that has not returned yet is right-censored data - we know it has already taken at least this long - and the survival curve uses that lower bound instead of pretending the job does not exist. The rule is an invariant: a censored observation never becomes a measurement, and it never disappears. Dropping still-running jobs is exactly how a naive method reports a queue as faster than it is.

Do you reject outliers?

No, and that is deliberate. A slow batch is the number a customer most needs to see, so cutting the tail would hide the very thing we measure. We tried the opposite - an "8 x median" cut - and measured the damage on gpt-5-nano before removing it: it discarded 36% of the measurements and reported a p90 of 194 seconds when the true figure was 7817, roughly forty times too low. We keep every measurement in the percentile and report separately how many sit above a heavy-tail threshold.

Can one big account skew the number?

A single contributor cannot run away with a percentile. We estimate the law of a typical job as a mixture across contributors by their share, and the pooled share of untrusted, un-established contributors is capped at 20%. A dataset that is really one account is graded low however large it is, because it measures that account, not the queue.

What does a confidence grade mean?

Each number carries a grade - high, medium, low or very low - that blends how many measurements there are, how fresh the newest one is, and how wide the range is. How many contributors stand behind it is reported next to the grade rather than folded into it, so you can weigh breadth yourself. A figure a single source could still move - a pooled percentile - is capped low, because that is a property of the number rather than of who measured it.

What is the 50% batch discount?

The 50% batch discount is the providers' OWN published price - OpenAI, Anthropic and Google all charge half the synchronous rate for batch on both input and output tokens. We quote it as theirs, never as a batchwatch measurement.

Do you collect our prompts or API keys?

No. A measurement is metadata only: provider, model, token counts, timestamps, status and region. There is no field for prompt text or model output, and we never receive your provider API keys - a client reports what its own job did, it does not route the job through us.

How we measure the LLM price tiers

What a probe does

A probe is one of our own measurement jobs. On a schedule we submit a small batch to each provider, exactly the way a customer's job would go in, and we record when it started and when the result came back - timed on our server, not on a client clock that can drift. Probe rows are marked as ours, so a dataset that is really one account can never masquerade as the world.

Queue time, and generation time

The number this site is built on is queue time: how long a batch job waits before the provider starts serving it. That is the part you cannot see and cannot plan around - the provider's backlog, not your job. It is what "within 24 hours" hides, and it is what we measure.

There is a second thing that adds to when a job finishes: generation time - how long the model takes to actually write the output. A job that produces two thousand tokens takes longer to finish than one that produces five, and that extra time is the model doing your work, not the queue holding it. We measure this too, on a small output ladder, and we report it separately: it is labelled as generation time and is never folded into the queue-time number. Keeping them apart is the honest thing to do - the queue time is the provider's to answer for, the generation time is your own workload's, and calling one the other would sell a precision that is not there.

The shape of the jobs we measure

Our probes are small by design; a real customer's batch can hold thousands of requests. A batch API's queue is a scheduler, so the size of a job - not only the model - can change how long it waits: a percentile measured on one-request probes need not describe a thousand-request batch. Rather than assume an answer, we measured it - the same job submitted at a range of batch sizes on one model, reported the way we report generation time: separately, and labelled.

The live effect - the exact ratio between a one-request probe and a large batch - is published at /v1/probe, recomputed as the data grows.

Until the prober submits representative batch shapes across the board, our own probes stay far smaller than a real customer's batch - so a percentile that leans on them describes that shape, while a contributed measurement carries whatever shape its author ran. Either way, we are the ones who checked, and the ratio above is what that check is worth to you: a correction you can apply to our numbers, measured rather than guessed at.

Crowdsourced contributions

Probes get us off the ground; contributions are what make a number represent everyone. Anyone can report a measurement - when their own batch started and finished - with or without an API key. A keyed contribution earns a vote in the published percentile, because a figure that many independent accounts agree on is worth far more than one large account's median. The more contributors behind a model, the higher its confidence grade climbs.

The percentile method

We draw a distribution once a model has at least 2 measurements - two points are the mathematical minimum for a curve. We only write a headline percentile, such as "nine in ten finished within X", once a model has at least 8 measurements. Below that threshold the points are shown and the reader can count them, but the claim is not made: a 90th percentile computed from three measurements is literally just the slowest of the three, and we will not dress that up as a percentile. Show the data; do not infer too much from it.

Where a model is still thin, we fall back to the provider's own published completion window - typically 24 hours - and label it as the contract it is, not a measurement.

How the number is computed

The sections above are the whole story for most readers. This one is for the engineer who wants to check the part that actually produces the p90 before routing production on it - so here is the estimator itself, in plain terms. Every claim in it is drawn from our public design of record, and a test fails our build if this page and that design ever disagree.

A still-running batch is data, not a gap. A job that has not come back yet has already taken at least as long as it has been waiting, and we use that: it enters the curve as a right-censored observation - a known lower bound on its true wait - rather than being dropped. The rule is an invariant we hold to everywhere: a censored observation never becomes a measurement, and it never disappears. Dropping the jobs that are still running is precisely how a careless method reports a queue as faster than it really is, and it is the single largest correction we make.

The p90 is read off a real curve. We build the distribution with the Kaplan-Meier survival estimator, which is what lets those censored jobs count, and place each measurement using Weibull plotting positions with linear interpolation. That convention is the one we adopted because it has a property you can check on paper - it puts each point at its expected rank - so the 90th percentile we quote is genuinely the 90th percentile, not the slowest of a small sample relabelled.

We do not reject outliers - on purpose. The slowest batches are the numbers a customer most needs, so trimming the tail would throw away the measurement. We know the cost precisely because we ran the alternative: an "8 x median" cut, measured on gpt-5-nano before we removed it, discarded 36% of the measurements and reported a p90 of 194 seconds when the true figure was 7817 - roughly forty times too low. Every measurement now stays in the percentile; we report separately how many sit above a heavy-tail threshold, so you see the shape without the number being bent. And no single account can run away with a figure: the pooled share of untrusted contributors is capped at 20%.

Delay and freshness

Public figures are precomputed and delayed by 15 minutes, and anyone may read them at that delay, for nothing. Contributing removes the delay altogether: send five completed measurements in seven days and every figure you read is live. Live access is earned by measuring, never bought. Every page writes an absolute timestamp for the last measurement - a fixed instant, never a relative age like "moments ago" - so a cached copy says the same true thing an hour later.

Confidence grades

Every number carries a grade that blends how much we measured, how fresh it is, and how wide the range is. How many independent contributors stand behind it is shown beside the grade rather than folded into it, so you can weigh breadth for yourself. A percentile a single source could still move is capped low - that is a property of the number, not of who measured it.

High. Many measurements, fresh and tightly grouped, on a percentile no single source could move. A number you can route production on.
Medium. Enough data to be useful, with a caveat worth reading - usually a wider range than we would like.
Low. Real but thin: few measurements, a wide spread, or a percentile still pooled across sources. Treat it as a hint, not a contract.
Very low. The signal barely clears the floor. We show the points so you can judge them yourself, and we do not write the headline claim.

The numbers we did not measure

The 50% batch discount is the providers' own published price - OpenAI, Anthropic and Google each charge half the synchronous rate for batch on both input and output tokens, and each documents a 24-hour completion window - and we quote both as theirs. When we estimate savings, The 30% batchable share is your own input - a deliberately conservative default that is yours to set, since only you know your own workload, and most teams can batch more. Our queue times are measured; the share is the one number you bring - and being straight about which is which is the whole product. We tell you which numbers we observed and which we assumed, because that line is the one everyone else blurs.

What we do not collect

No prompts. No keys. No content.

A measurement is metadata only: provider, model, token counts, timestamps, status and region. There is no field anywhere in the API for prompt text or model output, so we could not store your content even if we wanted to. We never receive your provider API keys either - your client reports what its own job did; it does not route the job through us. On the free trial we identify a caller by a salted hash, never a raw IP address.

Questions about the method

Where do the queue times come from?: Every figure is an observed completion of a real batch job. A client opens a measurement when it submits a batch and closes it when the provider returns the result; the wait is the difference, timed on our server rather than the client's clock. Nothing is predicted and nothing is simulated.
What is a probe?: A probe is one of batchwatch's own measurement jobs: we submit a small batch to a provider on a schedule and record how long it takes, exactly as a customer's job would. Probe rows are marked as ours and can be filtered out, so a dataset that is really one account says so instead of hiding it.
How does crowdsourcing work?: Anyone can contribute a measurement, with or without an API key, by reporting when their own batch started and finished. Contributions are what make a percentile represent the world rather than one account, and only a keyed contribution earns a vote in the number.
How do you turn measurements into a percentile?: We publish a curve once a model has at least 2 measurements, and a headline percentile such as "nine in ten finished within X" only once it has at least 8. Below that the points are shown but the claim is not written, because a 90th percentile drawn from three points is just the slowest of the three.
How is the p90 actually computed?: We build a survival curve with the Kaplan-Meier estimator using Weibull plotting positions with linear interpolation, then read the 90th percentile off it. That convention has a property we can check - it places each measurement at its expected rank - so the p90 is a real percentile, not the largest of a handful of points dressed up as one. The full derivation is public in our estimator design of record.
What happens to a batch that is still running?: It is kept, not thrown away. A job that has not returned yet is right-censored data - we know it has already taken at least this long - and the survival curve uses that lower bound instead of pretending the job does not exist. The rule is an invariant: a censored observation never becomes a measurement, and it never disappears. Dropping still-running jobs is exactly how a naive method reports a queue as faster than it is.
Do you reject outliers?: No, and that is deliberate. A slow batch is the number a customer most needs to see, so cutting the tail would hide the very thing we measure. We tried the opposite - an "8 x median" cut - and measured the damage on gpt-5-nano before removing it: it discarded 36% of the measurements and reported a p90 of 194 seconds when the true figure was 7817, roughly forty times too low. We keep every measurement in the percentile and report separately how many sit above a heavy-tail threshold.
Can one big account skew the number?: A single contributor cannot run away with a percentile. We estimate the law of a typical job as a mixture across contributors by their share, and the pooled share of untrusted, un-established contributors is capped at 20%. A dataset that is really one account is graded low however large it is, because it measures that account, not the queue.
Why are the public numbers delayed?: Public figures are precomputed and delayed by 15 minutes, and that costs nothing to read. Contributing removes the delay: five completed measurements in seven days and every figure you read is live. The page writes an absolute timestamp for the last measurement, never a relative age, so a cached copy stays true.
What does a confidence grade mean?: Each number carries a grade - high, medium, low or very low - that blends how many measurements there are, how fresh the newest one is, and how wide the range is. How many contributors stand behind it is reported next to the grade rather than folded into it, so you can weigh breadth yourself. A figure a single source could still move - a pooled percentile - is capped low, because that is a property of the number rather than of who measured it.
What is the 50% batch discount?: The 50% batch discount is the providers' OWN published price - OpenAI, Anthropic and Google all charge half the synchronous rate for batch on both input and output tokens. We quote it as theirs, never as a batchwatch measurement.
Do you collect our prompts or API keys?: No. A measurement is metadata only: provider, model, token counts, timestamps, status and region. There is no field for prompt text or model output, and we never receive your provider API keys - a client reports what its own job did, it does not route the job through us.

See it for yourself

All of it is public.

The live dashboard shows every model with its distribution, its n and its confidence grade. /v1/coverage is the same data as JSON, no key required, and the questions answer the searches people actually type. Contribute measurements and you help build the number you are reading.