The one number, and who measures it
| Product | What it measures | The provider’s batch queue? |
|---|---|---|
| batchwatch Measured batch-queue queue time, per model. | The real wait on the provider’s cheap tiers — median (p50) and p90 per model, plus how often flex refuses to run, with the sample size and a confidence grade behind every number. | This is what we publish Ranked, live, and free at /compare. |
| Helicone LLM observability and an AI gateway (open source). | Logs and times the API calls your own application makes — cost, latency and tokens per request, grouped into sessions and users. Its public status page reports whether Helicone itself is up. | It measures the requests your app sends, not the provider’s batch queue, so there is no measured batch-queue wait to read from it. |
| OpenRouter An LLM routing gateway across many providers and models — and it has a real batch API. | Publishes per-model latency and throughput percentiles (p50–p99) measured on live streaming calls over a rolling five-minute window — time-to-first-token and tokens per second. | Those percentiles are the speed of a synchronous call; the batch model pages show no measured queue-time distribution for the async :batch tier. It runs the batch job; batchwatch times how long that tier waits. |
| Langfuse An open-source LLM engineering and observability platform. | Traces each step of your application — LLM calls, retrieval, tool use — with per-operation latency, token usage and cost, for debugging and evaluation. | It times the operations inside your app, not the provider’s batch queue, so it does not publish a batch-queue wait either. |
| Portkey An AI gateway with routing, governance and observability. | Its "unified batch inference" runs and routes batch work across providers (on enterprise plans) and logs your traffic — it is execution and control, not a published measurement. | It runs the batch; it does not publish how long the provider’s queue took. |
| Cloudflare AI Gateway A managed LLM proxy with caching, logging and analytics. | Logs each request through the gateway — prompt, response, tokens, cost and duration — with caching and rate limiting. | Per-request duration through the proxy is a synchronous-call metric, not a measured distribution of the async batch queue. |
Each product’s row is what it measures, checked against its own live site on 2026-08-29 — follow the link to verify it. These are statements about what a product measures, not about whether it is good; every one of them is worth using for what it is for.
The status pages — "is it up?" is not "how long is the wait?"
The other place a buyer looks when the queue feels slow is a status page. A status page is the right tool for one question — is the service up? — and that is a different question from ours. "Up" and "the batch queue is running six hours behind today" are both true at the same moment, and only the second one moves your deadline.
| Status page | What it measures | The provider’s batch queue? |
|---|---|---|
| Downdetector Crowdsourced outage reports across thousands of services. | Surfaces spikes in user-reported problems to tell you whether a service looks down right now. | It reports whether the service is up, not how long the provider’s asynchronous batch queue is running behind. |
| StatusGator A status-page aggregator with website uptime monitoring. | Watches thousands of providers’ official status pages and pings endpoints, and alerts you when one reports an incident or goes down. | It aggregates up/down status, not a measured distribution of how long the batch tier waits. |
| Atlassian Statuspage The tool a provider uses to publish its own service status. | Lets a provider communicate the operational state of each part of its service — up, degraded, down or in maintenance. | It publishes the service’s own up/down state; it does not time the batch queue, and a provider has little reason to. |
Each status page above was checked against its own live site on 2026-08-29 — follow the link to verify it. Downdetector, StatusGator and Statuspage are good at reporting whether a service is up; none of them separates the health of the asynchronous batch queue from the health of the synchronous endpoint.
The wider routing and observability landscape
LiteLLM, Requesty, Vellum and the rest of the routing-and-observability landscape sit alongside the gateways above: they route your calls, or log and time them, and they do it well. As of 2026-08-29, none of them publishes a measured distribution of how long the provider’s asynchronous batch queue waits, per model — which is the one number this page is about. We name them so your tool is accounted for, and we keep the claim at the level we can stand behind rather than inventing a feature line for each.
The half-price tier nobody times
The asynchronous batch API costs 50% of the synchronous rate — the providers’ own published price, the same at all three (OpenAI, Anthropic, Google). What you trade for the half price is a queue wait behind a "within 24 hours" ceiling, and that wait is the number every gateway and dashboard leaves unmeasured. batchwatch measures it, per model, and ranks it at /compare.
Why a gateway can’t simply publish this
The probe is copyable in a week; the dataset is not. A measurement that represents everyone needs real, diverse batch jobs across many providers, models, regions and times of day — and a gateway only ever sees its own customers’ traffic, which is narrower and self-selected. A provider, for its part, has no reason to publish that its own queue runs slower than a rival’s. The crowdsourced breadth, the published method, and the history that builds over time are the defensible part — and every measurement a contributor sends makes it broader.
What stands behind every number here
The reason to trust this page is that it shows its working. Every figure at /compare carries its sample size (n), its source, and a confidence grade, and we publish a percentile only once a model has at least 8 real measurements — below that we show the honest count and say so, because a percentile we would not stand behind is worth less than the plain number of jobs we have seen. The full method is public at /how-we-measure and the raw data is at /v1/coverage, no key required. That is the part a competitor cannot copy in a week, and it is the same standard we hold every competitor claim above to: named, dated, and linked to a source you can open.
Questions this answers
- Isn’t this just Helicone or Langfuse?
- No — they solve a different problem well. Helicone and Langfuse are observability: they log and time the calls your own application makes, so you can debug and cost them. batchwatch measures the one thing that happens outside your app entirely — how long the provider’s asynchronous batch queue takes — across many accounts, not just yours. If you run batch jobs you likely want both.
- OpenRouter already shows latency percentiles — how is this different?
- OpenRouter’s percentiles measure the speed of a live streaming call (time-to-first-token and tokens per second) over a rolling five-minute window. That is a real and useful number, and it is a synchronous one. batchwatch measures the asynchronous batch tier instead: how long a submit-and-wait job sits in the queue before its result is ready, which is a different question with a different answer.
- What about a status page — Downdetector, StatusGator, Statuspage?
- A status page tells you whether a service is up or down. "Up" and "the batch queue is running six hours behind today" are both true at the same time, and only one of them changes your deadline. Downdetector, StatusGator and Statuspage each report the first — official status, crowdsourced outage reports, or a published status board. batchwatch measures the second one: the queue wait itself.
- What stops OpenRouter or a provider from just doing this next month?
- The probe is the easy part; the dataset is the hard part. A measurement that represents everyone needs real, diverse batch jobs across many providers, models, regions and times of day — and a gateway only ever sees its own customers’ traffic, while a provider has no reason to publish that its own queue is slower than a rival’s. That crowdsourced breadth, plus a published method and a growing history, is the part that is hard to copy — and every measurement you contribute makes it broader.
- How do I know your numbers are real?
- Because we publish the method and the evidence. Every model shows its sample size and a confidence grade, and we only publish a percentile once a model has at least 8 measurements — below that we show the honest count instead of a figure we could not stand behind. The full method is at /how-we-measure and the raw data is at /v1/coverage, no key required.
See the measurement
The number none of them publishes is one click away, and it is public.
The head-to-head at /compare ranks every batch model by its measured median, fastest first. How we measure lays the method bare, and /v1/coverage is the same data as JSON, no key required. Send five measurements in seven days and the delay is gone: you read every figure live, and the decision endpoints open (/v1/should-i-batch, /v1/estimate-batchtime).