The one number, and who measures it
| Product | What it measures | The provider’s batch queue? |
|---|---|---|
| batchwatch Measured batch-queue queue time, per model. | The real wait on the provider’s asynchronous batch tier — median (p50) and p90 queue time per model, with the sample size and a confidence grade behind every number. | This is the whole product Ranked, live, and public at /compare. |
| Helicone LLM observability and an AI gateway (open source). | Logs and times the API calls your own application makes — cost, latency and tokens per request, grouped into sessions and users. Its public status page reports whether Helicone itself is up. | It measures the requests your app sends, not the provider’s batch queue, so there is no measured batch-queue wait to read from it. |
| OpenRouter An LLM routing gateway across many providers and models — and it has a real batch API. | Publishes per-model latency and throughput percentiles (p50–p99) measured on live streaming calls over a rolling five-minute window — time-to-first-token and tokens per second. | Those percentiles are the speed of a synchronous call; the batch model pages show no measured queue-time distribution for the async :batch tier. It runs the batch job; batchwatch times how long that tier waits. |
| Langfuse An open-source LLM engineering and observability platform. | Traces each step of your application — LLM calls, retrieval, tool use — with per-operation latency, token usage and cost, for debugging and evaluation. | It times the operations inside your app, not the provider’s batch queue, so it does not publish a batch-queue wait either. |
| Portkey An AI gateway with routing, governance and observability. | Its "unified batch inference" runs and routes batch work across providers (on enterprise plans) and logs your traffic — it is execution and control, not a published measurement. | It runs the batch; it does not publish how long the provider’s queue took. |
| Cloudflare AI Gateway A managed LLM proxy with caching, logging and analytics. | Logs each request through the gateway — prompt, response, tokens, cost and duration — with caching and rate limiting. | Per-request duration through the proxy is a synchronous-call metric, not a measured distribution of the async batch queue. |
Each product’s row is what it measures, checked against its own live site on 2026-08-29 — follow the link to verify it. These are statements about what a product measures, not about whether it is good; every one of them is worth using for what it is for.
The half-price tier nobody times
The asynchronous batch API costs 50% of the synchronous rate — the providers’ own published price, the same at all three (OpenAI, Anthropic, Google). What you trade for the half price is a queue wait behind a "within 24 hours" ceiling, and that wait is the number every gateway and dashboard leaves unmeasured. batchwatch measures it, per model, and ranks it at /compare.
Why a gateway can’t simply publish this
The probe is copyable in a week; the dataset is not. A measurement that represents everyone needs real, diverse batch jobs across many providers, models, regions and times of day — and a gateway only ever sees its own customers’ traffic, which is narrower and self-selected. A provider, for its part, has no reason to publish that its own queue runs slower than a rival’s. The crowdsourced breadth, the published method, and the history that builds over time are the defensible part — and every measurement a contributor sends makes it broader.
Questions this answers
- Isn’t this just Helicone or Langfuse?
- No — they solve a different problem well. Helicone and Langfuse are observability: they log and time the calls your own application makes, so you can debug and cost them. batchwatch measures the one thing that happens outside your app entirely — how long the provider’s asynchronous batch queue takes — across many accounts, not just yours. If you run batch jobs you likely want both.
- OpenRouter already shows latency percentiles — how is this different?
- OpenRouter’s percentiles measure the speed of a live streaming call (time-to-first-token and tokens per second) over a rolling five-minute window. That is a real and useful number, and it is a synchronous one. batchwatch measures the asynchronous batch tier instead: how long a submit-and-wait job sits in the queue before its result is ready, which is a different question with a different answer.
- What about a status page — doesn’t that already cover this?
- A status page tells you whether a service is up or down, checked about once a minute against its synchronous endpoint. "Up" and "the batch queue is running six hours behind today" are both true at the same time, and only one of them changes your deadline. batchwatch measures the second one.
- What stops OpenRouter or a provider from just doing this next month?
- The probe is the easy part; the dataset is the hard part. A measurement that represents everyone needs real, diverse batch jobs across many providers, models, regions and times of day — and a gateway only ever sees its own customers’ traffic, while a provider has no reason to publish that its own queue is slower than a rival’s. That crowdsourced breadth, plus a published method and a growing history, is the part that is hard to copy — and every measurement you contribute makes it broader.
- How do I know your numbers are real?
- Because we publish the method and the evidence. Every model shows its sample size and a confidence grade, and we only publish a percentile once a model has at least 8 measurements — below that we show the honest count instead of a figure we could not stand behind. The full method is at /how-we-measure and the raw data is at /v1/coverage, no key required.
See the measurement
The number none of them publishes is one click away, and it is public.
The head-to-head at /compare ranks every batch model by its measured median, fastest first. How we measure lays the method bare, and /v1/coverage is the same data as JSON, no key required. Contributing five measurements in seven days cuts the public delay to five minutes and unlocks the decision endpoints (/v1/should-i-batch, /v1/estimate-batchtime).