batchwatch

batchwatch › Reading an answer

Reading an answer

Most responses carry the same four disclosure fields: confidence, basis, n and freshness. They exist because the alternative — a single number with no provenance — cannot be checked by the caller, and this product's only asset is that it can be checked.

Most responses carry the same four disclosure fields: confidence, basis, n and freshness. They exist because the alternative — a single number with no provenance — cannot be checked by the caller, and this product's only asset is that it can be checked.

Source: src/confidence.js, src/robust.js, src/stats.js.


Why there is never a point ETA

You will not find a field that says "your job will finish in 23 minutes". The closest thing is estimate_batchtime_s, and even that is described in the response itself as "measurement": "observed completions of comparable jobs" and "not": "a model of how your job will behave". /v1/wait says it even more bluntly: "not": "a prediction for your job".

Three reasons, all in the code:

1. The measurement is of the queue, not of your job. Batch queue time is mostly the provider's backlog. /v1/estimate-batchtime will tell you so directly when it can: when jobs of your size are indistinguishable from jobs of other sizes, size_effect.note reads "Size makes no measurable difference here... The number below is really just the queue."

2. The two errors are not equally expensive. Being wrong towards sync costs double price. Being wrong towards batch can blow a deadline. So the judgement is never made on the midpoint: it is made on the upper end of the interval, and where the tail cannot be bounded at all, on the provider's own published completion window. That is why you routinely see plan_for_s: 86400 sitting next to an observed p90 of 1144s. The observed figure is what happened; the planning figure is what a deadline has to survive.

3. Every number carries the width of the ignorance behind it. A distribution-free interval built on four measurements simply cannot bracket a p90, and the code says so by leaving hi as null and hiBounded as false rather than inventing a bound.

So the shape of an honest answer is: here is what was observed, here is how wide the uncertainty is, here is the number you should plan against, and here is why they differ.


n

The number of completed measurements behind the answer, after two filters:

n is not the same as the number of rows used to draw a curve. No single source is allowed to dominate: each source is trimmed to at most half of what all the others contribute together, and the trim is an even sample in sorted order, not the first N — taking the first N would take whichever rows happened to be inserted first and skew the shape. With only one source there is nothing to trim against, and nothing is trimmed; in that case the low contributors count and the pooled robustness method are what tell you so.

One important trap for anyone reading responses programmatically: when the answer came from a precomputed rollup, the individual durations are not carried, only n and the computed figures. Use n to decide whether measurements exist, never the length of any array.

contributors

Distinct keys behind the measurements, with everything key-less counted as one source. This is an upper bound on how independent the data is — it counts keys, not sources that have earned a vote.

A single contributor caps confidence.level at low, however many measurements there are. A dataset from one account measures that account's setup, not the queue, and the code cannot tell the difference.

basis

Where the figures came from, best to worst:

ValueMeaning
exact_modelMeasurements of this exact provider/model/mode.
provider_pooledFewer than 10 measurements for the exact model, so measurements were pooled across that provider's models. confidence.score is multiplied by 0.7.
sla_priorNo measurements at all. The number is the provider's published completion window — 24 hours for every provider in SLA_WINDOW_S — and confidence.level is none with score: 0.

/v1/estimate-batchtime uses a different basis vocabulary for the same field: size_matched, all_sizes or sla_prior.

plan_basis (and planning_basis on /v1/should-i-batch) is a separate field with its own two values: interval_upper when the tail could be bracketed, sla_prior when it could not.

no_coverage

One boolean, the same meaning on all four query routes — /v1/wait, /v1/should-i-batch, /v1/estimate-batchtime and /v1/curve. It answers the question you actually asked: do you have measurements of this model?

ValueMeaning
trueNo. Nothing was measured for this provider/model/mode, or too little was, so the answer came from the provider's published window or from other models on the same provider.
falseYes. There are measurements of this exact model behind the figures. How thin the evidence is lives in confidence — that is a sliding scale and does not belong in a boolean.
nullWe could not determine it from the response. Treat it as unverified, and please report it: it is our bug. Absence is reported as absence, never as false.

Alongside it, coverage says where the numbers came from:

{
  "no_coverage": true,
  "coverage": {
    "determinable": true,
    "measurements_behind_answer": 3,
    "contributors": 1,
    "window": "30d",
    "reason": "pooled_substitute",
    "numbers_from": "measurements_provider_pooled",
    "why": "We answered from other models on the same provider, pooled, ...",
    "graded_by": null,
    "basis": "provider_pooled"
  }
}

numbers_from has three values, not two:

Value
measurements_exact_modelThe model you asked about.
measurements_provider_pooledReal measurements — of other models on that provider. Useful as a shape, not an answer about yours.
sla_priorNothing measured. The provider's published window.

measurements_behind_answer is named for what it is. There is no contradiction between no_coverage: true and measurements_behind_answer: 3: three measurements are behind the number, they are simply not of your model.

Why the line is drawn at "nothing for this model" rather than at a minimum count: the thresholds differ per route on purpose — a curve draws on two points, a full distribution with an hourly profile needs twenty for privacy reasons. A no_coverage that tracked the threshold would mean four different things under one name, which is the exact problem the field exists to remove.

Before this field existed you had to derive the same fact from n, verdict, basis and confidence — four fields, and any client that got the derivation wrong would plan a deadline against a brochure figure.

confidence

An object, on most routes:

{
  "level": "very_low",
  "score": 0.21,
  "basis": "exact_model",
  "n": 24,
  "contributors": 3,
  "newest_age_s": 85633,
  "why": "Limited by an upper bound we cannot pin down, nothing measured for 24 hours."
}

score is the geometric mean of four independent weaknesses, so any one of them is enough to pull the whole thing down:

FactorFormulaReads as
Quantityn / (n + 20)Few measurements.
Spreadcontributors / (contributors + 2)All from one place.
Width1 / (1 + log2(upper / point))counted twiceAn interval from 2 minutes to 9 hours cannot carry a decision.
Freshness1 / (1 + age_s / 86400)Last month's measurements do not describe today's queue. Half-life one day.

Width counts double because the other three describe how well we measured; width describes whether the answer can be used at all.

level is banded from scorehigh at 0.62, medium at 0.45, low at 0.28, very_low below — and then three caps are applied:

  1. Fewer than 2 contributors caps it at low.
  2. robustness.method === "pooled" caps it at low.
  3. An interval more than 25× the point caps it at very_low; more than 8× caps it at low.

why names the specific weaknesses in plain language, so the level can be acted on without reverse-engineering the score.

Two routes use a different, coarser confidence. On /v1/conditions and /v1/distribution it is a bare string from confidenceOf(n, keys, minN, minKeys): insufficient, low, medium or high, based purely on counts against MIN_N and MIN_KEYS. On /v1/coverage it is the level string from the object version. Check the type before you parse it.

robustness

How the percentile was computed, and this is the field that says whether one source could have moved the number.

methodMeaning
per_contributorAt least 3 sources had earned a vote. The percentile is computed inside each source and then the median of those is taken. More data cannot move it; more accounts cannot move it. Only being in the majority of established contributors can.
pooledFewer than 3 sources could vote, so it is an ordinary pooled percentile and a single source could move it. confidence.level is capped at low for exactly this reason.
noneNo measurements.

contributors_voting is how many sources actually voted. note explains the method in prose, because a method the caller cannot check is not a method.

The asymmetry is deliberate and worth understanding: the interval is still computed on the whole pooled set, so a bad actor can drag its upper end up. That is the harmless direction — it makes the service more cautious, recommends sync, and costs money rather than deadlines. Dragging the point down is the dangerous direction, and that is what the votes close off. In the same spirit, with an even number of votes the code takes the upper of the two middle values rather than their average.

This is also why p50_s can differ between routes for the same data: /v1/wait and /v1/curve report the robust per-contributor figures, while /v1/distribution reports distribution() computed straight over the pool. Both are correct; they answer slightly different questions.

freshness

"freshness": {
  "last_measurement_at": "2026-08-25T13:39:43.000Z",
  "last_measurement_age_s": 1497,
  "last_measurement_age": "25 min",
  "oldest_measurement_at": "2026-08-25T13:39:43.000Z",
  "stale": false
}
FieldMeaning
last_measurement_atWhen the newest measurement behind this answer finished. ISO-8601, UTC.
last_measurement_age_sIts age in seconds, never negative.
last_measurement_ageThe same, humanised: 25 min, 24 hours, 3 days.
oldest_measurement_atThe far end of the window. /v1/wait only.
staletrue when the newest measurement is more than 24 hours old.

Freshness is separate from delayed_by_s. The delay is what your tier lets you see; freshness is how old the underlying data actually is. A paid caller with delayed_by_s: 0 can still get stale: true if nobody has measured that model today.

Five hundred measurements from last month look exactly like five hundred from this morning unless the age is stated — that is why it is in every answer, and why it is one of the four factors in the confidence score.

A worked reading

Taking the real production response from /v1/wait on 2026-08-25:

The correct reading is: one job on this model took 19 minutes, and that is all anyone knows; do not build a deadline on it. The response says all of that without a single field having to be interpreted charitably.