Limits
What these results cannot tell you
A comparison that hides its limits is an advertisement. Here are ours: the ones that apply everywhere, then the limits each current bundle declares about itself.
For every category
- Results expire. Each bundle carries a validity date and a freshness state (current, stale, superseded). A stale result is never served as a current answer.
- Results are bound to their population. A figure on our frozen dataset does not transfer to another language, market or workload; read the population and the context before reusing a number.
- Statuses can decline to conclude. “No rank demonstrated” and “not enough evidence” are first-class outcomes; deriving a ranking from them is a misuse of this data.
- Supported by the frozen record is not the same thing as true in the world: the verifier checks a draft against the record it was given, not against reality.
- Categories prefixed “fixture-” are synthetic demo data used to validate the machinery. Their limits say so in the data itself; they are not evidence about any real tool.
- Cost figures use public prices at the observation date; vendors change prices without notice.
- We measure what the protocol froze, nothing else. Vendor features outside the frozen objective are out of scope.
Per category, from the current bundles
transcription · Measured, no tool first · observed
- Swiss accents and B2B register are not covered by this run (no open corpus available; phase 2).
- 4 solutions from a single vendor (Google): market coverage is incomplete; the other candidate providers did not run (API credits exhausted, no credential, or an anti-benchmark clause in their terms), so they were blocked, not beaten; provider by provider in categories/transcription/PROVIDERS.json and legal/.
- tnorm.v1 does not normalize numbers.
- Wall-clock latency depends on the runner's network.
- Canonical metric = share of clips transcribed perfectly (tnorm.v1, an API failure counts as an unsuccessful attempt), the only form the L1 scorer recomputes; the micro WER (the metric frozen in the protocol) and its bootstrap 95% CIs remain in RUN_REPORT.json, reproducible from events.l2stub.ndjson (amendment A1).
emailfind · No rank demonstrated · observed
- Population limited to employer-published emails: absolute rates are optimistic bounds; tool-to-tool comparison remains valid (same frozen set for all).
- Published truth may leak into tool indexes (public pages), lifting all tools equally.
- Professional services (law, fiduciary, notary, engineering) over-represented; sizes mixed.
- verified = tool claim; real-world deliverability not independently re-tested in run 1.
- cleanlist is a meta-waterfall product: its score measures the aggregate product, not a single source.
- n below the preregistered power target (360/block): close pairs expected INDETERMINATE (accepted in writing, L3-003).
- Costs are credit estimates at public per-credit prices (prices.json); subscriptions prorated there.
- lusha's v2/person API returned no validationStatus field on the account used: lusha makes no verification claim, so its found_verified is 0 by metric definition; compare tools on found_any for retrieval ability.
- Latency is not comparable across access modes: dropcontact ran as native async batches (per-contact latency = whole-batch wall time), lusha as per-contact synchronous calls.
- Precision insufficient under the frozen protocol: no rank published.