Method

The rules below are policy, written by humans. Everything on a category or run page that looks like a result (numbers, ranks, statuses) is generated from the run bundle, never written by hand.

1. The protocol is frozen before the start

Each combat has a pre-registered analysis plan and a protocol lock (PROTOCOL.lock): objective, primary metric, secondary metrics (full cost, latency, incidents), eligible population, assignment rule, budget per tool, time window, exclusions, stopping rules. Its hash is committed before the first API call, and gladiator verify proves the lock predates every event.

2. Apples to apples

All tools run on the same frozen dataset (or on disjoint, matched, randomly assigned blocks when the task consumes its data), with the same budget, the same window and the same operator care. Tools without a machine interface are marked access.api = false rather than silently excluded.

3. Uncertainty is part of the result

Every estimate carries an interval. If the precision is insufficient, the status is INDETERMINATE or INSUFFICIENT_EVIDENCE and no rank is published. A rank requires replication on at least two blocks or windows; a single run is published as locally valid only.

4. Everything is reproducible

A run is a bundle: protocol, append-only event log, scoring code, costs, conflicts, hashes. gladiator reproduce <RUN> recomputes the published result from the published (pseudonymized) data without any oral explanation. Raw re-identifying events never enter the public bundle.

5. Neutrality is structural

No vendor money in public comparisons; funding is disclosed per run. Any devlo-related product is excluded from rankings and marked as such in the conflict register. Terms of service are read before each category; a tool whose terms forbid benchmarking is BLOCKED and reported as market state, never worked around.

6. One canonical result, three projections

The JSON bundle is the source of truth. The CLI, the MCP server and this site are projections of the same file; a contract test (SURFACE_PARITY) verifies they answer identically. The answer rule on every surface: never “the best tool”, always the best observed evidence for this task, this population, at this date, with interval, cost, conflicts and limits.