EnviroBench

1Design

EnviroBench separates what is measured from what is being measured. The public core defines benchmark types, answer formats, scoring rules, the adapter protocol and the report. It does not run any product or model. Datasets come from plugins, and each system under test is an adapter. An organization can keep its own plugin, adapter and object store private while using the same public scoring.

2Tasks and metrics

A benchmark type is one class in lib/envirobench/benchmarks/. It declares an identifier, a result format, a scoring version and its score profiles, and implements score_case, which returns, for each profile, the case's errors and metrics. Errors are comparable values, so the same mistake in two runs is recognized as the same error.

2.1Errors and headline metrics

Each profile is a separate way of scoring the same prediction. The first profile is the default gate. Headline metrics summarize a run; the gate uses only per-case errors.

Scoring versions are recorded in every run, for example envirobench.table-separators.scores.v1. A change to a formula gets a new version.

2.2Tiers

Each benchmark type declares a tier: what a case hands the system. The component tier hands it a crop, a cell or a page; the document tier one whole PDF; the site tier a site's whole document record, a few hundred pages of reports, certificates, letters, invoices and scans, most of them holding nothing that is asked for. Real work starts at the site tier: finding the pages that matter, reading them, and reconciling the same results reprinted across years of reports, drafts and finals, scanned copies and correction letters. The lower tiers show where a system loses accuracy on the way.

At the site tier cost and wall time per site are part of the result, reported beside every score: a system that is accurate only at a high price per site is a different product from one that is accurate cheaply, so site boards sort by both. Three public sites (a service station, a bulk plant on railway lands and a dry cleaner, from 275 to 806 pages) are invented and generated with their answers; real site records stay in private plugins.

2.3Category scores

Leaderboards summarise the tasks in six categories: Extract, Verify, Locate, Research, Legal and Site. Each component task is first turned into a score from 0 to 100, higher better. By default the component score is 100 × the task's primary headline measure; every current primary measure is a share or a mean between 0 and 1 where higher is better, and a lower-is-better primary would need its own published formula. Two tasks built around one dangerous error are the exceptions. OCR auto-accept scores 100 × max(0, (correct accepts − 10 × wrong accepts) ÷ readable cells), so one wrong value that would skip review cancels ten correct accepts and coverage cannot buy back a wrong accept cheaply. Instrument execution scores 100 × max(0, (correct statuses − 5 × instruments called executed when not) ÷ instruments), so a release or transfer wrongly reported as signed cancels five correct statuses, and calling every instrument executed scores 0. Party roles scores 100 × max(0, (parties with every role right − 5 × non-owners called owners) ÷ parties), the same weight for a party wrongly made an owner, and indemnity terms 100 × max(0, (contracts read exactly right − 5 × contracts overstating environmental protection) ÷ contracts), and chain of title 100 × max(0, (periods exactly right − 5 × periods from another parcel or invented) ÷ periods). In Research, missing information scores 100 × max(0, (questions right − 5 × invented answers) ÷ questions).

A category score is the weighted mean of its component scores, Σ w_is_i ÷ Σ w_i. Weights are per task, not per case, so a large test set does not outweigh a small one. Every weight is 1 except document extraction in Extract.

Document extraction is Extract's end-to-end task: a whole PDF in, measurement records out. It is the outcome environmental professionals actually use, so it has weight 4, as much as the four table tasks together (4 of the category's 9). The table tasks show where a system loses records on the way. Its primary measure is correct-record F1: a record counts only when it is found and its value is right.

• Complete or nothing. A system gets a category score only when it has results for every component task, on the test-set versions the leaderboard shares. Otherwise the category shows Incomplete (n of m tasks). A missing task is never counted as zero and never dropped.
• Repeats. With repeated runs, the score is the weighted mean of each task's mean, shown with a range: the weighted mean of each task's lowest and highest repeat.
• No overall score. The categories cover very different amounts of work, so they are not combined into one number.

The definition lives in core (lib/envirobench/categories.rb) with its own scoring version; the table below is generated from it.

3Evaluation protocol

One run covers one benchmark and one system. Figure 1 shows the lifecycle. The boundary between core and adapter is the only place data crosses, and answers never cross it.

3.1Answer isolation

The adapter is started as ADAPTER run --request FILE --output DIR. The request holds the run ID, model, spend limit, adapter options and an input-only view of the dataset: cases and a run-specific directory of checked source files. It never contains answer rows, the answer file's path, earlier runs' predictions or source links. The adapter returns only the candidate's predictions. Core checks each one with the type's validator; a technical failure blocks the report (exit status 1) and is never counted as a score.

3.2Paired comparison and the regression gate

A run is compared with saved runs rather than with a second model in the same run:

• --baseline-run DIR is the run the change is compared with.
• --control-run DIR, repeatable, is a same-code repeat of the baseline. A case is unstable in a profile when the baseline and any control disagree. Its regressions are reported but do not fail the gate. More controls catch more noise.
• --target-case ID names cases the change must get right. Any remaining error on a target fails the gate.

Per profile, a case regresses when the candidate has an error the baseline did not have. The gate passes when no stable case regressed and every target is error-free. Without a baseline the status is baseline required; when a profile cannot score every case and run, it is unavailable. A type may redefine when a case is worse: search reranking compares nDCG@10 and essential passages instead of passage-level errors, which change under any reordering.

3.3Offline rescoring

score --run DIR rescores a completed run from saved outputs without calling a model. It can re-pair with other baselines or controls, change the gate or targets, and re-key the run against a corrected revision of the same dataset when every input is unchanged, so an answer-key correction does not need a paid rerun.

3.4Review

Each run writes review-brief.json, which groups changed cases into worse, better, noisy and remaining, by category. review draft turns the brief into an unexamined draft with one numbered finding per group. A reviewer edits it into interpretation.json with evidence for every examined case. A later run can target the cases behind a recommendation with --from-review RUN_DIR --recommendation N. Reviews never change scores or the gate.

3.5Report

report.html is one file that makes no network requests: the verdict, headline metrics with changes from the baseline, coverage, and every case grouped by bucket and category, with a viewer for the type (table grid, image overlay or JSON). Image viewers load case images from a sibling report-assets/ directory. serve --run DIR serves a run read-only on 127.0.0.1 for browsers that block local file links.

4Datasets

A dataset is a directory with manifest.json, cases.jsonl, a separate answers.jsonl and the source files the cases use, each pinned by size and SHA-256. Its identity (ID, version and digest) is recorded in every run, and a baseline from another revision is refused. A revision may retire cases with a stated reason.

Core refuses to upload a private dataset to a public store, or to point a public dataset at a private one. Uploads verify each file's checksum, never replace a stored object with different bytes and never delete. Promoted baseline and control runs can be archived to a private store and are restored, checksum-verified, before a paired run on another machine.

5Running a benchmark

Requirements: Ruby 3.3.2. There are no third-party gems.

bin/test
bin/envirobench benchmarks
bin/envirobench datasets
bin/envirobench validate-dataset benchmarks/table-separators/datasets/synthetic-v1

Run a synthetic dataset through the reference adapter, which calls no model, then compare a second run with the first and open the report:

bin/envirobench run --adapter examples/reference-adapter --run separators-base \
  --model reference --benchmark table-separators \
  --dataset table-separators=envirobench/table-separators-synthetic-v1 \
  --adapter-option variant=baseline --budget 0
bin/envirobench run --adapter examples/reference-adapter --run separators-new \
  --model reference --benchmark table-separators \
  --dataset table-separators=envirobench/table-separators-synthetic-v1 \
  --baseline-run results/separators-base --gate logical --require-no-regressions --budget 0
bin/envirobench serve --run results/separators-new --open

Each run writes one directory:

results/<run-id>/
  run.json             how the run was made, dataset identity, gate result
  adapter-request.json input-only request
  adapter/             adapter-owned files, never modified by core
  comparison.json      answers joined with every compared run's predictions
  metrics.json         per-profile scores, gates and per-case errors
  report.html          the report; works offline
  report-assets/       case images for image viewers
  summary.md           plain-English verdict and next steps
  review-brief.json    changed cases grouped for a reviewer

5.1Saved profiles

A plugin can save how it runs each benchmark in benchmarks/NAME/bench.json: the dataset, adapter and options, gate, budget, the promoted baseline and controls per adapter, and a run store. bench turns a profile into the full run command:

bin/envirobench bench run table-separators --plugin ../acme-plugin --unpaired --run base
bin/envirobench bench promote table-separators ../acme-plugin/.private/runs/base \
  --control ../acme-plugin/.private/runs/control --plugin ../acme-plugin
bin/envirobench bench run table-separators --plugin ../acme-plugin --run my-change --plan
bin/envirobench bench run table-separators --plugin ../acme-plugin --run my-change

--adapter NAME|PATH selects the system under test. Baselines are kept per adapter, so a new adapter runs --unpaired until a baseline is promoted for it. --plan prints the commands without writing files or calling a model.

6Extending EnviroBench

6.1Adding a benchmark type

A new type is one Ruby file, a synthetic dataset and two prediction fixtures:

lib/envirobench/benchmarks/table_separators.rb       the type
benchmarks/table-separators/datasets/synthetic-v1/   manifest.json, cases.jsonl, answers.jsonl, assets/
test/fixtures/benchmarks/table-separators/           correct.json and wrong.json
envirobench-plugin.json                              one dataset entry

The type declares ID, FORMAT, LABEL and SCORING_VERSION, optionally PROFILES, HEADLINE, CATEGORIES and a viewer, and implements score_case. Core supplies pairing, the gate, rescoring, review and the report. bin/test runs a shared conformance suite over every registered type: answer isolation, malformed results, correct and wrong fixtures, pairing and gates, re-keying, review drafts and the report viewer.

6.2Writing an adapter

An adapter is any executable that reads the request and writes adapter-result.json to its output directory, naming the adapter, its version, the spend and a predictions file with exactly the requested case IDs:

{"id": "table-separators", "status": "completed",
 "format": "envirobench.table-separators.v1", "data": "predictions.json",
 "systems": [{"role": "candidate", "model": "provider/model", "cost_usd": 0.42,
              "execution_failures": 0, "unpriced_attempts": 0}]}

Unresolved execution failures, unpriced model calls and missing, duplicate or malformed results block scoring. An adapter may save logs and traces; EnviroBench does not score them. examples/reference-adapter implements the protocol for every public synthetic dataset.

7Status and limitations

• Adapters are trusted local programs. Third-party adapters cannot yet be run with limited file, network and process access.
• There are no repeated trials, confidence intervals or LLM judges. Same-code control runs are the current check for noise.
• The included datasets are small. The largest, 40 invented documents for document extraction, is generated with its answers, so its tables follow a limited set of layouts. The public development set and sealed holdout do not exist yet.
• EnviroBench is in development and its test sets are being finalized. No scores are published yet; they will be at release, first on the public synthetic test sets, then on the planned development set and sealed holdout.
• The repository, package and dataset licenses have not been chosen.

8Versions

9Citation

Until a paper is available, cite the software:

@software{envirobench,
  title   = {EnviroBench: a benchmark for AI systems that read environmental documents},
  author  = {{Statvis}},
  year    = {2026},
  version = {0.2.0},
  url     = {https://envirobench.com},
  note    = {Pre-release}
}

Questions: hello@statvis.com. For an overview of the tasks in environmental terms, see the home page.