feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control by sirotaoleg · Pull Request #133 · ThinkFlowLab/nullius

sirotaoleg · 2026-11-15T21:03:30Z

Stacked on #124: its commits come first, and this pull request's own are the last three. Until #124 is merged, the diff shows its files too, the builder's among them. It can be merged after #124.

What

A benchmark of the standard can now say more about how its units run and end, which a computer-use benchmark like OSWorld2 (#134) needs. The pins, gated, the new limits and prove's new options are opt-in. The relay's rules and the rules of how a unit ends change for every benchmark of the standard; Risk lists them.

Gated tasks (gated: true): a unit's failure that the relay's records explain, a model endpoint's (the agent's, the judge's or the simulated user's), says in its result.json the status and class alone, never the provider's words, which can quote the request. The trial's call records keep the provider's words, and an error a driver raises itself is kept as the driver words it.
A definition's providers (setups.original.pin for the Original's models, score.judge.pin for the official judge; OpenRouter only) join the run's --served-by pins (feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124) and win over --served-by for the same model: a run of the Original and its official judge always go to the providers the paper used. The pin follows the model sent, so a model the agent shares with the judge goes to the judge's providers. A refused route fails the unit with ApiProviderRouteError, an API error outside the agent's rate, not a miss:
- a pinned call answered 404 or 503 when it is that model's last call on its route;
- any pinned call whose reply.served_by shares nothing with the pin's slugs. A name that is not its slug (Google for google-vertex) fails nothing on a guess.
The agent's own last call's error comes first, and a setup error stays a setup error. Three units in a row whose pinned route has no endpoint stop the run, as a refused key does. A user's --served-by on any other model behaves as in feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124.
Limits of a benchmark's own:
- limits.score_seconds: how long score may take;
- limits.call_seconds: how long the relay waits for one model call, also relay serve --timeout;
- limits.run_again: false: a unit that ended rate-limited is not run once more, and its outcome stands;
- limits.turn_seconds, read before, is now documented.
How a unit ends:
- a driver's error that begins EnvironmentSetupError: , VerifierError: , CancelledError: or CauseUnknownError: keeps that kind;
- the agent's own last call refused as longer than the model's context window is ContextWindowExceededError, the agent's;
- a last 429 is read from the agent's own calls in the order they began, so a side model's later 200 does not cover it;
- a unit a guard's stop ended keeps the score its driver gave, its own last call's 429 notwithstanding. score can say the stop's reason (verifier_result.stopped);
- for an upstream's own answerer: OutputLengthExceededError only when its last reply has neither text nor a tool call. When the answerer fails and its last reply was a Responses object answered 200 and failed, that is an API error.
The relay:
- a non-streamed 200 to a model call with no reply in it reaches the caller and the record as a 502, or the 429 or 402 of an error it held (the record keeps answered: 200). feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 leaves counting an error inside a 200 to a pull request of its own; this one takes only a 200 with no reply in it;
- a stream cut off midway, after the caller had its 200, is recorded as a 502, and its booking against the spending cap stays whole;
- a call refused with 400, 413 or 422 records each inline image it sent by its declared type, the type its bytes show and its size, never a byte;
- a stop file beside the relay's records: the relay notes in it an agent's call refused for the model's context window, and that note refuses nothing. Once a benchmark's guard writes a line of its own there, the agent's later calls are refused unsent, while side models and the judge's and simulated user's routes go on;
- a failed Responses object with no usage records a cost of 0, and its booking is kept.
A job's name:
- a run holds its job name until its process ends, and one under a name another run still holds is refused before it writes anything;
- a rerun into the same job moves each rerun trial's earlier records (.work/relay, result.json, verifier/, agent/) aside to .work/earlier.<ns>.
prove and merge:
- nullius standard prove ... --bench-option NAME=VALUE sets the benchmark's own options for the units check and the controls;
- --job nop=MERGED (or oracle=) holds a merged launch of a control to its bound instead of running it, checking every trial's own record;
- merge accepts shards that differ in benchmark options and keeps each shard's in merged_from;
- merge refuses, unless --force, a trial whose record names another job, benchmark, version, control or agent, and a job itself merged despite differences. It lists the shards absent from a merge only when the launch has at most 1,000.
Smaller:
- a trace that cannot be built leaves agent/trace.error.txt;
- an Original's trajectory keeps its tool calls;
- each call's record keeps the request's sha256;
- thinking_tokens count as reasoning;
- a setup's comparable can list the models whose runs compare with the paper, and the builder's message for another model says so;
- a test checks that every file a host's snapshot takes of a bundled entry is package data.

Read first: failure, route_failure and routes_failure, then _hollow and the stop in Relay._forward (app/nullius/standard/relay.py). Next, pins_of, route_pins and the outcome rules in Run (app/nullius/standard/run.py), then the --job check in prove.py and the trial check in merge.py. Then app/tests/test_standard_relay_pin.py (classes Pinned and Wired), test_standard_gated.py, test_standard_context_window.py and test_standard_controls.py. The rest is the definitions' checks, the CLI options, the sidecar's argument and the pages.

Touches

the standard (app/nullius/standard/: relay, run, prove, definitions, merge, job, cli, container, recipe, fidelity) · docs and site (docs/standard.md, docs/cli/run.md, docs/cli/read-jobs.md)

Verified

At 369054a8, on Python 3.12: backend suite (python3 app/tests/parallel.py), 2114 tests in 121 modules, OK; route suite with Node on PATH, 433 tests in 41 modules, OK.
The 13 changed test modules on feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124's source (468406d8): 10 fail, 2 pass, and 1 does not end.
- test_standard_relay_pin: all 15; test_standard_context_window: 19 of 20; test_standard_controls: 17 of 43; test_standard_relay: 8 of 57; test_standard_gated: 4 of 6.
- test_standard_run: 2 of 50; test_standard_trace: 2 of 20; test_standard_parts, test_standard_served_by and test_standard_units: 1 each.
- test_standard_shard does not end within 4 minutes: one of its merges names 2^64 shards, and feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124's merge lists every absent one.
- test_standard_versions and test_dependency_surface pass on both.
Mutations. Each of these changes to the pinned route makes a named test of Wired or Pinned fail:
- the refusal read against every --served-by pin;
- the agent's own route read without the definition's pins;
- the judge's route not read;
- a route refusal not counted toward the three-in-a-row stop;
- a provider name taken from a reply unchecked.
Covered in feat(catalogue): add OSWorld2 with OSWorld-V2's own agents, its services on the machine and a null control #134, not here: that setups.original.pin is checked and reaches the run, and that limits.call_seconds reaches the relay and the sidecar's --timeout. OSWorld2 is the first entry to set them (test_osworld2_muse.py, and the tests it adds to the class Definition of test_standard_relay_pin.py).
feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124's own tests. Three were changed for two deliberate differences: a 200 with no reply is a 502, and a failed Responses object is read as failed. The rest of test_standard_served_by passes as feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 wrote it.
PYTHONPATH=app python3 -m nullius standard check: ok. python3 app/tests/catalogue_docs.py --write, then the check: up to date. python3 app/tests/field_contract.py: ok. app/site/readme_media.py --check: current. The docs check reports the same two findings in AGENTS.md as on feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124's branch.
Against OpenRouter, through OSWorld2 (feat(catalogue): add OSWorld2 with OSWorld-V2's own agents, its services on the machine and a null control #134). c3637ffd is this code with main's two later commits merged in and OSWorld2 on top, a commit of the port's working branch:
- on c3637ffd, the null control on one task with an LLM grader: the grader's call went out with request.routing {"order": ["anthropic"], "allow_fallbacks": false} from the definition's pin, reply.served_by named Anthropic, and standard.served reads held: true;
- on c3637ffd plus a test commit that adds one more model to OSWorld2's Original with its pin to MiniMax, an agent episode: all 68 calls named their provider, held: true, no error.
Not run: a pinned route refused by OpenRouter in a live run (404 and 503 are covered by the tests); a test of a pinned simulated user's route in a run; a benchmark already on main, live, on this branch; the container backend with --timeout; the docker tier; the site build; a wheel's check.

Risk

For every benchmark of the standard:
- a non-streamed 200 with no reply is a 502, and a stream cut off midway is recorded as one;
- the agent's own last call refused for its context window is the agent's error (ContextWindowExceededError), where it was an API error outside its rate;
- an agent's last 429 followed by a side model's 200 makes the unit rate_limited and runs it once more, where its score stood;
- an upstream answerer whose last reply was a failed Responses object is an API error, where it was the agent's;
- a driver's error of a kind it names: a setup error or a stop is left out of the agent's rate, and a verifier error or one of no known cause counts as unknown. A wrong kind would move a unit between them;
- prove runs no control when the units check failed, and fails a control job with a unit that has neither a score nor an error;
- a run under a job name another run still holds is refused, and a rerun into an existing job moves the rerun trials' records aside;
- merge refuses a trial whose record names another job unless --force.
Pins. Only a definition that pins a model is affected; none on main does. For such a model the builder still offers Served by, and the run keeps the definition's providers whatever it says.
Records. Stored records gain fields and keep the old ones.
Stacked on feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 (468406d8), with no conflict. Once feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 is merged, this branch is rebased on main.
Undone by reverting the commits.

On OpenRouter one model is served by several providers, and a run could neither say which may answer nor see which one did. `--served-by [MODEL=]SLUG[,SLUG...]`, repeatable, names them for a model (the run's when MODEL= is left out). Every relay of the run, a container's sidecar included, puts `provider: {order: [...], allow_fallbacks: false}` on each request it sends as that model, on any route and in the three API shapes, in place of the caller's own routing field. A pinned Chat Completions request asks for OpenRouter's routing facts, where its reply names the provider that served it. Without the option a request goes on byte for byte as before. The option is refused before any unit when the run's provider routes no model to several (only openrouter does), when it names a model the run does not call, or when a slug is malformed or given twice. Each call's record keeps the routing sent (and the caller's own when it was replaced), the serving provider the reply names, how the reply ended and an error the reply holds under a 200, which now makes the call a failed one. The job's config records the pin by model; each trial and the job say the answered calls by serving provider and, for a pinned model, whether every one was served by a pinned provider. merge refuses shards whose pins differ, unless --force.

The run page gains the option and its rules: the slugs, what the relay sends, the refusals, the record and how a pin reads as held. The standard page names the record's new fields and the pin among merge's checks, and the providers page says that a run may choose OpenRouter's serving providers.

…e of their own

…ge no outcome without a pin A review found the pin's record could copy credentials, miss calls and change what a run without the option records and decides. Now: - The facts of a reply (who served it, how it ended, an error under a 200) are read by reply_facts, apart from read_reply, which is as before: the text, tool calls and usage a record held are read as they were, and no fact can raise into the delivery of a reply. Every choice is read for its end and its error, by index; a top-level error is read whatever else the reply holds; a completed Responses event's routing facts name who served it; a fact of the wrong type is left out. - An error under a 200 is recorded beside the reply and nothing else follows from it: the call keeps its status and failure reads it as before. - The recorded routing objects and facts are cleaned: a value under a credential's name, the relay's own key and key-shaped strings are removed. What is sent is unchanged. - The routing-facts header goes to OpenRouter alone, as enabled whatever the caller sent. - One check of a pin, shared by the run, its command line and the relay's own command; a control is refused a pin on a model it never calls. - Who served a unit counts its judge's and simulated user's routes, and the job the benchmark's own relay's calls; the tokens and the cost stay the agent's. A trial whose answered calls name no provider keeps its count. A name reads as a slug by its case and spaces alone. - merge compares pins as mappings, and a forced merge of different pins reads held unknown for such a model.

…held cannot tell The pages now say that an error under a 200 is recorded and changes nothing else, that the record's copies are cleaned of credentials, that who served a run counts every route, how a name reads as a slug, and that held names a provider, never its endpoint variant.

… it must refuse a pin

A run started from the builder could not pin its serving providers. On OpenRouter, the Model step now shows Served by under each model field (the model, the user simulator's and the judge's): the providers that may answer that model, tried in their order and no other, typed as slugs or added from the model's providers. Empty, the default, pins nothing. The recipe holds them as served_by, by field. The server checks them as --served-by is checked, refuses them for another provider or for a model no job calls, and compiles them into --served-by for each job that calls that model, by the name the run sends it as. Review shows them in its Model row. A saved recipe keeps them, and choosing another provider drops them. A recipe with no pin compiles to the command it did. The model's providers come from a new route, POST /new-run/endpoints, which asks OpenRouter's endpoint list as /new-run/models asks the model list: local and form-token guarded, the key kept on this machine and sent when set, the answer cached by the page for five minutes, and a fixed message when the list cannot be had, the field then taking a typed slug.

… model's providers

… line

…del it was set for A review of the pin found records and summaries that could read held where another provider answered, credentials that could reach a record, and pins that the builder dropped or moved to another model. Now: - The provider recorded is the one of the reply's end: a stream's latest event that names one, and the last attempt of OpenRouter's routing facts; an earlier, failed attempt is never taken, and a last attempt that names none gives none. - An error keeps its code, type and message alone, of those kinds; the rest of the object is not kept. A recorded routing object's keys are cleaned as its values are. - A merged job's summary is made of each shard's whole one, its benchmark relay's calls and an unknown included. - A recipe's pins are resolved by model, then given to every job that calls that model, a control's judge job included. The compile route returns each field's model as the run sends it, and Served by lists that model's providers. - A control is refused a pin on its simulated user's model, which it never calls. relay serve refuses a pin unless its base is OpenRouter's (or --routing is given) and the model is one it sends; a container's relay is given the pins of the models it sends. - The builder clears a field's pin when its model changes, a preset or a benchmark included; keeps a saved pin as written, for the server to refuse what is wrong in sight; keeps an empty part of a typed list; puts its help in a tooltip; and keeps the keyboard on the model field once the form is drawn again.

…d where held reads unknown A variant may be a quantisation, a speed or a region. Of several choices' errors the record keeps the top-level one, else the first choice's. A provider whose name is not its slug once lowercased reads unknown though it served the call; the example of a name that holds is DeepInfra.

…facts mark as selected OpenRouter's routing facts list the endpoints available for a request and mark the one that served it as selected; they hold no list of attempts. The relay now reads that entry, at the top of a reply and in a completed Responses event's response, and asks for the facts on a pinned Responses request too: a Responses reply names its provider there alone, so a pin on a model an agent calls through Responses could never read as held.

… pin no provider serves gets 404, and why held may read unknown

…ake no name a chunk gave before

…'s change

… asks for the routing facts and where held reads unknown

main gained the OSWorld1 benchmark and the run option --units since this branch started. Seven files met: the command line, the job's record, merge, the run and three pages. Both sides are kept whole, main's first: a run takes --units and --served-by together, config.json records standard.units_file and standard.served_by, and merge compares both (different unit lists are always refused, different pins unless forced).

…s record A run given --units and --served-by from the command line records both in config.json, with and without --from-run.

…n the field guide The field guide names both config fields, the routing fields of a call, the serving counters and held, and says that merge refuses different unit lists always and different pins unless forced.

main gained the settings that say what Nullius may use on each machine, with the run option --constraint, and a fix of the settings page. Seven files met: the command line, the job's record, the recipe, the run, a test module and two pages. Both sides are kept whole, main's first: a run takes --constraint and --served-by together, config.json records standard.constraints and standard.served_by, and a recipe compiles each target's constraints and the pins of the models its jobs call.

Seven cases where the two meet: both records from the command line, with and without --from-run; the order of the refusals; a scope's enforcement with a valid pin; a container relay's caps beside its pins; each target's compiled command; the builder's checks; and shards whose machine limits differ.

…ving pin in the field guide The field guide says that standard.constraints, standard.units_file and standard.served_by may stand together in config.json, and which of them a merged job keeps from its first shard.

…on top of --served-by: gated tasks, a definition's pins as its serving pins, a pinned route's refusal as the unit's, limits of their own, and prove of a merged control A definition's pin (setups.original.pin, score.judge.pin) joins the run's --served-by pins and wins over --served-by for its model; ApiProviderRouteError reads the pin and reply.served_by. A failed Responses object reads reply.end and reply.error.

…eep the score of a unit a guard stopped despite a last 429 limits.run_again false: a unit that ended rate-limited is not run once more; its outcome stands. A unit whose driver scored it after a guard's stop keeps that score, marked stopped, though its last own call got a 429.

… own last 429, not a side call's or a context window's stop The rule reads the agent's own last call, a guard's line in the stop file and a score stopped by other than the context window; a cap's refusal is still the cap's. limits.run_again takes null as unset. The rate limits row names the opt-out.

pierreleca and others added 26 commits November 13, 2026 14:54

docs(standard): state the serving providers' differences in a sentenc…

76305ac

…e of their own

test(standard): run the relay's own command in a bounded process when…

325e014

… it must refuse a pin

docs(standard): say what held reads in a sentence without a contrast

fbcbcdf

docs(builder): describe Served by on the Model step and its list of a…

c0ea2bb

… model's providers

fix(builder): set Served by below its model field's row, clear of its…

c47d9ff

… line

docs(standard): say a Responses request asks for the routing facts, a…

f092fa3

… pin no provider serves gets 404, and why held may read unknown

test(standard): check that routing facts marking no single endpoint t…

fe81ec1

…ake no name a chunk gave before

refactor(builder): drop a comment written twice above the model field…

7b3f8f3

…'s change

docs(standard): say in sentences of their own why a Responses request…

d6f6514

… asks for the routing facts and where held reads unknown

test(standard): keep a unit list and a serving pin together in a run'…

7cde14d

…s record A run given --units and --served-by from the command line records both in config.json, with and without --from-run.

docs(standard): describe a job's unit list and serving pin together i…

a7317da

…n the field guide The field guide names both config fields, the routing fields of a call, the serving counters and held, and says that merge refuses different unit lists always and different pins unless forced.

sirotaoleg mentioned this pull request Nov 15, 2026

feat(catalogue): add OSWorld2 with OSWorld-V2's own agents, its services on the machine and a null control #134

Draft

Navigation Menu

Repository navigation

feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control - #133

feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control#133
sirotaoleg wants to merge 26 commits into
ThinkFlowLab:mainfrom
sirotaoleg:feat/benchmark-pins-limits

sirotaoleg commented Nov 15, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sirotaoleg commented Nov 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Touches

Verified

Risk

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sirotaoleg commented Nov 15, 2026 •

edited

Loading