[sirotaoleg]

Stacked on #133, itself on #124: this pull request's own commits are the last five. Until they are merged, the diff shows their files too. It can be merged after both.

What

OSWorld-V2's 108 tasks become a bundled benchmark, osworld2, on the Nullius standard, at the upstream's release osworld-v2.1. Each task is a unit in an Ubuntu VM of its own on the upstream's docker provider, set up by the task's own class and scored from 0 to 1 by its own evaluate(). The tasks and their assets are gated on Hugging Face. Nullius does not fetch them, and the entry is gated (#133): a unit's failure from a model endpoint says its status alone, and the driver names an upstream exception by a known class's name at most, never by its words. The Original setup runs one of OSWorld-V2's own agents, M3 or Muse Spark, chosen by the Agent option, from its published script, with every model call sent to the relay. The Original's models are pinned to their providers on OpenRouter: M3 to MiniMax, Muse Spark to Meta. Claude Sonnet 4.6 is pinned to Anthropic; it is the LLM grader of 21 tasks and the simulated user of 6. Only M3's runs count as comparable with the paper's row. Its Review row and its generated page name the setup "OSWorld-V2 agents". A null control runs one DONE action, then the task's evaluate() on the untouched VM. The entry declares agent.computer/v1. No agent from the Agents page runs it yet. Its texts are one Good to know line and 25 limitations. Settings lists GitLab's root token under Custom keys, for the two GitLab tasks.

The mock websites of 40 tasks are OSWorld-web at the same tag. The guide's steps build them from source and serve them on the machine the units run on. A unit that needs them is refused before its VM starts when they do not answer. For 026 and 041, the driver starts GitLab anew from its snapshot, one task at a time, and Overleaf's 2 tasks get Overleaf started anew. Each of the 8 tasks that ask for a proxy gets a proxy of its own. The options are Steps (100, 300 or 500), Thinking for M3, the website host suffix, the GitLab host and its write share, and disk write limits for each VM. By default a run takes 100 steps with thinking off, as the upstream's M3 script does; the paper ran M3 at 500 steps with thinking on. Muse Spark runs as published at this release, while its published score came from newer upstream code; a limitation says what differs. There are six datasets: All tasks (108, the upstream's test_v2.json), Offline (82), and four by the services their tasks need. Tasks 103 and 104 are not supported: their evaluate() needs FreeCAD on the driver's machine. Review and the run's start ask for Docker and for the scoring programs (ImageMagick, ffmpeg, tesseract) on the driver's machine. As for OSWorld1, the Where step greys out the container card under the Original, whose docker provider needs Docker on the machine. A unit that ends rate-limited is not run again by itself (limits.run_again: false): an episode is long and paid. A guide, docs/benchmarks/osworld2-machine.md, prepares a machine, runs the null control on one task as its last setup step, and says what a run takes in time and money.

Touches

catalogue · templates and static (the mark) · docs and site (docs/benchmarks/, the generated pages, AGENTS.md) · routes (tests only) · packaging (the entry's .yml, .Dockerfile and startup/*.py as package data in pyproject.toml)

Before and after

Before	After
The Benchmarks page before this change	OSWorld2 listed, with its mark
The builder's Benchmark step under Computer use: OSWorld1, beside the selected GAIA	OSWorld1 and OSWorld2
The Agents page: "n/7 benchmarks it runs"	"n/8": OSWorld2 counts, and no agent runs it
Settings, Custom keys: "No custom keys yet."	OSWorld2's GITLAB_PRIVATE_TOKEN, not set

Taken at 1400 px, light theme, on a fresh profile; the last three pairs are the top 1000 px of their page.

Verified

• At 57851a44, on Python 3.12: backend suite (python3 app/tests/parallel.py), 2489 tests in 136 modules, OK; route suite with Node on PATH, 433 tests in 41 modules, OK.
• OSWorld2's 15 test modules, 371 tests, OK; they need the entry, so none passes without it. 10 are skipped here: they run only when the environment names the upstream with its gated task classes, OSWorld-web or GitLab. Three route test modules and three backend test modules changed for the new entry: the benchmark lists and counts, the Where step's container card, Settings' custom keys, and OSWorld2's cases of gated tasks and pinned routes (test_standard_gated.py, its tests in the class Definition of test_standard_relay_pin.py).
• In a headless Chromium, light and dark, 1400 and 390 px: the Benchmarks page, OSWorld2's page, the Agents page, Settings and the builder, no page error or cut text. /ui-sweep --only osworld2 at fc0bdb1f, this pull request's third commit: 6 screenshots, none with a page error or cut text.
• PYTHONPATH=app python3 -m nullius standard check: ok. python3 app/tests/catalogue_docs.py --write, then the check: up to date. python3 app/tests/field_contract.py: ok. app/site/readme_media.py --check: current. The docs check reports the same two findings in AGENTS.md as on feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124's branch.
• Where the runs below ran. The commits named here are of the port's working branches, not in this repository:
  • c3637ffd is this pull request's tree with main's two later commits merged in;
  • f74d1a00 and 58c2b586 add one test commit: two JSON files, no code. It gives OSWorld2's Original one more model, MiniMax M3.1 Flash, the name it is served as, its pin to MiniMax and --max_tokens 32768. That model is a test stand, not in this pull request;
  • 58c2b586 is this entry on the core before feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124;
  • 34e35370 and 8e2e381b are of the port before its move to this repository's history.
• The controls, recorded in standard.json's proven:
  • Offline, no services, on 2026-11-04, on an earlier commit: the 36 ids are the same twice, and the null control scores 0 on 4 tasks;
  • All tasks, on 2026-11-15, on fc0bdb1f, this pull request's third commit; the fourth only writes this record, and the fifth changes three tests. The null control ran on all 108 tasks, one launch with each task a shard of its own, at 16 MB/s and 160 IOPS a VM, in six lanes with at most seven VMs at a time; it took 1 h 32 min. 108 and 093 failed their setup under that load and passed alone at 24 MB/s and 240 IOPS, in 24 more minutes. The grader cost $0.14. nullius standard prove osworld2 --dataset test_v2 --job nop=<merged job> passed: the 108 ids are the same twice, 0 of 108 at full score, a mean of 0.0021 against a bound of at most 0.01. Five tasks score above 0 on the untouched VM, as a limitation says;
  • every task is in line with the same control on all 108 tasks on 34e35370, which had the same mean; 077's LLM grader gave 0.054 both times.
• An agent's score on every kind of task, each above the same task's null control (0.0 in each):

  • Seven kinds, on 58c2b586, by the test stand's model in OSWorld-V2's M3 agent, from the command line:

    • desktop apps, 045: 1.0;
    • mock websites, 052: 1.0;
    • Overleaf, 072: 0.775;
    • GitLab, 026: 0.5, a partial score;
    • a proxy, 075: 1.0 (recorded 0.9999999999999999), with the unit's proxy carrying new outbound connections while the agent acted;
    • the public internet, 027: 1.0;
    • the LLM grader, 003: 1.0, with both grader calls answered 200.

    Each of the seven gives its recorded score again at c3637ffd when its outcome is computed from its saved records, with no VM and no evaluate().

  • The eighth, the simulated user: 007, 0.8, by Muse Spark on 8e2e381b; the simulated user answered 7 times. It was not run on this base.

• Live on this base, against OpenRouter:
  • on c3637ffd, the null control on 087, a graded task: 0.0. The grader's call went to Anthropic by the definition's pin, and standard.served reads held: true;
  • on f74d1a00, the test stand's model on 052: 1.0. All 68 calls went to MiniMax by the test commit's pin and named it, held: true, with no error.
• This pull request's own agents have no positive score on this base. M3 ran one task here, 017, on the core before feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 (90fde73e), and scored 0.0: the agent declared the task failed at its 120th step. Muse Spark ran none. Their scores before the move, on 34e35370 and earlier:
  • M3 on 9 tasks: 070 1.0 twice, 030 1.0, 045 1.0, 013 1.0 then 0.0, 031 0.714, 017 0.429, 010 0.0 three times and once an environment error; 026 and 098 left no score (a full context window, a spend guard);
  • Muse Spark on 3 tasks: 045 1.0, 007 0.8; 052 left no score (a launch error of ours, then a spend guard).
• Every attempt before the move, two test stands' models included, was recorded and triaged: 61 attempts on 33 tasks. 43 were scored (21 full, 11 partial, 11 zero), 10 ended in an environment error and 8 were stopped by a guard or by hand. None was an agent error. Each of their full and partial scores was checked for the false-success patterns the port's audits found: none.
• Not run: a positive score of M3 or Muse Spark on this base; the simulated user on this base; task 041 with an agent; a parity run of the paper's 108 tasks; the 10 skipped tests with the upstream named; the docker tier; the site build; a wheel's check.

Risk

• OSWorld2's driver and OSWorld-V2's scripts run only when someone runs OSWorld2. The Agents page counts eight agent benchmarks, and the generated agent pages gain an OSWorld2 column. No agent there declares agent.computer/v1. Settings shows OSWorld2's GitLab token under Custom keys to every user.
• Scores. A run compares with the paper's row by its model alone: Docker instead of AWS, no US residential proxy, release osworld-v2.1, the grader through OpenRouter, and by default 100 steps with thinking off. A wrong error kind would move a unit between "environment error" (left out of the adjusted rate) and the agent's error (counted).
• The machine.
  • VMs, the mock websites, GitLab and the proxies run on the machine the units run on, outside a run's constraints in Settings; -j bounds them.
  • GitLab's Docker in Docker and the websites' image build run as privileged containers.
  • In each of the driver's processes, startup/sitecustomize.py wraps the creation of the VM's container: its network, its ports on 127.0.0.1, its disk and its write limits. The VMs need a free Docker subnet.
  • The VM's clock is UTC+8: a dated task set up before 16:00 UTC and acted on after can score 0.
• Secrets. GitLab's root token is a licence of the benchmark, handed to the driver only. Model keys stay with the relay. Three tasks' mail profiles lose their saved sign-ins before the VM starts.
• Stacked on feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control #133 and feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124, with no conflict. Once they are merged, this branch is rebased on main.
• Undone by reverting the commits; the catalogue entry goes with them.