Does v1 require resettable test data, or do generated tests own their data? #550

Activity

1. orioltf

   added

   app:unic-archon-dlc

   wayfinder:grillingWayfinder ticket: decide it by grillingWayfinder ticket: decide it by grilling

   on Sep 23, 2026
2. orioltf

   added a parent issue on Sep 23, 2026

   • wayfinder: e2e specs from acceptance criteria, at a second Consumer #538
3. orioltf

   self-assigned this

   on Sep 23, 2026
4. orioltf commented on Sep 23, 2026

   

   orioltf

   on Sep 23, 2026

   Last edited by orioltf

   MemberAuthor

   More actions

   Decisions from the grilling, 2026-09-23

   The ticket stays open. The maintainer has one more question to put before it closes.

   v1 does not wait for resettable data. By default the e2e Practice Pack generates read-only tests. Whether test data exists, and how, is context the Consumer declares.

   Each decision names the layer that owns it: the DLC (Boxes), the Practice Pack, or the Consumer.

   1. Read-only by default (Pack). An e2e test asserts on content already published in the Consumer's e2e environment. It creates and changes nothing. A Consumer that declares a data mechanism widens the scope, and the rule stays the same.
   2. /build writes e2e tests, never data provisioning (DLC). Data provisioning is the code that creates or restores data around a run. Its file shape is the Pack's business: a Playwright Pack calls it setup and teardown projects, and another stack may need none.
   3. A testable criterion whose content is not published becomes a skipped test (Pack default, Consumer may override). Each skip names the criterion ID and the missing content, so a skip is never a silent absence. In v1, whoever publishes that content re-enables the test by hand.
   4. Assert on structure and behaviour, never on content edited outside the repository (Pack). The criterion ID goes in the test title. The Consumer declares which content is edited outside the repository, what is stable on each page, and which environment is its e2e environment.
   5. Test-first for a new feature requires a declared data mechanism (Pack rule, Consumer declaration). To be recorded as a constraint on Does an e2e test become an outer loop around /build's unit TDD loop for a new feature?.
   6. Data provisioning is written by a person, or by its own ticket through the chain (Consumer chooses). An e2e test ticket never writes it as a side effect. The chain route needs two things declared by the Consumer: a write route to the system that holds the data, and an e2e environment that runs are allowed to dirty. Provisioning runs in CI with the suite, so it depends on nothing agent-side.

   For the second Consumer today: no data mechanism is declared, so v1 there is read-only. Its planned test site would be declared as that mechanism when it exists.

   Compared with the internal QA plugin: decisions 2 and 6 match how that plugin splits the work. Its playwright-setup agent builds setup and teardown on explicit request, and its spec generator never writes them. Decision 4 differs. Its reference architecture keeps expected copy per locale in data files, which fits a Consumer whose copy lives in the repository.

   Proposed glossary terms

   For apps/claude-code/unic-archon-dlc/CONTEXT.md, to land with the spec rather than from this map. All three are framework-neutral on purpose.

   • Test specification: what /specs writes, criterion by criterion: what an e2e test must check. Written with the feature, or after the feature is on the trunk. Avoid: spec (it also names a Playwright spec file).
   • E2e test: the code /build writes from a test specification, in whatever framework the Practice Pack names. Avoid: spec file (that is one framework's shape).
   • Data provisioning: the code that creates or restores the data e2e tests depend on, around a run. Owned by the Consumer. Avoid: setup, teardown, seeding (framework shapes or one half of it).
5. orioltf commented on Sep 23, 2026

   

   orioltf

   on Sep 23, 2026

   Last edited by orioltf

   MemberAuthor

   More actions

   Addendum, 2026-09-23: checked against the original chain sketch

   The chain in #508 comes from an architecture sketch dated 2026-08-26. The sketch lists test data as a precondition of the pilot: present, stable and resettable per run, with no data generated by the chain. It says that a candidate that fails a precondition is the first finding.

   Two changes to the decisions above follow from that.

   1. Decision 1 relaxes that precondition on purpose. Under the sketch, Are the second Consumer's acceptance criteria machine-usable, and is its test data resettable? is that first finding: the second Consumer fails the data precondition. This map keeps the second Consumer as the pilot and makes read-only tests the default. A Consumer with resettable data declares its mechanism, and the Pack rule stays the same. The sketch's author is to be told.
   2. The chain route in decision 6 is out of v1. The sketch parks test data generation as follow-up work, and v1 follows it. Decision 6 still says who may write data provisioning later. It asks nothing of v1.

   Two terms stay apart. Test data generation, as the sketch and the internal QA plugin's qa-test-data-generator use it, produces data values. Data provisioning creates or restores data in a system.
6. orioltf

   closed this as completedon Sep 23, 2026
7. orioltf commented on Oct 20, 2026

   

   orioltf

   on Oct 20, 2026

   MemberAuthor

   More actions

   Update, 2026-10-20: who designs the test data

   The second Consumer's test team designs its own test data. That work stays outside the DLC, because each Consumer owns its test data. The DLC creates and runs tests from what the Consumer documents, or shows by example in its own repository, with the commands set in /setup. The maintainer agreed this with the team, and the team accepted it.

   This keeps the decision above: e2e tests are read-only by default, and /build never provisions data. Once the team's data design exists, the Consumer declares it in its context. That widens what a test may cover, and it is the declared data mechanism that test-first for a new feature needs (#541).