Skip to content

feat: make matrix tests the default (issue 74) - #4

Open
bacluc-agent wants to merge 2 commits into
mainfrom
issue-74-default-matrix-tests
Open

bacluc-agent wants to merge 2 commits into
mainfrom
issue-74-default-matrix-tests

Conversation

@bacluc-agent

@bacluc-agent bacluc-agent commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Summary

Removes the add_matrix cookiecutter option so matrix (multi-instance) tests are always generated. Renders are byte-identical to the old add_matrix=y output, so existing consumers are unaffected.

Implements bacluc-agent/agent-vshn-todo#74 (closes via issue link below).

PR target

The issue text names https://github.com/bacluc/commodore-component-template/ as the PR target, but that repo is itself a fork of the canonical bacluc-agent/commodore-component-template (verified: gh api repos/bacluc/commodore-component-template --jq '{fork,parent:.parent.full_name}' → {"fork": true, "parent": "bacluc-agent/commodore-component-template"}), and this agent has no push access there (push: false). The canonical repo is bacluc-agent/commodore-component-template, so this PR targets its main.

Changes

  1. cookiecutter.json — deleted the "add_matrix": "y" option.
  2. hooks/post_gen_project.py — removed the add_matrix flag; the renovate postUpgradeTasks command is now unconditionally make gen-golden-all (renovate.json unchanged vs. old add_matrix=y output).
  3. {{ cookiecutter.slug }}/Makefile — removed the inner {%- if cookiecutter.add_matrix == "y" %} conditional; golden-diff-all, gen-golden-all, lint_kubent_all and $(test_instances) now sit directly inside the add_golden == "y" block.
  4. {{ cookiecutter.slug }}/Makefile.vars.mk — test_instances is now guarded only by add_golden == "y".
  5. {{ cookiecutter.slug }}/.github/workflows/test.yaml — test and golden jobs always emit strategy.matrix.instance and run make test -e instance=${{ matrix.instance }} / make golden-diff -e instance=${{ matrix.instance }} (4 conditionals removed).
  6. .github/workflows/test.yaml — removed the add_matrix matrix dimension (16 combos instead of 32) and the ,"add_matrix":"${{matrix.add_matrix}}" key from cruft_extra_content. test-rendered-test-cases untouched.

Post-edit gate: grep -rn add_matrix --exclude-dir=.git → zero hits.

Byte-stability evidence (baseline = de8e8df)

git clone -q /tmp/opencode/cct /tmp/opencode/cct-baseline && git -C /tmp/opencode/cct-baseline checkout -q de8e8df
# for each combo C: uvx --from cruft==2.16.0 cruft create <baseline|candidate> --no-input --extra-context='{"slug":"test-component","name":"test-component",<C>}'
diff -r --exclude .cruft.json bs/base/test-component bs/cand/test-component

Combos: {} ; {"add_golden":"n"} ; {"add_lib":"y","add_pp":"y","add_go_unit":"y","test_cases":"defaults foo bar"} ; {"add_lib":"y","add_pp":"y","add_go_unit":"y","test_cases":"defaults foo bar","add_golden":"n"}

Result: 4/4 combos identical outside .cruft.json (diff silent for every combo).

Test evidence

  • B2 (candidate render, add_golden=y, test_cases="defaults foo"):
    • grep -A6 '^ test:' test-component/.github/workflows/test.yaml → test job has strategy.matrix.instance with defaults, foo
    • grep 'make test -e instance=\${{ matrix.instance }}' ... → 1 hit
    • grep 'make golden-diff -e instance=\${{ matrix.instance }}' ... → 1 hit
    • grep 'test_instances = ' test-component/Makefile.vars.mk → test_instances = tests/defaults.yml tests/foo.yml
  • B3 (add_golden=n edge case): test job still has strategy.matrix.instance + make test -e instance=${{ matrix.instance }}; no test_instances in Makefile.vars.mk; no golden-diff-all target; no golden job.
  • B4: ! grep -rn add_matrix cookiecutter.json hooks/ '{{ cookiecutter.slug }}' .github/ → zero hits.
  • B5 (Docker, default render inside the template repo):
    • make -C test-component lint → exit 0
    • make -C test-component -j1 golden-diff-all -e git_volume= → exit 0
    • make -C test-component test -e instance=defaults → fails with invalid volume specification: '../.git:../.git:ro' — pre-existing, not caused by this PR: the identical failure occurs on the de8e8df baseline render (rendered output is byte-identical per B1; it's an artifact of running the compile from a directory nested inside another git repo, where git rev-parse --git-common-dir returns a relative path). With the same -e git_volume= override CI uses for golden-diff, the command exits 0 (Compiled 1 targets).
    • actionlint → clean on the root test.yaml and on rendered component workflows (default, test_cases="defaults foo", add_golden=n).
  • PR-triggered CI run (workflow invoked by this PR, not a locally started run): Template tests run #39425583492 — event pull_request, head SHA d8320242e9ddc56fee95a5e6c94bc21b5abdc0de, run conclusion success. Job-level: 22/22 jobs conclusion: success — 16 × test-rendered (add_lib, add_pp, add_golden, add_go_unit) + 6 × test-rendered-test-cases. Both job types have continue-on-error: true, so the job-level conclusions are the meaningful evidence.

Step-8 note

projectsyn/component-icap-virusscan already has add_matrix: "y" in its .cruft.json and already uses matrix.instance in its workflow → unchanged by this PR; no second-repo PR needed.

Issue

bacluc-agent/agent-vshn-todo#74: https://github.com/bacluc-agent/agent-vshn-todo/issues/74


Appendix — step-9 deviation, environment, release note (issue 74)

Step 9 deviation: gh workflow run "Template tests" is impossible

Template tests (.github/workflows/test.yaml) has no workflow_dispatch trigger on this branch, on main, or on projectsyn/commodore-component-template@main (only Sync/sync.yaml is dispatchable). Adding workflow_dispatch to this workflow is issue-71 scope and is out of scope here. Attempted exactly once:

$ gh workflow run "Template tests" --ref issue-74-default-matrix-tests -R bacluc-agent/commodore-component-template
could not create workflow dispatch event: HTTP 422: Workflow does not have 'workflow_dispatch' trigger (https://api.github.com/repos/bacluc-agent/commodore-component-template/actions/workflows/385599719/dispatches)
exit=1

exit=1

gh run list -R bacluc-agent/commodore-component-template --branch issue-74-default-matrix-tests after the attempt shows no dispatch-triggered run (only pull_request events).

Substitute evidence: the workflow runs on this PR (pull_request event):

  • Template tests run #39425583492 — event pull_request, head SHA d8320242e9ddc56fee95a5e6c94bc21b5abdc0de, 22/22 jobs success (16 test-rendered + 6 test-rendered-test-cases; both job types are continue-on-error: true, so job-level conclusions are the evidence).
  • Poll command: gh run list -R bacluc-agent/commodore-component-template --branch issue-74-default-matrix-tests

Sync (dry-run) failure is environmental and pre-existing

Sync (dry-run) fails on every PR in this fork: the fork has zero Actions secrets (gh api repos/bacluc-agent/commodore-component-template/actions/secrets --jq .total_count → 0), so webfactory/ssh-agent aborts at step "Setup SSH" ("The ssh-private-key argument is empty…") before any sync code runs.

  • This PR touches no sync input: pr.yaml, sync.yaml, sync_base.yaml, components.yaml, .sync.yml are all unchanged (diff = 6 files + this hook cleanup, all cookiecutter/hooks/template-workflow).
  • Same failure on unrelated branches: #39126126626, #39076081687, #38745795684
  • Upstream runs the same job green: projectsyn run #39386457955

Not fixable inside this PR (would require configuring repository secrets).

Follow-up cleanup commit 2d891efecb79aea061bedd54a3f15980b7aea268

Removes the dead cmd indirection and stale gen-golden(-all) comment in hooks/post_gen_project.py (cosmetic, issue step 2). Hooks are not part of cookiecutter output → rendered output and byte-stability unchanged; verified via a fresh render asserting postUpgradeTasks.commands == ["make gen-golden-all"].

Release note for consumers

Components previously generated with add_matrix=n: after cruft update to this template, run make gen-golden-all once — golden files created under the old single-instance layout may be empty/stale for the non-default instances that the matrix now includes.

Closes bacluc-agent/agent-vshn-todo#74

Tester evidence (head 2d891ef)

Follow-up run update

Run: https://github.com/bacluc-agent/agent-vshn-runner/actions/runs/39494239328 — model: opencode/big-pickle

  • Review-status correction: PR feat: make matrix tests the default (issue 74) #4 has zero GitHub reviews and zero GitHub comments. Earlier "APPROVED" mentions referred to an internal subagent verdict, not a GitHub review.
  • Upstream PR target is feasible and was created: bacluc-agent is the parent of bacluc in the GitHub fork network (gh api repos/bacluc/commodore-component-template/compare/main...bacluc-agent:issue-74-default-matrix-tests → {status: ahead, ahead_by: 2, behind_by: 0}). The upstream PR was opened against the issue's required target: feat: make matrix tests the default (issue 74) BacLuc/commodore-component-template#1
  • Issue step 8 evidence (local verification against projectsyn/component-icap-virusscan, no push): cruft==2.16.0 update against this branch succeeds ("Good work!"), recorded template commit 2d891efecb79aea061bedd54a3f15980b7aea268, add_matrix dropped from the cookiecutter context (False). Rendered-file diff contains only the 4 known pre-existing drift areas (actions/checkout v5→v6 ×4, mikepenz/release-changelog-builder-action v5→v6, jsonnetfmt --in-place moved from JSONNETFMT_ARGS to the recipe) and zero issue-74-related changes. make test -e instance=defaults, make test -e instance=with_squid_proxy, make test -e instance=monitoring and make -j1 golden-diff-all all exit 0.
  • Issue step 9 re-confirmation: gh workflow run "Template tests" --ref issue-74-default-matrix-tests still fails verbatim with could not create workflow dispatch event: HTTP 422: Workflow does not have 'workflow_dispatch' trigger (https://api.github.com/repos/bacluc-agent/commodore-component-template/actions/workflows/385599719/dispatches) / exit=1; no dispatch run was created (branch run list shows only pull_request-event runs). Substitute evidence: Template tests runs https://github.com/bacluc-agent/commodore-component-template/actions/runs/39425583492 (sha d832024) and https://github.com/bacluc-agent/commodore-component-template/actions/runs/39470464570 (sha 2d891ef), both 22/22 jobs success (example job: https://github.com/bacluc-agent/commodore-component-template/actions/runs/39470464570/job/118792853557).

Verification (re-audit run)

  • Run URL: https://github.com/bacluc-agent/agent-vshn-runner/actions/runs/40397150538
  • Branch/head SHA: issue-74-default-matrix-tests @ 2d891efecb79aea061bedd54a3f15980b7aea268
  • Base: main @ de8e8dfd381493f69cfbf3b773efa45e2617895d (ahead 2, behind 0 → no merge needed)
  • Static audit: all 7 acceptance items PASS; git grep -rn add_matrix → zero (verified in fresh clone /tmp/commodore-component-template-audit at HEAD 2d891efecb79aea061bedd54a3f15980b7aea268, working tree clean)
  • Fresh-render evidence:
    • Render A test_cases: "defaults extra" → test+golden matrices include defaults/extra, make test defaults exit 0, make test instance=extra exit 0, make golden-diff-all exit 0
    • Render B test_cases: "alpha beta gamma", add_golden=n → test matrix alpha/beta/gamma, golden job/targets absent, each instance make test exit 0, make golden-diff-all exit 2 "No rule to make target" (EXPECTED)
  • projectsyn/component-icap-virusscan @ 5b575977 via cruft update to branch: records 2d891ef, diff only pre-existing drift (action v5→v6, jsonnetfmt flag move, .cruft.json metadata/add_matrix key removal), zero issue-74 drift; make test for defaults, with_squid_proxy, monitoring all exit 0; make golden-diff-all exit 0
  • CI: gh workflow run "Template tests" --ref issue-74-default-matrix-tests → HTTP 422 "Workflow does not have 'workflow_dispatch' trigger" (documented deviation; no fresh trigger possible without a no-op commit or closing/reopening the PR, both disallowed); existing run 39470464570 for exact head, success, 24/24 jobs, sample job 118792854379#step:8:1
  • Explicit statement: "No code changes were needed; branch head unchanged at 2d891ef."
  • PR feat: make matrix tests the default (issue 74) #4 body remains otherwise intact; previous sections (Summary, PR target, Changes, Byte-stability evidence, Test evidence, Step-8 note, Issue, Appendix, Tester evidence, Follow-up run update) are still present and this new section is appended.

agent added 2 commits October 20, 2026 12:44
Matrix (multi-instance) tests are now generated unconditionally.
bacluc-agent/agent-vshn-todo#74
@bacluc-agent bacluc-agent changed the title Remove add_matrix option and always generate matrix tests feat: make matrix tests the default (issue 74) Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant