Skip to content

chore(e2e): make run-e2e start fresh and accept --, update agent rules - #2261

Merged
chhoumann merged 5 commits into
masterfrom
dx-agent-traps
Oct 24, 2026
Merged

chhoumann merged 5 commits into
masterfrom
dx-agent-traps

Conversation

@chhoumann

@chhoumann chhoumann commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Summary

Agent telemetry showed recurring E2E traps that AGENTS.md was about to grow rules for. This removes two of them in .agents/run-e2e instead, and updates AGENTS.md with only the rules tooling cannot enforce plus stale-fact corrections.

Changes

  • run-e2e drops -- from its arguments. Before, run-e2e -- <spec> ran the whole suite (Vitest ignores everything after --) and then crashed reading the JSON reporter output it never wrote.
  • run-e2e stops any running instance and deletes this checkout's vault before starting, so window size, dead modal keymap scopes, data.json and files left from manual testing no longer leak into a suite run. Only the vault path the runner reports is deleted; other vaults under .obsidian-e2e-vaults are kept. The Templater step no longer unloads a reused Templater, since the instance is never reused.
  • Comments in obsidian-e2e.config.mjs and tests/e2e/e2eVault.ts said root styles.css is hand-written; the build generates it from src/styles.css.
  • AGENTS.md: no shared dev vault, main.js/styles.css are generated and gitignored, release PRs may be merged when the maintainer allows it, plus the run-e2e, modal-closing and clickWhenStill rules. CONTRIBUTING.md no longer asks to commit generated files.

The "run pnpm install after pulling" rule was dropped: pnpm 11 defaults verifyDepsBeforeRun to install, so pnpm run already reinstalls a stale node_modules.

Testing / validation

  • Reproduced each trap on the isolated instance (Obsidian 1.13.7, Linux), then confirmed it gone: run-e2e -- tests/e2e/field-label.test.ts ran 103 files before, 1 after; a 390x760 window plus a leftover note failed choice-builder-pages before, 12/12 passed after with the note gone and a sibling vault intact.
  • tests/orb-setup.test.ts covers both behaviors; the new tests fail against the old script.
  • pnpm run lint, pnpm run test, full .agents/run-e2e (101 files passed) and the Templater specs with OBSIDIAN_E2E_TEMPLATER=1.

Checklist

  • PR title follows Conventional Commits
    — it becomes the squash-merge commit and drives the released version.
  • Linked any related issue(s).
  • Noted release/migration impact, if any. No release impact: dev tooling and docs only.

Note

Low Risk
Changes are limited to the E2E runner script, contributor docs, and mock-based tests; no plugin runtime or release artifacts are modified.

Overview
.agents/run-e2e now resets the E2E environment on every run: it stops any running Obsidian instance, removes only this checkout’s managed vault under .obsidian-e2e-vaults (with path/symlink guards), then starts fresh. Standalone -- tokens are stripped so spec filters reach Vitest instead of running the full suite. The Templater setup no longer disables an already-loaded plugin, since instances are not reused.

Docs (AGENTS.md, CONTRIBUTING.md) reflect isolated per-checkout vaults, gitignored generated main.js/styles.css, updated release-PR guidance, and agent rules for run-e2e, modal teardown, and clickWhenStill. Minor comment fixes in obsidian-e2e.config.mjs and tests/e2e/e2eVault.ts note that styles.css is built from source.

tests/orb-setup.test.ts adds coverage for stop→provision→start ordering, vault wipe vs sibling vaults, refusing deletes through symlinked vault roots, and -- stripping.

Reviewed by Cursor Bugbot for commit 194d93b. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Make run-e2e start fresh, drop --, and update agent rules

  • run-e2e now stops an existing E2E instance, removes the stale provisioned vault, and only deletes it when the vault resolves to a physical directory directly under the repo vault root. This protects against deleting data behind symlinked paths.
  • The script strips standalone -- separators before passing filters to Vitest, so spec filters still apply.
  • Templater setup no longer disables the plugin before enabling it.
  • Updates AGENTS.md and CONTRIBUTING.md for per-checkout isolated vaults, fresh E2E runs, and removing generated main.js and styles.css from PRs. Adds tests in orb-setup.test.ts covering the fresh-instance flow, sibling-vault preservation, symlinked vault roots, and separator dropping.
  • Behavioral Change: run-e2e now deletes the provisioned E2E vault on each run; sibling vaults under the repo vault root are preserved.

Macroscope summarized 194d93b.

Summary by CodeRabbit

  • Testing
    • E2E test runs now stop any previous test instance and start with a clean, isolated vault.
    • Test filters are passed to the test runner without standalone -- arguments.
    • Added checks for setup order and safe cleanup of existing vaults.
  • Documentation
    • Contributor guidance clarifies that generated build files should not be included in pull requests.
    • Updated testing, build artifact, and release guidance, including isolated-vault use and release PR steps.

Vitest ignores everything after --, so run-e2e -- <spec> ran the whole
suite and then lost the JSON reporter flags.
A running instance kept its window size, modals and keymap scopes, and the
vault kept data.json and files from manual testing, so suite runs after
hand-testing failed for reasons unrelated to the change.
Drop the shared dev vault, say main.js and styles.css are generated and
gitignored, describe how a release PR may be merged, and add the run-e2e,
modal and clickWhenStill rules that tooling does not enforce.
@coderabbitai

coderabbitai Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0747a76c-b7c0-4791-9308-fc4ec177e9bb
Commits

Reviewing files that changed from the base of the PR and between c4eb2d7 and 194d93b.

Files selected for processing: 2
  • .agents/run-e2e
  • tests/orb-setup.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

Walkthrough

The E2E runner now resets its instance and vault before startup and removes -- from arguments passed to Vitest. Tests and E2E instructions reflect isolated vaults. Repository guidance also changes for generated artifacts, regression tests, and release PRs.

Changes: Isolated E2E workflow

Layer / File(s) Summary
Runner lifecycle and test coverage
.agents/run-e2e, tests/orb-setup.test.ts
The runner stops an existing E2E instance, removes a stale vault within the repository’s E2E vault directory, and passes Vitest arguments without --. Tests check this ordering, confirm a sibling vault remains, verify external symlink contents are preserved, and test argument filtering. Templater setup no longer disables the loaded plugin.
E2E setup guidance and artifact descriptions
AGENTS.md, obsidian-e2e.config.mjs, tests/e2e/e2eVault.ts
E2E guidance describes isolated vaults per checkout, native-spec execution, modal handling, and timing-safe clicks. Configuration and vault comments describe root plugin files as build outputs.

Repository and release guidance

Layer / File(s) Summary
Artifact, testing, and release instructions
AGENTS.md, CONTRIBUTING.md
Repository instructions describe generated files as gitignored, revise release-bot PR steps, and ask for regression tests that fail when a fix is reverted. The contribution guide no longer asks contributors to include generated files in PRs.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Other

Possibly related PRs

  • chhoumann/quickadd#2258: It changes the clickWhenStill helper and adds the same timing-safe click guidance referenced in this PR’s E2E instructions.
Merge Risk: Minimal · up to 194d9

The E2E vault reset is guarded against the reported symlinked-parent deletion, and no remaining merge-blocking issue is identified.

Pre-merge checks: Passed 5
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title accurately summarizes the main changes: refreshing the E2E runner, accepting standalone -- arguments, and updating agent rules. It is concise and specific.
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 3 files. (1 skipped: 1 …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 3 files. (1 skipped: 1 unsupported.)

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit checks the vault at dawn
Clears old notes before tests run on
A separator hops away
Fresh plugin files greet the day
Safe clicks guide the testing paw
Then carrot crumbs receive applause

Comment @coderabbitai help to get the list of available commands.

@chhoumann
chhoumann marked this pull request as ready for review October 24, 2026 20:09
@chhoumann

Copy link
Copy Markdown
Owner Author

FAIL

Independent verification of c4eb2d7bd804f031dbf0c198fff8d1f16664b697 against master 7cf79640, on Linux with Obsidian 1.13.7. Installed with pnpm install --frozen-lockfile; all Obsidian commands used env -u DISPLAY. No tracked files changed.

Blocking finding, P2. the reset checks only the textual parent path before deletion. A symlinked .obsidian-e2e-vaults lets it delete files outside the checkout before the later canonical-path guard rejects the run. In a disposable snapshot, I linked probe/.obsidian-e2e-vaults to /tmp/qa2261-review/outside and created outside/quickadd-probe/KeepOutside.md. Running .agents/run-e2e tests/e2e/field-label.test.ts with the master runner exited 1 and preserved the sentinel. The head runner deleted it, then exited 1 with "Start environment paths are not canonical isolated E2E paths." Validate the physical vault/root paths before rm -rf, and cover a symlinked parent directory with a regression test.

Requested behavior otherwise verified:

  • .agents/run-e2e -- tests/e2e/field-label.test.ts: master ignored the filter. I observed the live sandboxes for unrelated capture-cursor, template-discovery-actions, prompt-cancellation, and macro-recursion specs, then interrupted the run and stopped its instance. Head ran exactly 1 file / 2 tests, exit 0.
  • After the specified 390×760 window and Leftover.md setup, .agents/run-e2e tests/e2e/choice-builder-pages.test.ts: master failed 1/12 on command-bar overflow and retained the note. Head passed 12/12, removed the note, and preserved sibling keep.md.
  • Head pnpm run lint passed; pnpm run test passed 6,964 tests, with 28 skipped. In a disposable snapshot, pnpm run test tests/orb-setup.test.ts with master's runner failed exactly the two new tests; the other four passed.
  • OBSIDIAN_E2E_TEMPLATER=1 .agents/run-e2e with the three Templater spec paths passed 3 files / 16 tests.

A controlled provision failure emitting valid exports then exiting 23 propagated status 23, preserved the sentinel, skipped startup, and ran cleanup. No additional blocking argument-forwarding or documentation issue found. The pinned release workflow still requires the merger account to be chhoumann.

All started instances are stopped; sibling-check fixtures removed. No push or tracked-file edits.

A linked .obsidian-e2e-vaults made the reset delete the vault behind the
link before the canonical-path check rejected the run.
@chhoumann

Copy link
Copy Markdown
Owner Author

PASS

Independent round-2 verification of 194d93b531641403bb0c4382b4ec39ddc2c5b1c0, reviewing only c4eb2d7b..194d93b5. Tested on Linux with real Obsidian 1.13.7. All Obsidian commands used env -u DISPLAY; run output was saved to files.

  • Symlinked-parent probe in a disposable head snapshot preserved outside/quickadd-probe/KeepOutside.md with its original contents. The runner exited 1 at the canonical-path guard and stopped its instance. The round-1 deletion finding is resolved.
  • Normal layout on this checkout passed. Started Obsidian, created Leftover.md through the requested app.vault.create eval, confirmed the file existed, then ran .agents/run-e2e tests/e2e/field-label.test.ts. Exactly 1 file / 2 tests passed, exit 0. Leftover.md was gone; a sibling vault sentinel survived.
  • pnpm run test tests/orb-setup.test.ts passed all 7 tests. The new linked-directory regression test failed against the pre-fix c4eb2d7b runner in a disposable snapshot because its fixture vault was deleted.
  • pnpm run lint passed.

Both started instances are stopped, with their profiles removed and no matching app processes remaining. The working tree is clean. No tracked files changed and nothing pushed. No additional findings in the fix diff.

@chhoumann
chhoumann merged commit b3220f7 into master Oct 24, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant