Repository navigation
ci(bench): cover every command's startup path - #70
Conversation
The benchmark only timed --version, status and execute --local. Commands that reach the MCP session (whoami, search, git-credential) and kody dev load several times more code than --version, and nothing caught it. Each scenario now states its expected exit code, the login-gated commands run without any config so they stop at the "not logged in" error after loading everything the real call needs, and budgets cover the new scenarios. Co-Authored-By: bot_apk <apk@cognition.ai>
GitHub Actions ubuntu-latest Node 24.21 measured whoami/search at ~169ms over baseline, git-credential at ~159ms, and dev at ~228ms — Devin's 160/220 budgets (laptop-calibrated) fail the same way the original execute budgets did in #32. Raise the new rows to 240/300 (~75ms headroom over observed CI p50); leave --version/status/help/execute unchanged. Co-Authored-By: bot_apk <apk@cognition.ai> Co-authored-by: Kent C. Dodds <me+github@kentcdodds.com>
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ee2ca93. Configure here.
| whoami: { args: ['whoami'], env: noConfig, exit: 1 }, | ||
| search: { args: ['search', 'kody'], env: noConfig, exit: 1 }, | ||
| 'git-credential get': { args: ['git-credential', 'get'], env: noConfig }, | ||
| 'dev (no app entry)': { args: ['dev'], env: noConfig, exit: 1 }, |
There was a problem hiding this comment.
Bench misses OS credential stores
Medium Severity
noConfig only overrides XDG_CONFIG_HOME and KODY_API_TOKEN, so login-gated scenarios still load OS keychain sessions and macOS/Windows file stores. whoami and search then hit the real network or exit 0, and status fails the new exit: 1 check for anyone already logged in.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit ee2ca93. Configure here.
|
🎉 This PR is included in version 1.26.3 🎉 The release is available on: Your semantic-release bot 📦🚀 |
2 participants
The bench from #32 timed --version, status and the three execute --local cases. It now also times help, whoami, search, git-credential get and dev, each with its expected exit code declared on the scenario (replacing the status-only special case). The login-gated commands run with no config and an empty
KODY_API_TOKENso they stop at the "not logged in" error after loading everything the real call needs, which is exactly the import cost worth guarding; nothing touches the network. Budgets (ms over node baseline) cover the new rows. No source or behavior changes.Budget note (same pattern as #32): Devin's laptop-calibrated budgets (whoami/search/git-credential 160, dev 220) failed on GHA ubuntu-latest Node 24.21 — whoami/search ~169 over baseline, git-credential ~159, dev ~228. Raised the new rows to 240 / 300 (~75ms headroom over observed CI p50);
--version/status/help/ execute budgets unchanged.Baseline on main 1.26.2 (Node 24.21, 20 runs, p50 / p95 ms, loopback API; cold = fresh npm cache, 3 runs):
Authored by Devin (performance audit v2); applied via
git amwith authorship preserved. Budget bump added after GHA evidence.Note
Low Risk
Benchmark and CI budget config only; no CLI runtime or product behavior changes.
Reviewed by Cursor Bugbot for commit ee2ca93. Bugbot is set up for automated code reviews on this repo. Configure here.