Skip to content
This repository was archived by the owner on Oct 16, 2026. It is now read-only.

ci(bench): cover every command's startup path #70

Merged
kody-bot merged 2 commits into from
Oct 16, 2026
+15 −2
Merged

ci(bench): cover every command's startup path #70

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3cb48fa
ci(bench): cover every command's startup path
devin-ai-integration[bot] Oct 15, 2026
ee2ca93
ci(bench): raise new scenario budgets for GHA runners
devin-ai-integration[bot] Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions scripts/bench-budgets.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
{
"--version": 60,
"status": 60,
"help": 60,
"whoami": 240,
"search": 240,
"git-credential get": 240,
"dev (no app entry)": 300,
"execute --local empty": 280,
"execute --local kody:@ import": 280,
"execute --local runtime call": 280
Expand Down
12 changes: 10 additions & 2 deletions scripts/bench.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,10 +58,18 @@ await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve))
const apiUrl = `http://127.0.0.1:${server.address().port}`

const executeFlags = ['execute', '--local', '--api-url', apiUrl, '--token', 'kody_at_bench']
// Commands that need a login stop at the "not logged in" error, after loading
// everything they would need for the real call.
const noConfig = { XDG_CONFIG_HOME: '/nonexistent', KODY_API_TOKEN: '' }
const scenarios = {
'node baseline': { command: process.execPath, args: ['-e', ''] },
'--version': { args: ['--version'] },
status: { args: ['status'], env: { XDG_CONFIG_HOME: '/nonexistent' } },
status: { args: ['status'], env: noConfig, exit: 1 },
help: { args: ['help'] },
whoami: { args: ['whoami'], env: noConfig, exit: 1 },
search: { args: ['search', 'kody'], env: noConfig, exit: 1 },
'git-credential get': { args: ['git-credential', 'get'], env: noConfig },
'dev (no app entry)': { args: ['dev'], env: noConfig, exit: 1 },

@cursor cursor Bot Oct 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bench misses OS credential stores

Medium Severity

noConfig only overrides XDG_CONFIG_HOME and KODY_API_TOKEN, so login-gated scenarios still load OS keychain sessions and macOS/Windows file stores. whoami and search then hit the real network or exit 0, and status fails the new exit: 1 check for anyone already logged in.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit ee2ca93. Configure here.

'execute --local empty': {
args: [...executeFlags, '--code', 'export default async function main() { return null }'],
},
Expand Down Expand Up @@ -110,7 +118,7 @@ for (const [name, scenario] of Object.entries(scenarios)) {
const started = performance.now()
const result = await run(command, args, env)
const elapsed = performance.now() - started
if (name !== 'status' && result.status !== 0) {
if (result.status !== (scenario.exit ?? 0)) {
throw new Error(`${name} exited ${result.status}: ${result.stderr}`)
}
samples.push(elapsed)
Expand Down
Loading