Skip to content

Commit 8a13d76

Browse files
author
FlyBot Dev-3
committed
fix(live-e2e): one reading of DEEPSEEK_API_KEY for the preflight, the bench and the guard (#2214)
The CI preflight published key=true for any non-empty string, while the bench starter and the run guard refused the mock's key and anything under 16 characters. So a short or mock key skipped nothing: the bench started and failed the night for exactly the reason the #794 skip exists to avoid. One module (dev/live-e2e/key.mjs) now returns the verdict and the words for it, and all three readers use it. A key that is present but unusable is skipped with its own warning and its own summary, so the skip never reads as a pass and the reason is visible without reading a red log. A key long enough but not real is still not detected here; only the endpoint can judge that.
1 parent 77a6b4d commit 8a13d76

9 files changed

Lines changed: 115 additions & 18 deletions

File tree

‎.github/scripts/live-e2e-preflight.mjs‎

Lines changed: 25 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,19 @@
11
#!/usr/bin/env node
22
// The live E2E preflight (#794). The suite cannot run without DEEPSEEK_API_KEY, and a run that is
3-
// red every night for a reason nobody can act on teaches the team to ignore red. So a missing key
4-
// is a warning, not a failure: this step publishes `key=false`, the workflow skips every step that
5-
// needs the key, and the summary says the run measured nothing. A failure inside the suite still
6-
// fails the run, so the skip is never a pass.
3+
// red every night for a reason nobody can act on teaches the team to ignore red. So a key the suite
4+
// cannot use is a warning, not a failure: this step publishes `key=false`, the workflow skips every
5+
// step that needs the key, and the summary says the run measured nothing. A failure inside the
6+
// suite still fails the run, so the skip is never a pass.
7+
//
8+
// "Cannot use" is the suite's own reading (`dev/live-e2e/key.mjs`), not this step's: the bench and
9+
// the run guard refuse a key under 16 characters or the mock's, and a preflight that accepted them
10+
// let those keys through to a red bench instead of the skip.
711
//
812
// `.github/workflows/live-e2e.yml` runs it as `node .github/scripts/live-e2e-preflight.mjs`, with
913
// the secret mapped into DEEPSEEK_API_KEY step-scoped and nothing else in the environment.
1014
import { appendFileSync } from 'node:fs'
1115
import { fileURLToPath } from 'node:url'
16+
import { keyReading } from '../../dev/live-e2e/key.mjs'
1217

1318
export const WARNING = '::warning title=Live E2E skipped::DEEPSEEK_API_KEY is not configured as a repository secret; the four real-model checks and the evidence bundle are skipped, so this run measured nothing'
1419

@@ -20,15 +25,26 @@ export const SUMMARY = [
2025
'**This run is green because there was nothing to run, not because the suite passed.** Add the secret in Settings → Secrets and variables → Actions to measure the suite again (#794).',
2126
]
2227

28+
export const UNUSABLE_WARNING = '::warning title=Live E2E skipped::DEEPSEEK_API_KEY is set but is not a usable key (the mock\'s key, or shorter than 16 characters); the four real-model checks and the evidence bundle are skipped, so this run measured nothing'
29+
30+
export const UNUSABLE_SUMMARY = [
31+
'### Live E2E skipped: the repository secret is not a usable key',
32+
'',
33+
'`DEEPSEEK_API_KEY` is set, but the suite refuses it: the mock\'s key or fewer than 16 characters never reaches the real endpoint, so no step of the suite ran and no evidence bundle was produced.',
34+
'',
35+
'**This run is green because there was nothing to run, not because the suite passed.** Replace the secret in Settings → Secrets and variables → Actions with a real key to measure the suite again.',
36+
]
37+
2338
// A secret that is not configured reaches the step as an empty string, not as a missing variable,
2439
// so both have to mean the same thing here.
2540
export function plan(key) {
26-
const present = typeof key === 'string' && key.length > 0
41+
const { usable, reason } = keyReading(key)
42+
const missing = reason === 'missing'
2743
return {
28-
present,
29-
output: `key=${present}`,
30-
warning: present ? null : WARNING,
31-
summary: present ? null : SUMMARY.join('\n'),
44+
present: usable,
45+
output: `key=${usable}`,
46+
warning: usable ? null : (missing ? WARNING : UNUSABLE_WARNING),
47+
summary: usable ? null : (missing ? SUMMARY : UNUSABLE_SUMMARY).join('\n'),
3248
}
3349
}
3450

‎.github/scripts/live-e2e-preflight.test.mjs‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ import { join } from 'node:path'
1111
import { SUMMARY, WARNING, plan } from './live-e2e-preflight.mjs'
1212

1313
test('#794: a configured key runs the suite and says nothing', () => {
14-
const step = plan('sk-live-e2e')
14+
const step = plan('sk-live-e2e-0123456789abcdef')
1515
assert.equal(step.output, 'key=true')
1616
assert.equal(step.warning, null)
1717
assert.equal(step.summary, null)
@@ -52,10 +52,23 @@ test('#794: with a key the step writes key=true and adds no warning or summary',
5252
writeFileSync(summary, '')
5353
const result = spawnSync(process.execPath, [join(import.meta.dirname, 'live-e2e-preflight.mjs')], {
5454
encoding: 'utf8',
55-
env: { PATH: process.env.PATH, DEEPSEEK_API_KEY: 'sk-live-e2e', GITHUB_OUTPUT: output, GITHUB_STEP_SUMMARY: summary },
55+
env: { PATH: process.env.PATH, DEEPSEEK_API_KEY: 'sk-live-e2e-0123456789abcdef', GITHUB_OUTPUT: output, GITHUB_STEP_SUMMARY: summary },
5656
})
5757
assert.equal(result.status, 0)
5858
assert.equal(result.stdout, '')
5959
assert.equal(readFileSync(output, 'utf8').trim(), 'key=true')
6060
assert.equal(readFileSync(summary, 'utf8'), '')
6161
})
62+
63+
test('a key that is present but unusable skips like a missing one, and says why', () => {
64+
assert.equal(plan(' ').warning, WARNING, 'a blank secret is the same thing as no secret')
65+
for (const key of ['mock-key', 'short', 'a'.repeat(15)]) {
66+
const step = plan(key)
67+
assert.equal(step.output, 'key=false', `the preflight lets ${JSON.stringify(key)} through`)
68+
assert.match(step.warning, /^::warning /)
69+
assert.match(step.warning, /not a usable key/)
70+
assert.notEqual(step.warning, WARNING, 'a key someone set is a different warning from no key at all')
71+
assert.match(step.summary, /not a usable key/)
72+
assert.match(step.summary, /nothing to run, not because the suite passed/)
73+
}
74+
})
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
A `Live E2E` run no longer publishes a key the suite itself refuses: the preflight that decides whether any step runs, the bench starter and the run guard now read `DEEPSEEK_API_KEY` through one module, so the mock's key or one under 16 characters is skipped with its own warning — saying plainly that nothing ran — instead of reaching a bench that refuses it and failing the night for a reason the skip exists to avoid. A key long enough but not real is not detected here; only the endpoint can judge that ([#2214](https://github.com/ThinkFlowLab/FlyBot/issues/2214)).

‎dev/live-e2e/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ a real `DEEPSEEK_API_KEY` in the environment. Chrome comes from `CHROME_BIN`, el
5757
`google-chrome`/`chromium`(-`browser`) on PATH, else playwright's own download.
5858

5959
```sh
60-
export DEEPSEEK_API_KEY=... # real key; the runner fails closed without it
60+
export DEEPSEEK_API_KEY=... # real key (16+ chars, never the mock's); fails closed
6161
export LIVE_E2E_HOME="$HOME/.cache/flybot-live-e2e/live-e2e" # outside the checkout
6262
node dev/live-e2e/bench.mjs live-e2e 3196 reset
6363
node dev/live-e2e/run.mjs "$LIVE_E2E_HOME" 3196 --model deepseek-official/deepseek-flash

‎dev/live-e2e/bench.mjs‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,16 @@
88
// Data lives outside this checkout — `$LIVE_E2E_HOME`, else ~/.cache/flybot-live-e2e/<name> (never
99
// under ~/.dsh; `reset` wipes the Bot team and Sessions before starting). A home inside the
1010
// checkout is refused: that workspace would hold the grader (#398). Environment: DEEPSEEK_API_KEY
11-
// (required; fails loudly when missing or the mock's), DEEPSEEK_BASE_URL (optional; defaults to the
12-
// official endpoint — point it at dev/llm-tap.mjs to record per-call usage), DSH_HARNESS (a
13-
// deepseek-harness checkout with its dependencies installed).
11+
// (required; fails loudly when the suite's own reading refuses it — see key.mjs),
12+
// DEEPSEEK_BASE_URL (optional; defaults to the official endpoint — point it at
13+
// dev/llm-tap.mjs to record per-call usage), DSH_HARNESS (a deepseek-harness checkout with
14+
// its dependencies installed).
1415
import { spawn, spawnSync } from 'node:child_process'
1516
import { existsSync, mkdirSync, openSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
1617
import { dirname, join } from 'node:path'
1718
import { fileURLToPath } from 'node:url'
1819
import { benchHome } from './integrity.mjs'
20+
import { keyProblem, keyReading } from './key.mjs'
1921

2022
const ROOT = dirname(dirname(dirname(fileURLToPath(import.meta.url))))
2123
const [NAME, PORT_TEXT, ACTION = 'start'] = process.argv.slice(2)
@@ -29,7 +31,8 @@ const { home: HOME_DIR, problem: homeProblem } = benchHome({ name: NAME, root: R
2931
if (homeProblem) die(`${homeProblem}\n(the bench home must be outside ${ROOT}; export LIVE_E2E_HOME to choose it)`)
3032

3133
const apiKey = process.env.DEEPSEEK_API_KEY ?? ''
32-
if (!apiKey || apiKey === 'mock-key' || apiKey.length < 16) die('DEEPSEEK_API_KEY is missing, too short, or the mock\'s; a real key is required (never printed)')
34+
const key = keyReading(apiKey)
35+
if (!key.usable) die(`DEEPSEEK_API_KEY ${keyProblem(key.reason)}; a real key is required (never printed)`)
3336

3437
const harness = process.env.DSH_HARNESS ?? join(ROOT, 'deepseek-harness')
3538
if (!existsSync(join(harness, 'package.json'))) die(`no deepseek-harness checkout at ${harness}; set DSH_HARNESS (see CONTRIBUTING.md)`)

‎dev/live-e2e/key.mjs‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
// One reading of DEEPSEEK_API_KEY for the three places that judge it: the CI preflight that decides
2+
// whether any step runs (#794), the bench starter, and the run guard. They disagreed — the preflight
3+
// took any non-empty string, the other two refused a key under 16 characters or the mock's — so a
4+
// short or mock key published `key=true`, started the bench and failed the night for exactly the
5+
// reason the skip exists to avoid. A key present but unusable is skipped like a missing one, with
6+
// its own warning, so the skip never reads as a pass and nobody has to read a red log to find it.
7+
export const MIN_LENGTH = 16
8+
export const MOCK_KEY = 'mock-key'
9+
10+
export function keyReading(value) {
11+
const key = typeof value === 'string' ? value.trim() : ''
12+
if (key === '') return { usable: false, reason: 'missing' }
13+
if (key === MOCK_KEY) return { usable: false, reason: 'mock' }
14+
if (key.length < MIN_LENGTH) return { usable: false, reason: 'short' }
15+
return { usable: true, reason: null }
16+
}
17+
18+
// The verdict in words, shared so the three readers cannot drift apart in what they say.
19+
export function keyProblem(reason) {
20+
if (reason === 'missing') return 'is missing'
21+
if (reason === 'mock') return "is the mock's key"
22+
return `is shorter than ${MIN_LENGTH} characters`
23+
}

‎dev/live-e2e/report.mjs‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { join } from "node:path"
22
import { canonicalPath, isInside } from "./integrity.mjs"
3+
import { keyProblem, keyReading } from "./key.mjs"
34

45
// The outcome contract and the report assembly for the #204 live runner. Pure code,
56
// covered keyless by test/report.test.mjs.
@@ -30,10 +31,12 @@ export function budgetsFrom(env) {
3031

3132
// Fail-closed checks on the run's inputs. A loopback base URL is the mock bench (or the
3233
// usage tap, which names the real upstream in LIVE_E2E_TAP_UPSTREAM); anything else
33-
// without a key, or with the mock's key, is refused rather than run.
34+
// without a usable key (dev/live-e2e/key.mjs, the reading the preflight and the bench start share)
35+
// is refused rather than run.
3436
export function guardRun({ apiKey, baseUrl, tapUpstream, home, userHome, repoRoot, evidence, cwd }) {
3537
const problems = []
36-
if (!apiKey || apiKey === 'mock-key' || apiKey.length < 16) problems.push('DEEPSEEK_API_KEY is missing, too short, or the mock key')
38+
const key = keyReading(apiKey)
39+
if (!key.usable) problems.push(`DEEPSEEK_API_KEY ${keyProblem(key.reason)}`)
3740
let host = 'api.deepseek.com'
3841
let origin = 'default official endpoint'
3942
if (baseUrl) {
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
// The suite judges DEEPSEEK_API_KEY in three places: the CI preflight that decides whether any
2+
// step runs, the bench starter, and the run guard that fails the run closed. Two of them refused
3+
// a key the third one accepted, so a short or mock key published `key=true`, ran the bench and
4+
// failed the night for a reason the skip exists to avoid (#794). One reading, three readers.
5+
import assert from 'node:assert/strict'
6+
import { readFileSync } from 'node:fs'
7+
import { test } from 'node:test'
8+
import { plan } from '../../../.github/scripts/live-e2e-preflight.mjs'
9+
import { guardRun } from '../report.mjs'
10+
11+
const CASES = [
12+
['', 'a missing key'],
13+
[undefined, 'an unset key'],
14+
[' ', 'a blank key'],
15+
['mock-key', "the mock's key"],
16+
['short', 'a short key'],
17+
['a'.repeat(15), 'a key one character under the floor'],
18+
['a'.repeat(16), 'a key exactly at the floor'],
19+
['sk-' + 'a'.repeat(32), 'a real-looking key'],
20+
]
21+
22+
const keyProblems = (apiKey) => guardRun({ apiKey, baseUrl: undefined }).problems.filter((problem) => problem.startsWith('DEEPSEEK_API_KEY'))
23+
24+
test('the preflight and the run guard accept exactly the same keys', () => {
25+
for (const [key, what] of CASES) {
26+
assert.equal(plan(key).present, keyProblems(key).length === 0, `the preflight and the run guard disagree about ${what}`)
27+
}
28+
})
29+
30+
test('the bench starter reads the key through the same module, with no threshold of its own', () => {
31+
const bench = readFileSync(new URL('../bench.mjs', import.meta.url), 'utf8')
32+
assert.doesNotMatch(bench, /\blength\s*</, 'the bench keeps its own length threshold')
33+
assert.match(bench, /keyReading|keyState|keyProblem/, 'the bench does not read the shared verdict')
34+
})

‎docs/subsystems/live-e2e.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@ suite is one of its columns, not the map's owner.
3939
- When the key is absent altogether, `live-e2e.yml` skips the six steps that need it and
4040
warns instead of failing every night (#794). A skipped run proves nothing about the
4141
suite, and the run summary says exactly that; a failure inside the suite still fails.
42+
- One reading of the key is shared by the three places that judge it — the preflight, the
43+
bench start and the run guard (`dev/live-e2e/key.mjs`): a key under 16 characters or the
44+
mock's is skipped with its own warning, never passed to a bench that refuses it. A key
45+
long enough but not real is not detected here; only the endpoint can say that (#2214).
4246
- The mock suites stay separate: this bench never starts the mock model, and the mock
4347
benches never gain a real key.
4448
- Evidence is an explicit allow-list that passed the sanitizer (no `token=` URLs, no

0 commit comments

Comments
 (0)