Skip to content

Commit ff9c5db

Browse files
authored
Harden the memory search ranking guard so a broken order fails (#2849)
Fixes #2846 Owner: Dev-1 Reviewer: Birfy Written-by: Dev-1 Bot: Dev-1 Claim-Id: 1afedce05a604ff9a3f7ea8edf0d79fc Risk tier: T1 Base: 429668e ## The guard `core/test/memory-search-edge.test.js` is the edge-case file for `searchIndex`, and its determinism test called the same expression on both sides of `deepEqual`: ```js assert.deepEqual(searchIndex(index, 'flight bookings').map(hit => hit.doc.key), searchIndex(index, 'flight bookings').map(hit => hit.doc.key)) ``` Two calls on the same index with the same query, compared with each other — no input, no ordering rule, nothing that can disagree. Every other assertion in `core/test/**` (253 files, 13,829 assertions scanned) either pins a literal or constrains a value; this was the only literal self-comparison left. The ranking it names is user-visible: `recall` answers with `searchIndex(buildIndex(docs), query, { k: RECALL_MAX + 1 })` (`core/src/host/memory.js:1130`), and the index is rebuilt from the memory files on every read, so ordering is what the user sees when the list is long. ## What the tip asserts The test now pins the three things that "deterministic ranking" has to mean, through one helper that takes the entries and a query: - Two entries that share every term score the same, so the key breaks the tie: `['a', 'b']`. - Re-reading the files in another order does not reorder the result: the reversed array still ranks `['a', 'b']`. - An entry matching both query words ranks above one that matches a single word: `['both', 'only-window']`. No source file changes. 2 files, +11/−6 each (the test and its generated mirror). ## Red first On `429668e2` with only this test changed, I broke the behaviour under test in `core/src/host/memory-search.js` (twice, each time restored with `git checkout --`), and ran the file twice per mutation; the old assertion and the new one saw the same source. Mutation A — drop the `|| String(a.doc.key).localeCompare(String(b.doc.key))` tie-break at `memory-search.js:101`: ``` old assertion, run 1: # tests 5 / # pass 5 / # fail 0 old assertion, run 2: # tests 5 / # pass 5 / # fail 0 new assertion, run 1: # tests 5 / # pass 4 / # fail 1 new assertion, run 2: # tests 5 / # pass 4 / # fail 1 not ok 2 - a word stem matches across endings, and the ranking is deterministic at line 25: actual ['b','a'] expected ['a','b'] ``` Mutation B — sort worst-first while keeping the key tie-break (`a.score - b.score || String(a.doc.key).localeCompare(...)`): ``` old assertion, run 1: # tests 5 / # pass 5 / # fail 0 old assertion, run 2: # tests 5 / # pass 5 / # fail 0 new assertion, run 1: # tests 5 / # pass 4 / # fail 1 new assertion, run 2: # tests 5 / # pass 4 / # fail 1 not ok 2 - a word stem matches across endings, and the ranking is deterministic at line 29: actual ['only-window','both'] expected ['both','only-window'] ``` Source restored, the new test is `5/5` twice, and the three neighbouring suites (`memory-search`, `memory-search-edge`, `memory`) are `116/116`. ## What I ran at this tip - `core/test/memory-search-edge.test.js`: `5/5` green. - Neighbourhood (`memory-search`, `memory-search-edge`, `memory`): `116/116`. - `git diff --name-only 429668e..HEAD`: ``` core/test/memory-search-edge.test.js patches/ds-bot/files/test/memory-search-edge.test.js ``` - `TZ=UTC npm run verify` at the tip: exit 0 — core 3091/3091, social 619, tree 149, timeline 172, workflow 528, project 147, scripts+policy 675, interop 37, live-e2e 34; core lint 88 warnings / 0 errors; `10 patches apply cleanly in order` and canonical no-op. Its internal `arch-docs: OK` line counts against this clone's `origin/main` ref (`f42ad016`, the fork's older main), so its file count is not this PR's; the explicit `ARCH_DOCS_BASE=429668e2` run below is. - `ARCH_DOCS_BASE=429668e2 node core/scripts/check-arch-docs.mjs` on the committed tree: `arch-docs: OK (2 changed file(s))`. Not run: `build-client` (no `core/src/client/` change), the smoke test (no host boot change), and the merge-queue checks (CI runs them). ## Generated artifacts `node scripts/ds-bot-patches.mjs generate` rewrites exactly one path beyond the source test: its mirror `patches/ds-bot/files/test/memory-search-edge.test.js` (+11/−6, same bytes as the source file). All ten `patches/ds-bot/0*.patch` files are byte-identical to `429668e2` — the mirror is not referenced by any patch. `check` says `ok ds-bot/ = Fei2411/DeepSeek-Bot@2b10c70` / `ok 10 patches apply cleanly in order` / `ok core/ equals ds-bot/ + patches/ds-bot/`, and `canonical` says `ok patches/ds-bot/ is what generate writes (generate is a no-op)`. ## Scope | file | +/− | | --- | --- | | `core/test/memory-search-edge.test.js` | +11/−6 | | `patches/ds-bot/files/test/memory-search-edge.test.js` | +11/−6 (generated) | Tier basis: `risk-tier: T1; basis: core/test/memory-search-edge.test.js=T1, patches/ds-bot/files/test/memory-search-edge.test.js=T1` (18 changed lines in the 1 non-generated file; the generated mirror is not counted). ## Why no changelog fragment Test hardening only: no user-visible behaviour changes, and the file the fragment would describe (`recall` ordering) behaves exactly as it did before. The repository asks for a fragment when a user-visible fix ships; this ships a guard, not a fix. Verified commit: 958d0bc52e8432b358259c9072f45dc22a0d04b2 (tree ab74309); the pushed head below is that tree rebuilt on live main. ## Checkpoint <!-- flybot:checkpoint --> ```json { "pr": 2849, "claim": "Dev-1", "issue": 2846, "summary": "The determinism guard in core/test/memory-search-edge.test.js compared the same searchIndex call with itself, so it could never fail. It now pins the tie-break by key, stability across a reordered index rebuild, and best-first ordering — the order recall returns. Two source mutations made the old assertion pass 5/5 and the new one fail 1 of 5; at this tip the file is 5/5 and verify is green.", "files": [ "core/test/memory-search-edge.test.js", "patches/ds-bot/files/test/memory-search-edge.test.js" ], "checks": { "ran": "TZ=UTC npm run verify", "result": "green", "at": "50b7324175b26863a1dc2f77d54d5822f32c575c" }, "next": "none — test-only T1, ready for the T1 lane" } ``` <!-- /flybot:checkpoint -->
1 parent 429668e commit ff9c5db

2 files changed

Lines changed: 22 additions & 12 deletions

File tree

‎core/test/memory-search-edge.test.js‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,18 @@ test('a Chinese query finds a Chinese entry, with no word split', () => {
1515
assert.ok(!searchIndex(index, 'noodles').some(hit => hit.doc.key === 'seat'))
1616
})
1717

18-
test('a word stem matches across endings, and ranking is deterministic', () => {
18+
test('a word stem matches across endings, and the ranking is deterministic', () => {
1919
const docs = [{ key: 'a', ref: 'The user books flights' }, { key: 'b', ref: 'Likes strong coffee' }]
20-
const first = searchIndex(buildIndex(docs), 'flight bookings').map(hit => hit.doc.key)
21-
assert.deepEqual(first, ['a'])
22-
// The same index and query give the same order every time.
23-
const index = buildIndex(docs)
24-
assert.deepEqual(searchIndex(index, 'flight bookings').map(hit => hit.doc.key), searchIndex(index, 'flight bookings').map(hit => hit.doc.key))
20+
const ranked = (entries, query = 'flight bookings') => searchIndex(buildIndex(entries), query).map(hit => hit.doc.key)
21+
assert.deepEqual(ranked(docs), ['a'])
22+
// Two entries that share every term score the same, so the key breaks the tie: the recall list is
23+
// the same on every read, and re-reading the files in another order does not reorder it.
24+
const tied = [{ key: 'b', ref: 'the window seat' }, { key: 'a', ref: 'the window seat' }, { key: 'c', ref: 'nothing here' }]
25+
assert.deepEqual(ranked(tied, 'window seat'), ['a', 'b'])
26+
assert.deepEqual(ranked([...tied].reverse(), 'window seat'), ['a', 'b'])
27+
// An entry matching both query words ranks above one that matches a single word.
28+
const partial = [{ key: 'only-window', ref: 'window' }, { key: 'both', ref: 'window seat' }]
29+
assert.deepEqual(ranked(partial, 'window seat'), ['both', 'only-window'])
2530
})
2631

2732
test('an empty query, and a tokenization with nothing to keep, return no hits', () => {

‎patches/ds-bot/files/test/memory-search-edge.test.js‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,18 @@ test('a Chinese query finds a Chinese entry, with no word split', () => {
1515
assert.ok(!searchIndex(index, 'noodles').some(hit => hit.doc.key === 'seat'))
1616
})
1717

18-
test('a word stem matches across endings, and ranking is deterministic', () => {
18+
test('a word stem matches across endings, and the ranking is deterministic', () => {
1919
const docs = [{ key: 'a', ref: 'The user books flights' }, { key: 'b', ref: 'Likes strong coffee' }]
20-
const first = searchIndex(buildIndex(docs), 'flight bookings').map(hit => hit.doc.key)
21-
assert.deepEqual(first, ['a'])
22-
// The same index and query give the same order every time.
23-
const index = buildIndex(docs)
24-
assert.deepEqual(searchIndex(index, 'flight bookings').map(hit => hit.doc.key), searchIndex(index, 'flight bookings').map(hit => hit.doc.key))
20+
const ranked = (entries, query = 'flight bookings') => searchIndex(buildIndex(entries), query).map(hit => hit.doc.key)
21+
assert.deepEqual(ranked(docs), ['a'])
22+
// Two entries that share every term score the same, so the key breaks the tie: the recall list is
23+
// the same on every read, and re-reading the files in another order does not reorder it.
24+
const tied = [{ key: 'b', ref: 'the window seat' }, { key: 'a', ref: 'the window seat' }, { key: 'c', ref: 'nothing here' }]
25+
assert.deepEqual(ranked(tied, 'window seat'), ['a', 'b'])
26+
assert.deepEqual(ranked([...tied].reverse(), 'window seat'), ['a', 'b'])
27+
// An entry matching both query words ranks above one that matches a single word.
28+
const partial = [{ key: 'only-window', ref: 'window' }, { key: 'both', ref: 'window seat' }]
29+
assert.deepEqual(ranked(partial, 'window seat'), ['both', 'only-window'])
2530
})
2631

2732
test('an empty query, and a tokenization with nothing to keep, return no hits', () => {

0 commit comments

Comments
 (0)