fix(plugin): reject conversational acts in L2 policy induction (v3) - #2324
fix(plugin): reject conversational acts in L2 policy induction (v3)#2324smoryan wants to merge 2 commits into
Conversation
Policies whose core action is talking to the user (asking clarifying questions, requesting confirmation, reporting status) are one-off dialogue behaviours, not reusable procedures — once induced they crystallize into dead skills that no agent can invoke. Add a "Conversational acts are not policies" section to the L2 induction system prompt with a wrong/right example pair: if the underlying trace really contains a reusable check, express the CHECK itself as the action. Bump the prompt version v2 -> v3 so the re-induction pass rewrites affected policies.
🤖 Open Code ReviewTarget: PR #2324 🔍 OpenCodeReview found 1 issue(s) in this PR. 1.
|
Co-Authored-By: LamzQ <linxlam@foxmail.com>
|
Thanks — confirmed and fixed in 42d68d5: the version-bump sentence now reads v3 (the v2 sentence was upstream history left stale by the v3 edit). Pure JSDoc, no behavior change; domain tests still 57/57. |
✅ Automated Test Results: PASSEDAll tests passed (14/14 executed). memos_local_plugin/unit: 14/14. Duration: 4s [advisory, non-gating] AI-generated tests on branch test/auto-gen-f13e586fd64f9c82-20260902095939: 16/17 passed, 1 failed — these do NOT affect the PR verdict; review the branch manually. Branch: |
Summary
Conversational acts — asking clarifying questions, requesting confirmation,
notifying, reporting status — are one-off dialogue behaviours, not reusable
procedures. When the L2 induction LLM writes them as a policy
actionanyway,they later crystallize into dead skills that no agent can invoke. This PR adds
an explicit negative list to the induction system prompt and bumps the prompt
to v3 so the re-induction pass rewrites affected policies.
Problem
An internal 25-day production audit (2026-08-28) of the crystallized-skill
pool identified conversational-act policies as a recurring failure mode: they
are induced and stored like any other policy, but no agent can ever invoke
them, so they accumulate as dead weight. In the same window the pool received
207 fresh generations of which 98% of candidates were never invoked — the
quality of what enters the pool is the upstream half of that problem, and the
induction prompt had no guidance separating reusable checks from dialogue.
Change
core/llm/prompts/l2-induction.ts— new "Conversational acts are notpolicies" section: a negative list (clarify / confirm-with-user / notify /
report status) plus a wrong/right example pair steering the model to express
a reusable CHECK as the action instead (e.g. resolve the skill id via list
search rather than ask the user to confirm it). Doc comment records the v3
history.
version: 2→3— induction op names embed the prompt version(
l2.${id}.v${version}), so the bump drives re-induction onto the newprompt on the next pass.
tests/unit/memory/l2/induce.test.ts,tests/unit/memory/l2/l2.integration.test.ts—fakeLlmmock keysl2.l2.induction.v2→.v3(9 occurrences). Without this the fake LLM hasno mock for the new op and 8 induction tests fail with
llm_failed—which doubles as confirmation the version bump takes effect.
Tests
npx vitest run tests/unit/llm/prompts.test.ts tests/unit/memory/l2/→ 57 passed (10 files)npx vitest run tests/unit/→ 1549 passed, 2 skipped, 0 failednpx tsc --noEmit→ clean (exit 0)Related
this PR addresses an upstream cause — the induction prompt steering the LLM
toward conversational acts in the first place.
Type of change
How Has This Been Tested?
npx vitest run tests/unit/llm/prompts.test.ts tests/unit/memory/l2/— 57 passed, 10 files)npx vitest run tests/unit/— 1549 passed, 0 failed)tsc --noEmitclean)Checklist