From ad220efcabb2dbd09f6050335bcabdc72fbc1517 Mon Sep 17 00:00:00 2001 From: MemOS AutoDev Date: Wed, 2 Sep 2026 09:02:53 +0800 Subject: [PATCH] fix(memos-local-plugin): guard L2 induction against conversational-act actions (#2318) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit L2_INDUCTION_PROMPT (v2) had a boundary block that rejected declarative L3 world-model facts but was silent on a symmetric drift: dialogue acts (ask/confirm/notify/report status) leaking into the `action` field. When a trace cluster's dominant behaviour was dialogue, induction minted policies whose action WAS the dialogue, and those policies later crystallised into dead skills — retrieved as candidates, never callable. Add a second boundary subsection that (a) enumerates the dialogue-verb classes to reject, (b) shows a wrong/right contrast on the audit's canonical skill-viewing scenario, and (c) offers an explicit {"abstain":true,"reason":"..."} escape hatch for the dialogue-only cluster case. Bump prompt version 2 → 3 and mechanically migrate the 11 op-key fixtures across the vitest suite. Full plugin suite: 183 files, 1558 passed / 2 skipped. tsc --noEmit clean. --- .../core/llm/prompts/l2-induction.ts | 70 +++++++++++++++++-- .../tests/e2e/v7-full-chain.e2e.test.ts | 2 +- .../adapters/openclaw-full-chain.test.ts | 4 +- .../tests/unit/llm/prompts.test.ts | 29 ++++++++ .../tests/unit/memory/l2/induce.test.ts | 6 +- .../unit/memory/l2/l2.integration.test.ts | 12 ++-- 6 files changed, 106 insertions(+), 17 deletions(-) diff --git a/apps/memos-local-plugin/core/llm/prompts/l2-induction.ts b/apps/memos-local-plugin/core/llm/prompts/l2-induction.ts index beed29fa5..c15bf98eb 100644 --- a/apps/memos-local-plugin/core/llm/prompts/l2-induction.ts +++ b/apps/memos-local-plugin/core/llm/prompts/l2-induction.ts @@ -11,15 +11,22 @@ import type { PromptDef } from "./index.js"; * Boundary contract (see `docs/GRANULARITY-AND-MEMORY-LAYERS.md` §6): * an L2 policy is **procedural** ("how to do it") — it MUST contain an * action template. Anything declarative ("the environment looks like X") - * belongs to the L3 world model, not here. The system prompt explicitly - * rejects environment-fact drift to keep the two layers semantically - * orthogonal. Bumping the version to v2 captures that change. + * belongs to the L3 world model, not here. + * + * v3 (issue #2318): adds a second boundary against **conversational acts** + * (asking, confirming, notifying, reporting status) leaking into the + * `action` field. Such policies passed schema validation but crystallised + * into dead skills — always retrieved, never callable. The revised prompt + * (a) enumerates the dialogue verbs to reject and (b) tells the model to + * abstain when the cluster's only shared behaviour is dialogue. + * + * v2 history: added the L3 world-model drift guard. */ export const L2_INDUCTION_PROMPT: PromptDef = { id: "l2.induction", - version: 2, + version: 3, description: - "Distill an L2 policy (procedural sub-task strategy) from a cluster of similar L1 traces, with explicit boundaries against L3 world-model drift.", + "Distill an L2 policy (procedural sub-task strategy) from a cluster of similar L1 traces, with explicit boundaries against L3 world-model drift and against conversational-act 'actions'.", system: `You induce reusable **procedural policies** from agent experience. A policy is a "how-to": "when you see condition X in the agent's state, @@ -83,6 +90,59 @@ libs by default": "Alpine container images ship only the pure-Python tier of the Python dependency stack." +──────────────────── Boundaries — conversational acts are NOT policies ──────────────────── + +A **conversational act** is an I/O behaviour aimed at the user (or +another agent) — not a procedure applied to the environment. When a +trace cluster's dominant shared behaviour is dialogue, do NOT wrap that +dialogue as an ACTION. Such policies pass schema validation but produce +"dead skills" — the crystalliser promotes them, retrieval surfaces +them, and no agent can meaningfully invoke them because there is +nothing to invoke. + +Dialogue verbs / patterns to REJECT as ACTION templates (non-exhaustive): + - **asking the user** to clarify, choose, or provide input + ("ask the user to confirm the skill name before viewing it") + - **requesting confirmation** before proceeding + ("confirm with the user before deleting the file") + - **notifying / informing** the user about state + ("notify the user that the build finished") + - **reporting status** back to the user + ("report the current pipeline status to the user") + - **explaining / clarifying** to the user + ("explain to the user why the request failed") + +Contrast: + + Wrong (dialogue as action — dead skill): + trigger: "user hasn't specified which skill they want to view" + action: "ask the user to confirm the skill name before viewing it" + + Right (dialogue folded into trigger; real procedure as action): + trigger: "user's view-skill request omits an unambiguous skill id + AND at least two skills fuzzy-match the phrase they used" + action: "1. look up all skills whose name/tags fuzzy-match the + user's phrase; 2. if exactly one hits, resolve to that + skill's id; 3. if multiple hit, present the top 3 with + ids and let the user pick by id in the next turn" + +The distinction: the "wrong" version's action IS the dialogue; the +"right" version's action is a resolution procedure the agent can carry +out, and asking the user is at most a fallback branch, not the main +step. If your only shared behaviour across traces is "the agent talks +to the user", you have NO policy to induce — see the abstain rule +below. + +Abstain when the cluster IS dialogue-only: if every trace in the input +reduces to the same dialogue act with no follow-up procedure the agent +executes on the environment, **do not emit a policy**. Instead return: + + { "abstain": true, "reason": "cluster's shared behaviour is a + conversational act, not a reusable procedure — no policy to induce" } + +An abstained induction is a successful outcome, not a failure — it +prevents dead skills from entering the pool. + ──────────────────── Output ───────────────────── Return JSON: diff --git a/apps/memos-local-plugin/tests/e2e/v7-full-chain.e2e.test.ts b/apps/memos-local-plugin/tests/e2e/v7-full-chain.e2e.test.ts index 07e3bf60e..03686fc2b 100644 --- a/apps/memos-local-plugin/tests/e2e/v7-full-chain.e2e.test.ts +++ b/apps/memos-local-plugin/tests/e2e/v7-full-chain.e2e.test.ts @@ -189,7 +189,7 @@ function buildFullChainLlm(): LlmClient { }), // L2 induction — distills a policy from ≥2 similar traces. - "l2.l2.induction.v2": (input: unknown) => { + "l2.l2.induction.v3": (input: unknown) => { const text = lastUserMessage(input); const isPython = /python|pip|\.py\b/i.test(text); return { diff --git a/apps/memos-local-plugin/tests/integration/adapters/openclaw-full-chain.test.ts b/apps/memos-local-plugin/tests/integration/adapters/openclaw-full-chain.test.ts index fa12b9189..b55de7395 100644 --- a/apps/memos-local-plugin/tests/integration/adapters/openclaw-full-chain.test.ts +++ b/apps/memos-local-plugin/tests/integration/adapters/openclaw-full-chain.test.ts @@ -26,7 +26,7 @@ * - `capture.alpha.reflection.score.v1`— α scoring * - `capture.summarize` — trace-level summaries * - `reward.reward.r_human.v3` — R_human axis scoring - * - `l2.l2.induction.v2` — L2 policy induction + * - `l2.l2.induction.v3` — L2 policy induction * - `l3.abstraction.v2` — L3 world-model abstraction * - `skill.crystallize` — skill draft * @@ -234,7 +234,7 @@ function buildLlm(): LlmClient { reason: "concrete root-cause reflection", }), - "l2.l2.induction.v2": (input: unknown) => { + "l2.l2.induction.v3": (input: unknown) => { const evidence = (input as { evidenceTraces?: Array<{ id: string }> }) ?.evidenceTraces ?? []; return { diff --git a/apps/memos-local-plugin/tests/unit/llm/prompts.test.ts b/apps/memos-local-plugin/tests/unit/llm/prompts.test.ts index bee52730a..053f1d705 100644 --- a/apps/memos-local-plugin/tests/unit/llm/prompts.test.ts +++ b/apps/memos-local-plugin/tests/unit/llm/prompts.test.ts @@ -48,6 +48,35 @@ describe("llm/prompts", () => { expect(detectDominantLanguage(["GRPO / TRL / reward_fn.py"])).toBe("en"); }); + it("l2 induction prompt guards against conversational-act actions (issue #2318)", () => { + // Version must bump when we tighten guidance so downstream + // `promptId@version` records attribute the new behaviour correctly. + expect(L2_INDUCTION_PROMPT.version).toBe(3); + + const sys = L2_INDUCTION_PROMPT.system; + + // The boundary block must name conversational acts as a rejected + // ACTION shape (symmetric to the existing L3 world-model drift guard). + expect(sys).toMatch(/conversational act/i); + + // Enumerate the dialogue verbs we want the model to recognise as + // dead-skill precursors — must appear in the prompt so the LLM has a + // concrete list to check itself against. + expect(sys).toMatch(/\bask(ing)?\b.*\bthe user\b|\bask the user\b/i); + expect(sys).toMatch(/\bconfirm(ation)?\b/i); + expect(sys).toMatch(/\bnotify(ing)?\b|\bnotification\b/i); + expect(sys).toMatch(/\breport(ing)? status\b|\bstatus report\b/i); + + // When the trace cluster's ONLY shared behaviour is dialogue, the + // prompt must offer the model an "abstain" escape hatch instead of + // forcing it to fabricate a policy. + expect(sys).toMatch(/abstain|no policy|do not emit|omit the policy/i); + + // The existing L3-drift guard must still be present — this change is + // additive, not a rewrite. + expect(sys).toMatch(/L3|world model|world-model/i); + }); + it("retrieval filter prompt asks for ranked output without selected-field leftovers", () => { expect(RETRIEVAL_FILTER_PROMPT.system).toContain('"ranked"'); expect(RETRIEVAL_FILTER_PROMPT.system).not.toContain('"selected"'); diff --git a/apps/memos-local-plugin/tests/unit/memory/l2/induce.test.ts b/apps/memos-local-plugin/tests/unit/memory/l2/induce.test.ts index 06d61459f..b6df2e7bf 100644 --- a/apps/memos-local-plugin/tests/unit/memory/l2/induce.test.ts +++ b/apps/memos-local-plugin/tests/unit/memory/l2/induce.test.ts @@ -44,7 +44,7 @@ describe("memory/l2/induce", () => { it("returns {ok:true, draft} and fills support_trace_ids when the LLM omits them", async () => { const llm = fakeLlm({ completeJson: { - "l2.l2.induction.v2": { + "l2.l2.induction.v3": { title: "install system libs first", trigger: "pip install fails in container with missing system library", procedure: "1. detect missing lib 2. apk/apt-get install 3. retry pip", @@ -75,7 +75,7 @@ describe("memory/l2/induce", () => { it("cleans unsafe markup from LLM-derived policy fields", async () => { const llm = fakeLlm({ completeJson: { - "l2.l2.induction.v2": { + "l2.l2.induction.v3": { title: " install system libs", trigger: "pip fails [bad](javascript:alert(1))", procedure: "Use [safe](https://example.com), ignore [bad](javascript:alert(1))", @@ -149,7 +149,7 @@ describe("memory/l2/induce", () => { it("reason=llm_failed when the LLM draft is malformed (missing title)", async () => { const llm = fakeLlm({ completeJson: { - "l2.l2.induction.v2": { trigger: "no title", procedure: "..." }, + "l2.l2.induction.v3": { trigger: "no title", procedure: "..." }, }, }); const res = await induceDraft( diff --git a/apps/memos-local-plugin/tests/unit/memory/l2/l2.integration.test.ts b/apps/memos-local-plugin/tests/unit/memory/l2/l2.integration.test.ts index d9d062d1f..f3bc7068c 100644 --- a/apps/memos-local-plugin/tests/unit/memory/l2/l2.integration.test.ts +++ b/apps/memos-local-plugin/tests/unit/memory/l2/l2.integration.test.ts @@ -112,7 +112,7 @@ describe("memory/l2/integration", () => { const llm = fakeLlm({ completeJson: { - "l2.l2.induction.v2": { + "l2.l2.induction.v3": { title: "install missing system libs in container", trigger: "pip install fails in container with MODULE_NOT_FOUND due to missing system lib", procedure: "1. detect lib 2. use distro pkg manager 3. retry pip", @@ -367,7 +367,7 @@ describe("memory/l2/integration", () => { repos: handle.repos, llm: fakeLlm({ completeJson: { - "l2.l2.induction.v2": { + "l2.l2.induction.v3": { title: " 闲聊场景下避免使用emoji ", trigger: "用户发起非任务性闲聊，且未明确要求使用emoji", procedure: "以简洁、自然的文字回应，但不添加任何emoji或表情符号", @@ -457,7 +457,7 @@ describe("memory/l2/integration", () => { repos: handle.repos, llm: fakeLlm({ completeJson: { - "l2.l2.induction.v2": { + "l2.l2.induction.v3": { title: "结构化呈现外部数据查询结果", trigger: "agent通过工具调用获取到外部数据源的原始响应（搜索结果、API返回、数据库查询等），需要向用户呈现信息", procedure: "1) 从原始数据中提取关键信息要素；2) 按逻辑分类组织信息（时间、地点、数值、状态等）；3) 使用结构化格式呈现（分类标题、列表、表格等）；4) 可选：基于数据添加简短的实用性总结或建议", @@ -548,7 +548,7 @@ describe("memory/l2/integration", () => { repos: handle.repos, llm: fakeLlm({ completeJson: { - "l2.l2.induction.v2": { + "l2.l2.induction.v3": { ...shared, boundary: "仅适用于数据库事务和写入操作，不适用于天气API或公开搜索结果", rationale: "边界不同，不能复用天气呈现经验", @@ -630,7 +630,7 @@ describe("memory/l2/integration", () => { repos: handle.repos, llm: fakeLlm({ completeJson: { - "l2.l2.induction.v2": { + "l2.l2.induction.v3": { title: "工具全部失效时坦诚说明并回退到已有知识", trigger: "连续尝试多个同类工具（如多个搜索引擎、多个新闻源）后全部因反爬、封禁或网络错误失效，且用户问题需要实时信息", procedure: "1) 明确告知用户所有尝试过的工具都已失效（列出具体工具名）；2) 基于训练数据中的已有知识提供最接近的答案；3) 明确标注该信息的时间戳或来源时间，并提醒用户可能存在时效性差异；4) 建议用户通过其他渠道（如直接搜索、官方网站）验证", @@ -663,7 +663,7 @@ describe("memory/l2/integration", () => { const llm = fakeLlm({ completeJson: { - "l2.l2.induction.v2": { + "l2.l2.induction.v3": { title: "t", trigger: "tr", procedure: "pr",