Skip to content

docs(skill): state the mentor session model, not one surface's call sequence - #58

Merged
OS-joaocastilho merged 2 commits into
mainfrom
skill-mentor-session-model
Sep 7, 2026
Merged

docs(skill): state the mentor session model, not one surface's call sequence#58
OS-joaocastilho merged 2 commits into
mainfrom
skill-mentor-session-model

Conversation

@miguelglopes

Copy link
Copy Markdown
Collaborator

This PR is for... making the mentor guidance correct on every tenant, by describing the session model instead of one tool arrangement.

What was happening

The mentor tool surface is not the same on every tenant. How a conversation is split across calls is decided server-side, and the arrangement a tenant does not have is absent from its tools/list and rejected if called by name.

The skill docs described one arrangement. #57 removed the literal tool names, but the mechanics underneath them stayed, stated as facts of the transport rather than of one surface:

  • a signed session credential echoed back to resume a conversation
  • a resume-only flag for resetting the conversation
  • a turn-time ceiling on the call that starts a turn
  • a publish that takes that credential pair plus an environment key

An agent on the other arrangement follows all of it, is rejected, and finds nothing in the doc explaining why. It also never learns two things it needs: that a session cannot edit an app until the app is in it, and that releasing a session discards whatever in it was never published.

Three statements were also wrong on at least one surface, one of them dangerously:

  • app_key on publish was described as ignored. The schema is additionalProperties: false and has no such property, so passing one fails the whole call.
  • The publish was described as taking an environment key. On the other surface it takes the session and a comment, and lands in development.
  • A GC'd session was described as resuming transparently in one place and as losing unpublished work in another — the same doc gave both answers.

What was done

The mentor guidance is now written as invariants of the session model. Each holds however the calls are arranged, and the per-call specifics come from the live catalog:

  • a session holds the model and outlives the turn
  • a session edits exactly one app, and cannot edit it until the app is in it
  • a session runs one turn at a time
  • carry the session handle forward unchanged, newest wins
  • reuse one session for the whole task
  • an idle session does not live forever, and takes unpublished edits with it
  • release a session only once its work is published or deliberately abandoned, because releasing discards drafts
  • succeeded means the turn ended, not that the change landed
  • publishing takes the session, never an app key

Cursor paging, the reset flag, the turn-time field and a surface-specific error code became conditions or deferrals to the catalog. The environment on publish is described as a choice where the catalog exposes one. The workflow and the Kiro Power's Workflow 2 follow the same shape, and gain a release step.

Also removed: a claim I had added in an earlier draft that not every surface identifies a session with a UUID. Both do.

Version bumped 0.18.1 → 0.19.0.

Test Steps

  1. grep each doc for any tool name from either surface's live tools/list — expect none. Same for the surface-specific parameters and error codes (mentor_session_token, fresh_context, max_turn_time, turn_error, cursor_dropped, and the max-length error code).
  2. Lockstep: pick two distinctive phrases from ## Mentor session round-trip and confirm 5/5 across the five skill docs.
  3. Drift: diff the ## Tools at a glance-to-EOF span of each doc against root SKILL.md. Expect 18 / 5 / 4 / 4 for skills/, cursor/, copilot/, kiro/ — unchanged from main, i.e. no new host-specific divergence.
  4. End to end on a tenant of each arrangement: edit an app, publish it, and confirm the doc's sequence is followable without a rejected call. Not yet done — see below.

Screenshots

n/a, docs only.

Checklist

  • every supported harness accounted for, each one verified, not applicable with the reason, or a recorded gap with a follow-up (every row of the CLAUDE.md harness table)
  • no internal references leaked (stage hostnames, internal Jira projects, team jargon)
  • if skill content or behavior changed: kept in sync across skills/outsystems/SKILL.md, kiro/outsystems/steering/skill.md, copilot/skill.md, cursor/skills/outsystems/SKILL.md, and root SKILL.md, plus kiro/outsystems/POWER.md where its ## Conventions carries the same rules
  • if releasing: version bumped in all four manifests (.claude-plugin/plugin.json, .claude-plugin/marketplace.json, cursor/.cursor-plugin/plugin.json, .cursor-plugin/marketplace.json)

Harness table. The change is skill-doc wording only, so per CONTRIBUTING that means the lockstep check plus a read of the affected section in each doc — both done for all five docs and the Power. No install recipe, MCP config or manifest shape changed, so nothing needed a per-harness config resolve.

Worth stating plainly: Copilot and Cursor CLI receive their skill doc by manual copy, so a version bump does not reach existing users of those two. That is a pre-existing delivery gap, not one this PR introduces, and it is why this content fix does not by itself close the problem on every harness.

Why it is a draft

The rewrite is verified against both surfaces' live catalogs and an end-to-end call log, but the resulting doc has not been walked end to end on a tenant of each arrangement. That is step 4 above and the thing I would want before merging.

Three claims in the doc err on the safe side rather than resting on direct observation, and are called out here so a reviewer can weigh them:

  • Releasing a session discards unpublished work. The tool's own description says it releases the session's context, and an end-to-end log annotates a close as having discarded a draft — but nobody reopened the app afterwards to confirm the edits were gone.
  • After the idle limit the session is gone. Verified on one surface; on the other, sessions were observed to expire but without a limit value or an observed resume-after-expiry.
  • Whether a second app can be loaded into a live session. The server's own text says only one asset can be loaded or created per session, so the invariant is contract-backed; whether a second load is rejected or silently replaces the first is unobserved. The guidance — use a different session — is right either way.

…equence

How a mentor conversation is split across tool calls is a server-side
decision that differs between tenants, and the agent cannot control
which arrangement it gets. The skill docs described one arrangement, and
#57 removed the literal tool names without removing the assumption
underneath them: the signed credential echoed back on resume, the
resume-only context-reset flag, the turn-time ceiling, the cursor-paged
event stream, and a publish that takes that credential pair plus an
environment were all still stated as facts of the transport. An agent on
the other arrangement follows all of it, is rejected, and has nothing in
the doc to tell it why.

The mentor guidance is now written as invariants of the session model
that hold however the server arranges its tools, with every name, count
and argument left to the live tools/list:

- a session holds the OML and outlives the turn; a failed turn is
  continued in the same session, and a failure during session setup is
  fixed by redoing the setup step, not the prompt
- a session edits exactly one app and cannot edit it until that app is
  in it
- a session runs one turn at a time; a start that appears to do nothing
  while a run is live means poll the run you already have
- the session handle is carried forward unchanged from whichever
  response handed it over, newest wins
- one session serves the whole task; a new one starts from the
  published OML and drops unpublished edits
- an idle session does not live forever; after the idle limit it is
  gone and its unpublished edits with it, so publish before a long pause
- releasing a session discards whatever in it was never published, so
  release only after the publish landed or the edits were abandoned
- `succeeded` means the turn ended, not that the change landed; read the
  completion signals (change attempted, change applied, validation)
- publishing takes the session, never an app key; the target
  environment is the catalog's to expose, and the publish lands in
  development where it is not a choice
- a timeout is retried in the same session on the same app, raising the
  turn-time ceiling where the catalog exposes one and narrowing the
  prompt where it does not

Checked against the live tool catalogs of both arrangements. On one the
terminal payload carries no session handle, so "the handle comes back on
the terminal payload" is gone. On the other an extra app key on publish
is rejected rather than ignored. Closing a session discards its
unpublished draft, so "release when done" now says after the publish
landed. Both identify sessions with UUIDs, so the feedback field's claim
that some surfaces do not is dropped. Cursor paging and the max-length
error code were one arrangement's mechanics and are left to the tool
descriptions; the max-length condition is stated in words.

Applied across the five lockstepped skill docs (SKILL.md,
skills/outsystems/SKILL.md, copilot/skill.md,
cursor/skills/outsystems/SKILL.md, kiro/outsystems/steering/skill.md)
and, in its operator register, kiro/outsystems/POWER.md. Lockstep grep:
"runs one turn at a time" 5/5, "does not live forever" 5/5; the
Tools-at-a-glance-to-EOF drift against root stays at 18/5/4/4. Bump the
plugin version to 0.19.0 across the four manifest files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: converged — 1 round, 0 confirmed findings.

This run completed in one round. The panel was docs-reviewer (always) and consistency-reviewer (manifests changed). error-handling-reviewer and simplification-reviewer were skipped (docs-only diff).

Severity counts: MUST: 0 | SHOULD: 0 | COULD: 0


Minor / low-confidence (not posted as inline comments)

No findings reached this tier either.


Challenged and dropped

Finding A (docs-reviewer): POWER.md line 108 "succeeded alone means the turn ended" drops the contrast "not that the change landed"
REFUTED. The word alone qualifies succeeded as conveying only that the turn ended — not that it is sufficient evidence of success. The sentence is immediately preceded by a directive to read the completion signals (change attempted, change applied, validation errors). The instruction is architecturally identical to the five skill docs' version. In addition, ## Conventions line 134 of POWER.md carries the parallel standalone rule about succeeded/failed/cancelled being the only terminal statuses. No misleading gap exists.

Finding B (docs-reviewer): "cursor pattern" in the Don't bare-sleep bullet is a dangling cross-reference
REFUTED. The Pacing polls section still contains actionable cursor guidance: the Only status says a run finished bullet explicitly names a "stale-cursor error" and describes the recovery. The cross-reference to "Pacing polls" for "cadence and the cursor pattern" still resolves to meaningful content.


Panel that ran: docs-reviewer, consistency-reviewer

Consistency-reviewer results (passing):

  • All four manifests confirmed at 0.19.0: .claude-plugin/plugin.json, .claude-plugin/marketplace.json, cursor/.cursor-plugin/plugin.json, .cursor-plugin/marketplace.json
  • Six lockstep phrases checked across all five skill docs — all return 1/1/1/1/1
  • Plugin skills path resolves correctly in both plugin.json files
  • The two documented ## Rules wording exceptions ("lazy sign-in" vs "lazy authentication step"; "a setup fault, not a retry target" vs "routed back to setup, not retried") remain the only divergences across the five docs; no new drift introduced

End-to-end on a live tenant: issuing a second prompt while a run was in
flight returned status working and a fresh run identifier, and that run
later reached succeeded on its own. The bullet said such a start 'appears
to do nothing', which would have an agent looking for a symptom that does
not occur — the second identifier is indistinguishable from a good one.
The remedy is unchanged; only the symptom description was wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review verdict: converged — 1 round, 0 confirmed MUST/SHOULD findings.

Prior bot review: commit bc9a9d5 (2026-09-04), 0 confirmed findings, 0 inline comments.

Fix scorecard (prior findings vs. current code):
No prior findings existed, so no scorecard entries.

This push (bc9a9d5 -> 6d1374b) changed:

  • AGENTS.md: added as a symlink to CLAUDE.md (makes codebase instructions visible to Codex-family agents that look for AGENTS.md)
  • All 5 skill docs: updated the single-turn bullet in ## Mentor session round-trip to replace "A start that appears to do nothing while a run is live" with the more precise "A start issued anyway is not reliably refused — you may get back a run identifier that looks exactly like a good one"

Lockstep check (run against HEAD):

  • Phrase "not reliably refused": 1/1/1/1/1 across all five skill docs
  • Phrase "Keep polling the run you started first": 1/1/1/1/1 across all five skill docs

Severity counts: MUST: 0 | SHOULD: 0 | COULD: 1 (below)


Minor / low-confidence (not posted as inline comments)

Finding C1 (docs-reviewer, medium confidence): POWER.md Workflow 2 step 1 carries the old "appears to do nothing" phrasing

kiro/outsystems/POWER.md:107 (new-side line) still reads: "a session runs one turn at a time, so a start that appears to do nothing while a run is live means poll the run you already have."

All five skill docs now say the erroneous second start "is not reliably refused" and may return a convincing run identifier. The POWER.md phrasing implies the duplicate call is visually a no-op, which the skill-doc change was specifically designed to correct.

Why medium confidence: CLAUDE.md's lockstep rule applies to ## Conventions verbatim rules with a shared lead sentence; ## Common Workflows is listed as a section "rewritten for an operator" and the phrase-count grep does not cover workflow step prose. The right corrective action is unambiguous (update the clause), but whether it is mandatory under the documented policy is genuinely ambiguous. Classed COULD rather than SHOULD.

Proposed fix: Change the clause in POWER.md line 107 from "a start that appears to do nothing while a run is live means poll the run you already have" to something like "a start issued while a run is live is not reliably refused and may return a run identifier that looks valid, so keep polling the run already started."


Challenged and dropped

No findings were fully refuted this round. The POWER.md finding (C1 above) was partially challenged on the grounds that ## Common Workflows is operator-rewritten prose rather than a lockstepped rule, and that the behavioral gap is intentional (operators need the action, not the agent-level warning). The finding was retained at COULD rather than dropped, because the wording difference is real and the prose actively misleads on the observable behavior.


Panel that ran: docs-reviewer (always; docs changed). consistency-reviewer skipped (no new manifest or JSON changes in this push). error-handling-reviewer and simplification-reviewer skipped (docs-only diff).

@OS-joaocastilho
OS-joaocastilho marked this pull request as ready for review September 7, 2026 09:05
@OS-joaocastilho
OS-joaocastilho requested a review from a team as a code owner September 7, 2026 09:05
@OS-joaocastilho
OS-joaocastilho merged commit 497afa0 into main Sep 7, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants