Skip to content

feat(skills): give Claude Agents runs the skills attached to their variation - #136

Open
XieX wants to merge 13 commits into
mainfrom
xie/claude-agents-skills
Open

XieX wants to merge 13 commits into
mainfrom
xie/claude-agents-skills

Conversation

@XieX

@XieX XieX commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Base is main: #135 (host options) has merged.

A skill attached to an agent-mode variation never reached a model run through createClaudeAgentsHandler. Measured on Agent SDK 0.3.220 by reading the CLI's init message: skills are discovered, but the handler's tools: [] removes the Skill tool, and no query({ skills }) value brings it back.

createClaudeAgentsSkillsHandler(options), from the new @launchdarkly/ai-claude-agents/experimental entry point, fixes that. On every invoke and stream it:

  1. Writes skillRefs(config) with writeSkills, through the store configured with setSkillStore, into mkdtemp(<skillsTempDir>/ld-skills-)/launchdarkly/skills. Pinned versions are honored.
  2. Passes plugins: [{ type: 'local', path: <dir>/launchdarkly, skipMcpDiscovery: true }] and skills: ['launchdarkly:<key>', …], and appends Skill to tools (not to allowedTools).
  3. Removes the directory when the run ends, however it ends.
  • One directory per run. There's no shared root, no watcher and no prune. Detaching a skill from a variation takes effect on the next run, and concurrent runs never share files.
  • Fail closed. If the reconcile isn't ok (no store, store not yet initialized, a skill withheld by verification, or an absent pin), the run throws before query() and names each skill.
  • No-skills path. A variation with no skills runs exactly as with the core handler, and needs no store.
  • Telemetry. The tool hooks are installed whenever Skill is in tools, so a Skill call gets an execute_tool Skill span.
  • Stage boundary. Core reaches this code only through a package-internal prepare hook on createClaudeAgentsHandlerInternal, which isn't part of the public factory options (§0.3).
  • Packaging. ./experimental is an exports subpath plus a typesVersions entry and a second tsup entry. It typechecks under node10, node16 and bundler, and the integration test's manifest check now covers it.
  • Options. ClaudeAgentsSkillsHandlerOptions is ContentCaptureOptions & ClaudeAgentsHostOptions & { skillsTempDir? }. ClaudeAgentsHostOptions is now exported from both the root and /experimental, which clears the ae-forgotten-export warnings in both claude-agents API reports.
  • Not covered: skills through claudeAgents(), claudeGraph() and toClaudeAgents(). They take the host options (feat(claude-agents)!: accept trusted cwd and settingSources options, and load no host settings by default #135) but build the core handler, so a variation run through them doesn't get its skills. The README says so.

Tests: a new skills.test.ts with a mocked query() and a real store, filesystem and OTel provider, plus the usage row in sdk-usage.test.ts. Lint, typecheck/api:check and all 217 claude-agents tests pass locally.

Not verified yet: a live run where a model actually loads the skill, with Skill in tools but not in allowedTools. It's needed before GA. The spike key had no credits. The spike also showed that the handler returns the CLI's "Credit balance is too low" result as a successful output; that's tracked separately.

Spec: TESTING.md §2.x "Claude Agents Skills Integration" and A.21, in launchdarkly/ai-sdks-monorepo#59. Its EXPERIMENTAL.md row stays at Draft until this releases. Python twin: launchdarkly/python-ai-sdk#208, merged.

🤖 Generated with Claude Code


Note

Overview
Adds an experimental @launchdarkly/ai-claude-agents/experimental entry (dual tsup build, exports/typesVersions, API Extractor report) exposing createClaudeAgentsSkillsHandler, which materializes a variation’s LaunchDarkly skills into a per-run temp plugin directory and passes them to the Claude Agent SDK via local plugins, qualified skills, and the built-in Skill tool.

The core handler gains a package-internal prepare hook on createClaudeAgentsHandlerInternal (invoke and stream): optional extra query() options, appended built-ins, and finally cleanup. Variations with no skills behave like createClaudeAgentsHandler; if skills cannot be written (missing/uninitialized store, verification failure, bad pin), the run fails before query(). ClaudeAgentsHostOptions is now a documented public type on the main entry; claudeAgents / claudeGraph / toClaudeAgents are unchanged.

Docs cover setup with setSkillStore, plus a large skills.test.ts suite and packaging/integration checks for the experimental subpath.

Reviewed by Cursor Bugbot for commit 3fd62f3. Bugbot is set up for automated code reviews on this repo. Configure here.

XieX and others added 2 commits October 14, 2026 13:41
…tions

`createClaudeAgentsHandler({ cwd, settingSources })` passes each option
given to every `query()` call, on the blocking and streaming paths. They
configure the host process the Agent SDK starts, so they come from the
application: they are spread after `model.parameters`, which already drops
both, and a config can neither set nor override them. An option not given
is not sent, so defaults are unchanged; `settingSources: []` is sent as an
empty list, which is the Agent SDK's isolation setting.

The README now says that, with `settingSources` unset, a run loads the
host's user, project and local Claude Code settings, including their
plugins, skills, hooks and permission rules. Measured on Agent SDK 0.3.220,
an unconfigured handler on a developer machine reported that machine's
plugins and 26 skills in its `init` message.

Also pins the existing `tools: []` rule with tests: a config with no
native tools sends an explicit empty list (every built-in off), and
user-defined tools stay reachable through MCP.

Spec: TESTING.md §2.x "Claude Agents Handler — host options and built-in
tool set" and A.17 in launchdarkly/ai-sdks-monorepo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…riation

`createClaudeAgentsHandler` never handed a config's `skills` to the model,
and its `tools: []` removes the `Skill` tool the CLI loads skills through.
Measured on Agent SDK 0.3.220: skills are discovered either way, but no
`skills` option brings `Skill` back while `tools` is `[]`.

`createClaudeAgentsSkillsHandler` from the new
`@launchdarkly/ai-claude-agents/experimental` entry point fixes that. On
every invoke and stream it writes `skillRefs(config)` with `writeSkills`,
through the store configured with `setSkillStore`, into a new directory
under `skillsTempDir`, and passes it to `query()` as a local plugin
(`launchdarkly`), with only those skills enabled (`launchdarkly:<key>`)
and `Skill` added to `tools`. The directory is removed when the run ends,
on every path. Each run owns its directory, so there is no shared root,
no watcher and no prune, and detaching a skill takes effect on the next
run.

It fails closed: a reconcile that is not `ok` (no store, store not yet
initialized, a skill withheld by verification, an absent pin) throws
before `query()`, naming each skill. A variation with no skills runs
exactly as the core handler and needs no store.

The handler gains a package-internal `prepare` hook on
`createClaudeAgentsHandlerInternal`; it is not part of the public
factory's options, so core names nothing experimental. Tool hooks are now
installed when `Skill` is the only tool, so a `Skill` call gets an
`execute_tool Skill` span.

Not covered: `claudeAgents()`, `claudeGraph()` and `toClaudeAgents()`.

Spec: TESTING.md §2.x "Claude Agents Skills Integration" and A.17 in
launchdarkly/ai-sdks-monorepo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
XieX and others added 6 commits October 19, 2026 11:26
…t-options

# Conflicts:
#	packages/claude-agents/src/handler.ts
…p opts in

`settingSources` not given now sends `[]` instead of leaving the Agent
SDK default, which loads every source: user, project and local settings,
including the host's plugins, skills, hooks, permission rules and
`CLAUDE.md` (AIC-3584 saw 22 of 52 requests carry a developer's
`~/.claude/CLAUDE.md`). Applications that want host settings name them,
e.g. `settingSources: ['user', 'project', 'local']`. `cwd` is still sent
only when given.

`[]` still lets the SDK read the managed-settings policy tier; the README
and agents.md say so.

Requested in review of launchdarkly/ai-sdks-monorepo#59.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…skills

# Conflicts:
#	packages/claude-agents/src/handler.ts
Addresses review on #135. `toClaudeAgents` still sent no `settingSources`,
so every Anthropic node in a native graph, root and sub-agents, loaded the
host's user, project and local settings: the AIC-3584 leak the handler no
longer has. It now takes `cwd` and `settingSources` in `opts` and sends
`hostQueryOptions(opts)` to every node's `query()`, after that node's
`model.parameters`, so it defaults to `[]` like the handler.

`claudeAgents()` and `claudeGraph()` already defaulted to `[]`, but had no
way to opt back in. Both now accept `cwd` and `settingSources` and build
the handler with them; neither reaches `config()` or `graph()`.

The README gains a migration note: `[]` also drops settings-file `env`
(`ANTHROPIC_BASE_URL`, `CLAUDE_CODE_USE_BEDROCK` / `_VERTEX`, proxy
variables), `apiKeyHelper` and `model`, so a Bedrock, Vertex or gateway
deployment configured through `~/.claude/settings.json` fails to
authenticate until it sets those in the process environment or passes
`settingSources: ['user']`.

Tests assert the `[]` default and the opt-in through `claudeAgents`,
`claudeGraph` and `toClaudeAgents` (both graph nodes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jeffdupont pushed a commit that referenced this pull request Oct 23, 2026
…and load no host settings by default (#135)

`createClaudeAgentsHandler({ cwd, settingSources })` takes two trusted,
app-supplied options and passes each one given to every `query()` call,
on both the blocking and the streaming path.

- **Configs can't set them.** They configure the host process the Agent
SDK starts, so they're spread with the handler's own options, after
`model.parameters`. The allowlist in `model-parameters.ts` already drops
both, so a config can neither set nor override them.
- **Host settings load only on opt-in (breaking).** `settingSources` not
given now sends `[]` rather than leaving the Agent SDK default, which
loads every source (AIC-3584). Apps that want host settings pass
`settingSources: ['user', 'project', 'local']`. `cwd` not given still
isn't sent. `[]` doesn't stop the SDK from reading the managed-settings
policy tier.
- **Why.** On Agent SDK 0.3.220, a handler that left `settingSources`
unset, on a developer machine, reported that machine's own plugins and
26 skills in its `init` message. The README documents the opt-in.
- **The `tools: []` rule is pinned with tests.** A config with no native
tools sends an explicit `[]`, which turns every built-in off, and
user-defined tools stay reachable through MCP. Python omits the key
today; the spec now settles on `[]` for both languages, and the Python
fix will follow as `fix(claude-agents)!`.

**No new exported names.** The option type is inline in the signature,
so the package root's approved name list doesn't change.

**Spec:** TESTING.md §2.x "Claude Agents Handler — host options and
built-in tool set" and A.18, in launchdarkly/ai-sdks-monorepo#59.

**Stacked on top of this:** the experimental skills integration,
#136.

**CI on `main` is already red:** the client package's root-export
snapshot is missing `makeGraphTrackData` (#101). That failure shows up
here too, and is being fixed separately.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **Breaking:** Runs no longer load the host machine’s Claude Code
settings unless the app opts in. The handler always passes
`settingSources: []` to the Agent SDK (instead of leaving the SDK
default that loads user/project/local plugins, skills, hooks,
`CLAUDE.md`, and settings-file `env`). Deployments that relied on
`~/.claude/settings.json` for Bedrock/Vertex/gateway auth need process
env vars or `settingSources: ['user']` (documented in the README
migration section).
> 
> **Trusted host options** — `cwd` and `settingSources` are added to
`createClaudeAgentsHandler`, `claudeAgents`, `claudeGraph`, and
`toClaudeAgents` via shared `hostQueryOptions()`. They are applied on
every `query()` call (invoke, stream, and all graph nodes) **after**
`model.parameters`, so flag config cannot set or override them.
> 
> **Built-in tools** — When no `NativeTool` sentinels are attached, the
handler sends **`tools: []`** so Claude Code built-ins stay off;
user-defined tools remain available through MCP.
> 
> Tests cover defaults, forwarding, config isolation, and graph paths.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a829919. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from xie/claude-agents-host-options to main October 23, 2026 16:24
@XieX
XieX requested a review from jeffdupont October 23, 2026 17:36
@XieX
XieX marked this pull request as ready for review October 23, 2026 17:36
XieX and others added 2 commits October 23, 2026 13:39
# Conflicts:
#	packages/claude-agents/src/__tests__/sdk-usage.test.ts
#	packages/claude-agents/src/handler.ts
…al entry point

`main` now commits an API Extractor report for every public entry point
(#153), and a new entry point adds its config and report in the same PR.
This PR adds `@launchdarkly/ai-claude-agents/experimental`, so it gets
`api-extractor.experimental.json` and
`etc/ai-claude-agents-experimental.api.md`, and the package's `api:check` /
`api:update` run both configs, as the client package's do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@XieX
XieX requested a review from a team as a code owner October 23, 2026 17:40
jeffdupont and others added 2 commits October 23, 2026 12:16
…p parent first

ClaudeAgentsSkillsHandlerOptions is now ContentCaptureOptions & ClaudeAgentsHostOptions plus
skillsTempDir, with ClaudeAgentsHostOptions exported from the root and experimental entries, so
neither API report carries a forgotten export. realpath now runs on the parent before mkdtemp,
so a realpath failure cannot leak the new directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jeffdupont

Copy link
Copy Markdown
Contributor

Merged main in and pushed 667abc0:

  • ClaudeAgentsSkillsHandlerOptions is spelled out as ContentCaptureOptions & ClaudeAgentsHostOptions & { skillsTempDir? }. ClaudeAgentsHostOptions is now exported from both the root and /experimental, which clears the ae-forgotten-export warnings in both claude-agents reports, including the one main already had.
  • realpath now runs on the parent before mkdtemp, so a realpath failure can no longer leak the directory.
  • Removed the stale stacked-on-feat(claude-agents)!: accept trusted cwd and settingSources options, and load no host settings by default #135 and red-main notes from the body.

Locally, lint, typecheck/api:check and claude-agents tests pass (217). This needs a human reviewer. It still needs one live run with Skill in tools but not in allowedTools before GA. Python twin: python-ai-sdk#208.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
XieX added a commit to launchdarkly/python-ai-sdk that referenced this pull request Oct 23, 2026
…riation (#208)

**Stacked on #201 (host options and built-in
tool set).** Review that one first. This PR's base is its branch.

A skill attached to an agent-mode variation never reached a model run
through `create_claude_agents_handler`. #201's `tools=[]` removes the
`Skill` tool, and nothing hands the CLI the skill files.

`create_claude_agents_skills_handler(*, capture_content=None, cwd=None,
setting_sources=None, skills_temp_dir=None)`, from the new
`launchdarkly_ai_claude_agents.experimental.skills` module, fixes that.
On every invoke and stream it:
1. Writes `skill_refs(config)` with `write_skills`, through the store
configured with `set_skill_store`, into
`mkdtemp(<skills_temp_dir>/ld-skills-)/launchdarkly/skills`. Pinned
versions are honored.
2. Passes `plugins=[{"type": "local", "path": <dir>/launchdarkly}]` and
`skills=["launchdarkly:<key>", …]`, and appends `Skill` to `tools` (not
to `allowed_tools`).
3. Removes the directory when the run ends, however it ends.

- **One directory per run.** There's no shared root, no watcher and no
prune. Detaching a skill from a variation takes effect on the next run,
and concurrent runs never share files.
- **Fail closed.** If the reconcile isn't `ok` (no store, store not yet
initialized, a skill withheld by verification, or an absent pin), the
run raises before `query()` and names each skill.
- **No-skills path.** A variation with no skills sends exactly the core
handler's options and needs no store.
- **`setting_sources` is never `None`.** With `skills` set and
`setting_sources=None`, `claude-agent-sdk` sends
`--setting-sources=user,project`. The core handler already always sends
it (`[]` unless given), and this handler inherits that.
- **Telemetry.** The tool hooks are installed whenever `Skill` is in
`tools`, so a `Skill` call gets an `execute_tool Skill` span, with no
`$ld:ai:tool_call` event.
- **Stage boundary.** Core reaches this code only through a private
`_prepare` keyword argument on `_create_claude_agents_handler`. It is
not an option of the public factory. The package root doesn't import the
experimental module, and a test checks that in a fresh interpreter.
- **Not covered:** `claude_agents()`, `claude_graph()` and
`to_claude_agents()`. The README says so.

## Differences from TypeScript

- **No `skipMcpDiscovery`.** `claude-agent-sdk`'s `SdkPluginConfig` has
no such key, so the plugin entry carries only `type` and `path`. The
directory holds no MCP config, so there is nothing to discover.
- **Synchronous removal.** The directory is removed with a synchronous
`shutil.rmtree`, so a cancellation during teardown can't interrupt it. A
failed removal is logged and doesn't replace the run's outcome.

## Tests

- A new `test_skills.py` (49 tests) uses a mocked `query()` with a real
`InMemorySkillStore`, a real filesystem (`tmp_path` as
`skills_temp_dir`) and a real OTel provider. Each spec bullet runs on
both invoke and stream where it applies.
- `test_sdk_usage.py` adds the
`experimental.createClaudeAgentsSkillsHandler` row.
- Mutation-checked: making the cleanup a no-op fails 9 tests. Dropping
`Skill` from the hook-install condition fails the span tests.

Checks:
- `uv run pytest`: everything passes except `opencode-agents`
`test_a_fake_opencode_is_started_spoken_to_and_stopped`, the macOS
`/private/var` temp-path failure already on `main`.
- `make lint`, `make format-check` and `make typecheck` are clean.

## Follow-ups in the monorepo

- The A.18 Python rows and an `EXPERIMENTAL.md` row for Python.
-
`api-surface/python/launchdarkly_ai_claude_agents.experimental.skills.txt`.
- `python` on `experimental.createClaudeAgentsSkillsHandler` in
`usage-ids.txt`.

**Not verified yet:** a live run where a model actually loads the skill.

**Spec:** TESTING.md §2.x "Claude Agents Skills Integration" and A.18,
in launchdarkly/ai-sdks-monorepo#59. The TypeScript side is
launchdarkly/js-ai-sdk#136.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Adds **experimental** `create_claude_agents_skills_handler()` under
`launchdarkly_ai_claude_agents.experimental.skills` so Claude Agent runs
get the **skills attached to their AI Config variation**. The core
`create_claude_agents_handler()` path is unchanged and still ignores
variation skills.
> 
> On each invoke/stream with skills, the handler writes skill files from
`set_skill_store` into a **per-run temp directory**, passes it to
`query()` as a local `launchdarkly` plugin with qualified
`launchdarkly:<key>` skill names, enables the **`Skill`** built-in (via
new internal `_prepare` / `QueryPreparation` on
`_create_claude_agents_handler`), and **removes the directory** in a
final cleanup on success, failure, cancel, or stream abandonment.
Reconcile failures **fail closed** before `query()`; empty skills behave
like the core handler.
> 
> Docs, API report, usage tracking for
`experimental.createClaudeAgentsSkillsHandler`, and a large
`test_skills.py` suite cover materialization, concurrency, telemetry
(`execute_tool Skill`), and that `claude_agents` / graph helpers are
**not** wired yet.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
ac743e0. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Jeff Dupont <jdupont@launchdarkly.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants