agent-skill-forge

A spec-driven AI agent skill generator and verification pipeline for building, testing, repairing, and packaging production-ready agent skills.

Most AI agent skills are still created as prompt files and manually inspected. agent-skill-forge turns skill creation into an engineering pipeline:

Skill requirement → Skill spec → Generated skill package → Eval cases → Verification → Repair loop → Quality gate → Installable skill package.

It is designed for agent skill development, Claude-style skills, tool-calling workflows, LLM evaluation, and production-ready AI agent engineering.

Skill Requirement (YAML)
  -> Spec Generator
  -> Skill Package Generator
  -> Eval Case Generator
  -> Verification Runner
  -> Failure Analysis
  -> Repair Loop
  -> Quality Gate
  -> Verified Package (.zip)

The problem

Most agent skills are prompt artifacts: someone writes a SKILL.md, eyeballs it, and ships it. That does not scale for production. Skills need what any other production software has — specs, tool contracts, eval suites, failure analysis, repair loops, and release gates. Skill creation should be an engineering pipeline, not prompt-and-pray.

What it does

• Reads a skill requirement YAML (goals, non-goals, tools, safety requirements, acceptance criteria, quality gate thresholds).
• Generates a skill spec: an explicit behavioral contract (output contract, tool boundaries, failure behavior).
• Generates a skill package: SKILL.md, skill.manifest.json, tools.json, examples, README.
• Generates machine-checkable eval cases: golden, negative, edge, and tool-failure cases with expected statuses.
• Runs verification and computes pass rate, schema validity, unsupported claims, and tool error metrics.
• Analyzes failures into typed categories (missing confirmation, unhandled partial failure, unsupported claim, ...).
• Repairs the skill from the spec and re-verifies, comparing before/after metrics.
• Packages the skill into a release zip — only if the quality gate passes.
• Emits a static HTML report, a machine-readable summary.json, structured verification events, and replay artifacts for every failed case.

What can I use this for?

Use agent-skill-forge if you want to:

• generate AI agent skills from structured requirements
• create Claude-style skills with clear tool contracts
• build eval cases for agent skills
• test agent skills before release
• package verified skills as installable artifacts
• add quality gates to LLM-powered agent workflows
• avoid prompt-and-pray skill development

Quickstart

npm install
npm run forge
# open the report:
#   runs/latest/reports/report.html

The default demo is fully offline: no API keys, no paid providers, no external services.

To watch the failure → repair → re-verify loop in action:

npm run forge:flaky
# report lands in runs/latest-flaky/reports/report.html

The flaky demo uses a model adapter that intentionally generates a flawed skill (missing confirmation rules, dropped partial_success handling, an unsupported "guarantees perfect results" claim). Verification fails the gate, the repair loop regenerates the affected files from the spec, and the rerun passes.

Example input

examples/calendar-scheduling-skill.yaml:

name: calendar-scheduling-skill
version: 1.0.0
description: Help an agent schedule calendar events safely.

goals:
  - Check user availability
  - Propose meeting slots
  - Create calendar events after confirmation

non_goals:
  - Do not delete existing events
  - Do not create events without user confirmation

tools:
  - name: check_availability
    type: read
    description: Check user calendar availability.
  - name: create_event
    type: write
    description: Create a calendar event after explicit confirmation.

safety_requirements:
  - Require user confirmation before write operations
  - Use idempotency keys for event creation
  - Respect provider rate limits

quality_gate:
  min_pass_rate: 0.90
  min_schema_valid_rate: 1.00
  max_unsupported_claim_rate: 0.02
  max_tool_error_rate: 0.05

Two more examples ship with the repo: email-triage-skill.yaml (classification, privacy rules, confirmation-gated send) and codebase-understanding-skill.yaml (source-grounded answers with citations, works with any codebase context provider).

Example output

runs/run-<id>/                       # canonical run directory (mirrored to runs/latest)
├── input/skill-request.yaml
├── workspace/
│   ├── generated-skill/             # SKILL.md, manifest, tools.json, eval-cases.json, ...
│   └── repaired-skill-attempt-1/    # only when a repair ran
├── artifacts/
│   ├── normalized-requirement.json
│   ├── skill-spec.json
│   ├── verification-summary.json
│   ├── verification-events.jsonl
│   ├── failure-analysis.json
│   └── repair-plan-attempt-1.json
├── replay-artifacts/                # one JSON per failed eval case
├── reports/
│   ├── report.html                  # full pipeline report
│   ├── verification-report.html
│   └── summary.json
└── dist/
    └── calendar-scheduling-skill-1.0.0.zip

The release zip contains the verified skill package plus the verification report, metrics summary, and a package-info.json provenance record (source run, model, verifier, gate evidence).

CLI

npm run forge -- \
  --input examples/calendar-scheduling-skill.yaml \
  --model mock \
  --verifier local-demo \
  --max-repair-attempts 2 \
  --output runs/latest

Flag	Default	Purpose
--input	calendar example	Skill requirement YAML
--model	mock	mock, mock-flaky, anthropic-stub, openai-stub, ollama-stub
--verifier	local-demo	local-demo, local-demo-flaky, verification-template
--max-repair-attempts	2	Repair loop budget when the gate fails
--output	runs/latest	Directory receiving a mirror of the run
--allow-failed-package	off	UNSAFE: package despite a failed gate (debugging only)

Exit codes: 0 gate passed, 1 gate failed, 2 configuration/pipeline error.

How it relates to agent-skill-verification-template

Verification is pluggable. The default local-demo verifier is offline and self-contained. An optional verification-template adapter can bridge to a local checkout of agent-skill-verification-template (a generic eval / observability / quality gate framework) — see docs/verification-integration.md for what the bridge does today and what is roadmap. The default demo never requires that repo.

Generic by design

agent-skill-forge is domain-agnostic and provider-agnostic. It generates and verifies calendar, email, codebase, customer support, security triage, or any other agent skill from the same requirement schema. It has no dependency on any specific codebase context provider or agent runtime.

Model adapters

Adapter	Status	Behavior
mock	working	Offline, deterministic; generates a valid spec, package, and eval cases
mock-flaky	working	Deterministically injects realistic flaws to demonstrate the repair loop
anthropic-stub	stub	No API call; fails with a clear message
openai-stub	stub	No API call; fails with a clear message
ollama-stub	stub	No API call; connection sketch in docs/model-adapters.md

All adapters implement the same ModelAdapter interface. Different models may generate different skills, but every output must pass the same verification gate.

Quality gates

A skill package is only released when all four hold:

• Pass rate ≥ min_pass_rate — share of eval cases whose expected behavior the package supports
• Schema valid rate ≥ min_schema_valid_rate — share of cases whose expected status is part of the declared output contract
• Unsupported claim rate ≤ max_unsupported_claim_rate — cases flagged for absolute claims ("guarantees", "never fails")
• Tool error rate ≤ max_tool_error_rate — failed tool-failure-handling cases

Any failed static check (missing file, invalid manifest, unconfirmed write tool) also fails the gate. A failed gate produces a failure summary and no release package by default.

Limitations (honest edition)

• The MVP uses a deterministic mock model by default; API provider adapters are stubs.
• The local-demo verifier is demo-level: it checks package structure and declared behavior against eval expectations; it does not execute the skill against a live model.
• The repair loop is simple: it regenerates affected files from the spec rather than performing semantic patching.
• The verification-template adapter is a partial bridge (reads that framework's report; does not yet export eval cases into it).
• Generated skills require human review before real production use.

Roadmap

• Real Anthropic / OpenAI / Ollama adapters
• Deeper integration with agent-skill-verification-template (eval case export, round-trip gating)
• MCP tool packaging and Claude Skill packaging examples
• Web UI over run history
• Model comparison matrix (same requirement, N models, one gate)
• OpenTelemetry traces for pipeline stages
• Richer semantic validators and a plugin system
• Skill registry with versioned, signed packages

Details in docs/roadmap.md.

Development

npm install
npm run build   # compile TypeScript to build/
npm test        # vitest suite
npm run forge   # offline demo pipeline
npm run clean   # remove build output, runs, and packaged zips

Reference docs: pipeline model · skill package format · verification integration · model adapters · roadmap

Guides: skill pipeline overview · skill generation · verification pipeline · quality gates · tool contracts · production practices · Claude Skill packaging (roadmap) · MCP integration (roadmap)

License

MIT