Skip to content

docs(skills): update curation workflow #322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gildesmarais merged 1 commit into from
Sep 1, 2026
Merged

docs(skills): update curation workflow #322

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 29 additions & 63 deletions .agents/skills/html2rss-config/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,87 +1,53 @@
---
name: html2rss-config
description: >-
Create or repair curated html2rss YAML feed configs in this repo (lib/html2rss/configs/),
including directory.topics, directory.title, selectors, Faraday vs Botasaurus triage, RSS probe-before-write,
and the AGENTS.md quality gate. Use when adding a new feed config, fixing a broken/zero-item
config, tightening selectors, diagnosing fetch failures, or shipping a user-requested batch
of configs (still one quality loop each). Do not use for html2rss gem core, html2rss-web,
or docs-only work.
Create or repair curated html2rss YAML configs in this repo. Router only —
quality gate and defaults live in AGENTS.md; mode deltas in reference/.
---

# html2rss-config

Thin workflow skill for **one config quality loop at a time** (a multi-config PR is OK only when the user asks). Quality-gate SSOT: repo root [`AGENTS.md`](../../../AGENTS.md). Do not duplicate that gate here. Campaign traps: [reference/pitfalls.md](reference/pitfalls.md). Batch pipeline (N=1 and N>1): [reference/batch.md](reference/batch.md).
One config quality loop at a time (multi-config PR only when the user asks).

## Modes
| Doc | Role |
| ------------------------------------------------ | ---------------------------------------------------- |
| [`AGENTS.md`](../../../AGENTS.md) | Quality gate, surface/selector defaults, MCP summary |
| [curation-verbs.md](reference/curation-verbs.md) | CLI/MCP verb table + stale-catalog fix |
| [pitfalls.md](reference/pitfalls.md) | Invariants (batch campaigns or quality issues only) |

| Mode | When | Reference |
| -------- | -------------------------------------------------------- | ------------------------------------------ |
| `new` | Add a YAML under `lib/html2rss/configs/<domain>/` | [reference/new.md](reference/new.md) |
| `repair` | Fix existing config (zero items, fetch fail, noisy feed) | [reference/repair.md](reference/repair.md) |
## Modes

Pick mode from the user ask. Grow later with more modes/references; keep this file short.
| Mode | When | Reference |
| -------- | -------------------------------------------------- | -------------------------------- |
| `new` | Add YAML under `lib/html2rss/configs/<domain>/` | [new.md](reference/new.md) |
| `repair` | Zero items, fetch fail, noisy feed | [repair.md](reference/repair.md) |
| `expand` | Batch-add across topics | [batch.md](reference/batch.md) |

## Before any write

1. Read [`AGENTS.md`](../../../AGENTS.md) (surface selection, selectors, drop rules).
2. Confirm canonical URL (`curl -I -L`); prefer **registrable-domain** folder. Watch for HTTPS→HTTP downgrades (Faraday will refuse).
3. Assign `directory.topics` (1–2) — see [reference/topics.md](reference/topics.md).
4. Write `directory.title`, optional `directory.summary`, and mirror `channel.title` — see [reference/catalog.md](reference/catalog.md).
5. Probe **that exact surface** with `scripts/probe_rss`. Exit `3` = first-party feed → drop/defer unless curated value is clearly higher. See [pitfalls.md](reference/pitfalls.md).

## Tool order

1. **user-html2rss MCP** — `capture_config` / `scrape_url` / `inspect_url` / `validate_config` when discovery works.
2. Else **core CLI** from PATH or sibling `../html2rss` — `scripts/check_config` resolves this (or raw `html2rss` / `bundle exec exe/html2rss`).
3. **Botasaurus** when Faraday returns zero items or scheme/redirect blocks Faraday: `BOTASAURUS_SCRAPER_URL=http://localhost:4010` (health: `/health`). `wait_timeout_seconds` **≤ 30** (work budget; total scrape wall 45s default).
4. **Chrome MCP** only if Faraday + Botasaurus fail or the item boundary is unclear. Report Chrome outage in handoff if unavailable.

If MCP discovery fails or the MCP process lacks `BOTASAURUS_SCRAPER_URL`, **skip MCP** and go straight to the CLI — do not burn the timebox retrying discovery.
Follow [`AGENTS.md`](../../../AGENTS.md) defaults. Then: canonical URL (`curl -I -L`), [catalog](reference/catalog.md) metadata (includes topics), `scripts/probe_rss` on the **exact** `channel.url` (exit `3` → defer/drop).

## Fast path (quick)
## Tools

Soft budget: one tight loop per site (~3–4 minutes of wall effort). Faraday → Botasaurus → Chrome. If still zero/noisy → **stop**, report drop/defer with evidence, unless the user says keep going.
MCP/CLI verbs → [curation-verbs.md](reference/curation-verbs.md) (journeys, envelope, catalog-mismatch fix). Quality gate → AGENTS.md.

Minimal selectors first: `items`, `title`, `url`. Omit brittle optional fields. Set `enhance: false` when chrome leaks in. Prefer nested title / `aria-label` over whole-card text.
**MCP broken but CLI works?** Treat as stale Cursor tool catalog, not bad selectors — use `html2rss inspect|recon|capture|test|apply|scrape` or scripts below until the catalog shows bare verb names.

## Scripts

Run from repo root. Prefer these over ad‑hoc CLI glue:
Run from repo root:

| Script | Purpose |
| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`scripts/batch_recon`](scripts/batch_recon) | Parallel redirect + RSS + HTML cache → `BUILD`/`DEFER`/`DROP` ledger (`--file` TSV or URL args). |
| [`scripts/analyze_html`](scripts/analyze_html) | Selector hints from cached HTML / ledger (`--from-ledger`). No network. |
| [`scripts/probe_rss`](scripts/probe_rss) | First-party RSS probe (HTML `rel=alternate` then path guesses). Exit `0` = none; `3` = found (consider drop). Under `set -e`, check `$?` — do not treat `3` as failure. |
| [`scripts/check_config`](scripts/check_config) | `validate` + `feed` (fail on 0 items); optional `--fetch` / `--botasaurus`. Resolves CLI via PATH or sibling `../html2rss`. |
| [`scripts/register_botasaurus`](scripts/register_botasaurus) | Idempotent sorted add to `spec/support/botasaurus_fetch_configs.rb`. |
| Script | Purpose |
| ---------------------------------------------------- | ------------------------------------------------------------ |
| [`add_config`](scripts/add_config) | Automated create (single URL or `--file`) |
| [`batch_recon`](scripts/batch_recon) | Parallel recon → BUILD/DEFER/DROP ledger |
| [`analyze_html`](scripts/analyze_html) | Selector hints from cached HTML / ledger |
| [`probe_rss`](scripts/probe_rss) | Native RSS via `recon` (exit `3` = found) |
| [`check_config`](scripts/check_config) | CLI `validate` + `test`; optional `--fetch` / `--botasaurus` |
| [`register_botasaurus`](scripts/register_botasaurus) | Register Botasaurus-backed configs for fetch specs |

Examples:

```bash
.agents/skills/html2rss-config/scripts/batch_recon --cache-dir tmp/html2rss-recon --file candidates.tsv
.agents/skills/html2rss-config/scripts/analyze_html --from-ledger tmp/html2rss-recon/ledger.tsv
.agents/skills/html2rss-config/scripts/probe_rss 'https://example.com/news/'
.agents/skills/html2rss-config/scripts/check_config domain/file.yml
.agents/skills/html2rss-config/scripts/check_config domain/file.yml --fetch --botasaurus
.agents/skills/html2rss-config/scripts/register_botasaurus domain/file.yml
```

## Done checklist

From AGENTS.md Quality Gate, in order:

1. Prefer `scripts/check_config <path>` (or raw `html2rss validate` + `feed`)
2. `make validate` (this repo) when touching shared support files or multiple configs
3. `make test` (non-fetch)
4. Focused fetch via `scripts/check_config … --fetch` or:
- Faraday: `bundle exec rspec --tag fetch --example 'domain/file.yml' spec/html2rss/configs_dynamic_spec.rb`
- Botasaurus: same with `BOTASAURUS_SCRAPER_URL=http://localhost:4010`
5. If `strategy: botasaurus` (or fetch only works via Botasaurus): `scripts/register_botasaurus domain/file.yml` — **required**.
Gem facades (`recon`, `capture`, `test`, `apply`): [`scripts/html2rss_api.rb`](scripts/html2rss_api.rb).

## Handoff

Report: mode, files changed, accepted vs dropped/deferred + why, topics, Faraday vs Botasaurus, Chrome MCP availability, commands + exit honesty, residual risks (selector drift, localization, Botasaurus dependence).

Do not push or open a PR unless the user asks. Commits only when the user asks (global commit rules).
Mode, files changed, accepted vs dropped/deferred, strategy, commands run, residual risks. No commit/PR unless asked.
85 changes: 15 additions & 70 deletions .agents/skills/html2rss-config/reference/batch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,92 +1,37 @@
# Batch pipeline

Default workflow for **N=1** and **N>1**. Pass a one-URL list for a single add — same scripts and phases.
N=1 and N>1. Gate: [AGENTS.md](../../../../AGENTS.md). Wall-time: [pitfalls.md](pitfalls.md).

Quality-gate SSOT: [AGENTS.md](../../../../AGENTS.md). Wall-time constraints: [pitfalls.md](pitfalls.md).
**Automated batch:** [new.md](new.md) `add_config --file candidates.tsv`.

## Prerequisites

- Repo root as cwd.
- Sibling `../html2rss` or `html2rss` on `PATH` for validate/feed.
- Optional: `BOTASAURUS_SCRAPER_URL=http://localhost:4010` when Faraday cannot fetch items.
- Ruby with stdlib + Nokogiri (Gemfile / system gem used by this repo).

## Phase 0 — parallel recon
## Phase 0 — recon

```bash
.agents/skills/html2rss-config/scripts/batch_recon \
--cache-dir tmp/html2rss-recon \
'https://example.com/news/'

.agents/skills/html2rss-config/scripts/batch_recon \
--cache-dir tmp/html2rss-recon \
--file candidates.tsv
--cache-dir tmp/html2rss-recon --file candidates.tsv
```

`candidates.tsv` lines: `URL`, or `slug\tURL`, or `slug\tregion\tURL` (`#` comments ok).

**Success:** `tmp/html2rss-recon/ledger.tsv` with `BUILD` / `DEFER` / `DROP`, plus one `.html` cache file per slug.

Verdicts:
Ledger: `tmp/html2rss-recon/ledger.tsv` + cached `.html` per slug. Dry run: `--dry-run`.

| Verdict | Meaning |
| ------- | --------------------------------------------------------------------------------------------------- |
| `BUILD` | No verified first-party feed; HTML cached for selectors (may still need Botasaurus if thin/blocked) |
| `DEFER` | Native RSS/Atom verified on the surface |
| `DROP` | Unreachable, HTTP error, HTTPS→HTTP downgrade, or error page |
| Verdict | Meaning |
| ------- | --------------------------------------------- |
| `BUILD` | No native feed; HTML cached for selectors |
| `DEFER` | Native RSS/Atom on surface |
| `DROP` | Unreachable, HTTP error, HTTPS→HTTP downgrade |

Dry plan (no network): `batch_recon --dry-run --file candidates.tsv`

## Phase 1 — selectors from cache
## Phase 1 — selectors

```bash
.agents/skills/html2rss-config/scripts/analyze_html \
--from-ledger tmp/html2rss-recon/ledger.tsv
```

Write YAMLs only for `BUILD` rows under `lib/html2rss/configs/<registrable-domain>/`. Use MCP/`check_config` only when cache analysis is insufficient. Botasaurus scrape: thin/empty/blocked HTML only — one retry max (`wait_timeout_seconds` ≤ 30, work budget), then drop.

YAML notes: [new.md](new.md). Topics: [topics.md](topics.md).

## Phase 2 — batched verification

```bash
# offline validate (example: sibling CLI)
html2rss validate lib/html2rss/configs/domain/*.yml

# parallel feed checks (Faraday group)
.agents/skills/html2rss-config/scripts/check_config domain/a.yml &
.agents/skills/html2rss-config/scripts/check_config domain/b.yml &
wait

# one rspec boot per fetch lane
bundle exec rspec --tag fetch \
--example 'domain/a.yml' \
--example 'domain/b.yml' \
spec/html2rss/configs_dynamic_spec.rb
```

## Phase 3 — campaign gate

```bash
make validate
make test
.agents/skills/html2rss-config/scripts/register_botasaurus domain/bot.yml # if needed
```

## N=1 shortcut
Write YAML for `BUILD` rows only. YAML shape: [new.md](new.md).

Same pipeline; one URL:
## Phase 2 — verify

```bash
.agents/skills/html2rss-config/scripts/batch_recon \
--cache-dir tmp/html2rss-recon \
'https://example.com/news/'
.agents/skills/html2rss-config/scripts/analyze_html \
--from-ledger tmp/html2rss-recon/ledger.tsv
# write YAML → check_config → focused fetch → make validate/test when done
```
Parallel `check_config`; one rspec boot per fetch lane — AGENTS.md § Quality Gate step 6.

## Repair campaigns

For broken configs, treat paths as the candidate list: diagnose with `check_config`, then the same Faraday→Botasaurus→drop loop. See [repair.md](repair.md).
Broken configs: [repair.md](repair.md) diagnose loop, then same verify gate.
6 changes: 6 additions & 0 deletions .agents/skills/html2rss-config/reference/catalog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,12 @@ Describe feed intent in `directory.title`, not the template URL.
| `who.int/news.yml` | World Health Organization — News |
| `apnews.com/hub.yml` | AP News — Top stories |

## Topics

Required on every config. Vocabulary SSOT: `Html2rss::Config::Validator::DIRECTORY_TOPICS`. Prefer **1–2**; do not invent strings.

`sports`, `energy`, `tech`, `science`, `news`, `entertainment`, `jobs`, `finance`, `security`, `travel`, `environment`, `consumer`, `civic`, `product`, `research`, `health`, `culture`, `education`, `rights`, `transport`

## Verification

```bash
Expand Down
58 changes: 58 additions & 0 deletions .agents/skills/html2rss-config/reference/curation-verbs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Curation verbs (CLI + MCP)

Seven verbs — no `_url` / `_config` suffixes. Upstream: `html2rss/CONTEXT.md` § Frozen contract.

| Verb | Job | CLI / MCP |
| --- | --- | --- |
| inspect | Diagnostics: URL, status, alternates, surface | `inspect` |
| recon | BUILD / DEFER / DROP verdict + native_feed | `recon` |
| capture | YAML draft → test → apply | `capture` |
| validate | Schema only | `validate` |
| test | Schema + live extraction | `test` |
| apply | Ship RSS (zero items = error) | `apply` |
| scrape | Articles now, one-shot | `scrape` |

Batch: `batch_inspect`, `batch_recon`, `batch_scrape`. CLI aliases: `feed` → `apply`, `auto` → `scrape`.

## Three journeys

| Goal | Path |
| --- | --- |
| Articles now | `scrape` (empty items can still be `ok` — follow `guidance`) |
| Durable YAML | `capture` → `test` → `apply` (side door: `validate` → `test` → `apply`) |
| Should we build? | `inspect` → `recon` when alternates or surface warrant it |

**inspect ≠ recon:** inspect is cheap diagnostics; recon adds verdict + native_feed. Follow envelope `next_step` and `guidance` — do not parse scrape text as a raw item array.

## Envelope (every MCP tool)

`ok`, `next_step` (bare verb or `done` / `read_runtime`), `guidance`, `payload`. Resources: `html2rss://schema`, `extractors`, `strategies`, `runtime` (`botasaurus_configured` boolean only).

## Strategy

| Tool | `strategy: auto` behavior |
| --- | --- |
| `scrape`, `capture` | Faraday → Botasaurus fallback — **do not** retry explicit `faraday` after `auto` |
| `inspect` | Faraday only (cheap); pin `botasaurus` when you need browser rendering |

## Configs-repo note

`capture` → `payload.yaml` is a **draft**. Still add `registry.id`, `directory.topics`, titles, and catalog fields per [catalog.md](catalog.md) before shipping.

## MCP in Cursor

Namespace: `user-html2rss`. Prefer MCP when the catalog matches the server; **CLI is the fallback authority** (`html2rss …` from `../html2rss`).

### Catalog mismatch (common after gem upgrade)

**Symptom:** Cursor lists `inspect_url`, `capture_config`, … but calls return `-32602 Invalid params`; new names like `inspect` are “tool not found”.

**Cause:** MCP server upgraded (bare verbs); **client `tools/list` cache stale**.

**Fix (in order):** Settings → MCP → disable/enable `html2rss` → Reload Window → quit Cursor fully. Confirm catalog shows `inspect`, `capture`, `test`, `apply`, `scrape` (not `*_url` / `*_config`).

**Until fixed:** use CLI or skill scripts (`html2rss_api.rb`) — same verbs, same pipeline.

### Botasaurus

`BOTASAURUS_SCRAPER_URL` must be on the **MCP process** (`mcp.json` env), not only your shell. If `next_step` is `read_runtime` or scrape stays empty, check `html2rss://runtime`.
Loading
Loading