diff --git a/.agents/skills/html2rss-config/SKILL.md b/.agents/skills/html2rss-config/SKILL.md index 9081648f..fe6b6840 100644 --- a/.agents/skills/html2rss-config/SKILL.md +++ b/.agents/skills/html2rss-config/SKILL.md @@ -1,87 +1,53 @@ --- name: html2rss-config description: >- - Create or repair curated html2rss YAML feed configs in this repo (lib/html2rss/configs/), - including directory.topics, directory.title, selectors, Faraday vs Botasaurus triage, RSS probe-before-write, - and the AGENTS.md quality gate. Use when adding a new feed config, fixing a broken/zero-item - config, tightening selectors, diagnosing fetch failures, or shipping a user-requested batch - of configs (still one quality loop each). Do not use for html2rss gem core, html2rss-web, - or docs-only work. + Create or repair curated html2rss YAML configs in this repo. Router only — + quality gate and defaults live in AGENTS.md; mode deltas in reference/. --- # html2rss-config -Thin workflow skill for **one config quality loop at a time** (a multi-config PR is OK only when the user asks). Quality-gate SSOT: repo root [`AGENTS.md`](../../../AGENTS.md). Do not duplicate that gate here. Campaign traps: [reference/pitfalls.md](reference/pitfalls.md). Batch pipeline (N=1 and N>1): [reference/batch.md](reference/batch.md). +One config quality loop at a time (multi-config PR only when the user asks). -## Modes +| Doc | Role | +| ------------------------------------------------ | ---------------------------------------------------- | +| [`AGENTS.md`](../../../AGENTS.md) | Quality gate, surface/selector defaults, MCP summary | +| [curation-verbs.md](reference/curation-verbs.md) | CLI/MCP verb table + stale-catalog fix | +| [pitfalls.md](reference/pitfalls.md) | Invariants (batch campaigns or quality issues only) | -| Mode | When | Reference | -| -------- | -------------------------------------------------------- | ------------------------------------------ | -| `new` | Add a YAML under `lib/html2rss/configs//` | [reference/new.md](reference/new.md) | -| `repair` | Fix existing config (zero items, fetch fail, noisy feed) | [reference/repair.md](reference/repair.md) | +## Modes -Pick mode from the user ask. Grow later with more modes/references; keep this file short. +| Mode | When | Reference | +| -------- | -------------------------------------------------- | -------------------------------- | +| `new` | Add YAML under `lib/html2rss/configs//` | [new.md](reference/new.md) | +| `repair` | Zero items, fetch fail, noisy feed | [repair.md](reference/repair.md) | +| `expand` | Batch-add across topics | [batch.md](reference/batch.md) | ## Before any write -1. Read [`AGENTS.md`](../../../AGENTS.md) (surface selection, selectors, drop rules). -2. Confirm canonical URL (`curl -I -L`); prefer **registrable-domain** folder. Watch for HTTPS→HTTP downgrades (Faraday will refuse). -3. Assign `directory.topics` (1–2) — see [reference/topics.md](reference/topics.md). -4. Write `directory.title`, optional `directory.summary`, and mirror `channel.title` — see [reference/catalog.md](reference/catalog.md). -5. Probe **that exact surface** with `scripts/probe_rss`. Exit `3` = first-party feed → drop/defer unless curated value is clearly higher. See [pitfalls.md](reference/pitfalls.md). - -## Tool order - -1. **user-html2rss MCP** — `capture_config` / `scrape_url` / `inspect_url` / `validate_config` when discovery works. -2. Else **core CLI** from PATH or sibling `../html2rss` — `scripts/check_config` resolves this (or raw `html2rss` / `bundle exec exe/html2rss`). -3. **Botasaurus** when Faraday returns zero items or scheme/redirect blocks Faraday: `BOTASAURUS_SCRAPER_URL=http://localhost:4010` (health: `/health`). `wait_timeout_seconds` **≤ 30** (work budget; total scrape wall 45s default). -4. **Chrome MCP** only if Faraday + Botasaurus fail or the item boundary is unclear. Report Chrome outage in handoff if unavailable. - -If MCP discovery fails or the MCP process lacks `BOTASAURUS_SCRAPER_URL`, **skip MCP** and go straight to the CLI — do not burn the timebox retrying discovery. +Follow [`AGENTS.md`](../../../AGENTS.md) defaults. Then: canonical URL (`curl -I -L`), [catalog](reference/catalog.md) metadata (includes topics), `scripts/probe_rss` on the **exact** `channel.url` (exit `3` → defer/drop). -## Fast path (quick) +## Tools -Soft budget: one tight loop per site (~3–4 minutes of wall effort). Faraday → Botasaurus → Chrome. If still zero/noisy → **stop**, report drop/defer with evidence, unless the user says keep going. +MCP/CLI verbs → [curation-verbs.md](reference/curation-verbs.md) (journeys, envelope, catalog-mismatch fix). Quality gate → AGENTS.md. -Minimal selectors first: `items`, `title`, `url`. Omit brittle optional fields. Set `enhance: false` when chrome leaks in. Prefer nested title / `aria-label` over whole-card text. +**MCP broken but CLI works?** Treat as stale Cursor tool catalog, not bad selectors — use `html2rss inspect|recon|capture|test|apply|scrape` or scripts below until the catalog shows bare verb names. ## Scripts -Run from repo root. Prefer these over ad‑hoc CLI glue: +Run from repo root: -| Script | Purpose | -| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [`scripts/batch_recon`](scripts/batch_recon) | Parallel redirect + RSS + HTML cache → `BUILD`/`DEFER`/`DROP` ledger (`--file` TSV or URL args). | -| [`scripts/analyze_html`](scripts/analyze_html) | Selector hints from cached HTML / ledger (`--from-ledger`). No network. | -| [`scripts/probe_rss`](scripts/probe_rss) | First-party RSS probe (HTML `rel=alternate` then path guesses). Exit `0` = none; `3` = found (consider drop). Under `set -e`, check `$?` — do not treat `3` as failure. | -| [`scripts/check_config`](scripts/check_config) | `validate` + `feed` (fail on 0 items); optional `--fetch` / `--botasaurus`. Resolves CLI via PATH or sibling `../html2rss`. | -| [`scripts/register_botasaurus`](scripts/register_botasaurus) | Idempotent sorted add to `spec/support/botasaurus_fetch_configs.rb`. | +| Script | Purpose | +| ---------------------------------------------------- | ------------------------------------------------------------ | +| [`add_config`](scripts/add_config) | Automated create (single URL or `--file`) | +| [`batch_recon`](scripts/batch_recon) | Parallel recon → BUILD/DEFER/DROP ledger | +| [`analyze_html`](scripts/analyze_html) | Selector hints from cached HTML / ledger | +| [`probe_rss`](scripts/probe_rss) | Native RSS via `recon` (exit `3` = found) | +| [`check_config`](scripts/check_config) | CLI `validate` + `test`; optional `--fetch` / `--botasaurus` | +| [`register_botasaurus`](scripts/register_botasaurus) | Register Botasaurus-backed configs for fetch specs | -Examples: - -```bash -.agents/skills/html2rss-config/scripts/batch_recon --cache-dir tmp/html2rss-recon --file candidates.tsv -.agents/skills/html2rss-config/scripts/analyze_html --from-ledger tmp/html2rss-recon/ledger.tsv -.agents/skills/html2rss-config/scripts/probe_rss 'https://example.com/news/' -.agents/skills/html2rss-config/scripts/check_config domain/file.yml -.agents/skills/html2rss-config/scripts/check_config domain/file.yml --fetch --botasaurus -.agents/skills/html2rss-config/scripts/register_botasaurus domain/file.yml -``` - -## Done checklist - -From AGENTS.md Quality Gate, in order: - -1. Prefer `scripts/check_config ` (or raw `html2rss validate` + `feed`) -2. `make validate` (this repo) when touching shared support files or multiple configs -3. `make test` (non-fetch) -4. Focused fetch via `scripts/check_config … --fetch` or: - - Faraday: `bundle exec rspec --tag fetch --example 'domain/file.yml' spec/html2rss/configs_dynamic_spec.rb` - - Botasaurus: same with `BOTASAURUS_SCRAPER_URL=http://localhost:4010` -5. If `strategy: botasaurus` (or fetch only works via Botasaurus): `scripts/register_botasaurus domain/file.yml` — **required**. +Gem facades (`recon`, `capture`, `test`, `apply`): [`scripts/html2rss_api.rb`](scripts/html2rss_api.rb). ## Handoff -Report: mode, files changed, accepted vs dropped/deferred + why, topics, Faraday vs Botasaurus, Chrome MCP availability, commands + exit honesty, residual risks (selector drift, localization, Botasaurus dependence). - -Do not push or open a PR unless the user asks. Commits only when the user asks (global commit rules). +Mode, files changed, accepted vs dropped/deferred, strategy, commands run, residual risks. No commit/PR unless asked. diff --git a/.agents/skills/html2rss-config/reference/batch.md b/.agents/skills/html2rss-config/reference/batch.md index 08c6b9f6..10d043b2 100644 --- a/.agents/skills/html2rss-config/reference/batch.md +++ b/.agents/skills/html2rss-config/reference/batch.md @@ -1,92 +1,37 @@ # Batch pipeline -Default workflow for **N=1** and **N>1**. Pass a one-URL list for a single add — same scripts and phases. +N=1 and N>1. Gate: [AGENTS.md](../../../../AGENTS.md). Wall-time: [pitfalls.md](pitfalls.md). -Quality-gate SSOT: [AGENTS.md](../../../../AGENTS.md). Wall-time constraints: [pitfalls.md](pitfalls.md). +**Automated batch:** [new.md](new.md) `add_config --file candidates.tsv`. -## Prerequisites - -- Repo root as cwd. -- Sibling `../html2rss` or `html2rss` on `PATH` for validate/feed. -- Optional: `BOTASAURUS_SCRAPER_URL=http://localhost:4010` when Faraday cannot fetch items. -- Ruby with stdlib + Nokogiri (Gemfile / system gem used by this repo). - -## Phase 0 — parallel recon +## Phase 0 — recon ```bash .agents/skills/html2rss-config/scripts/batch_recon \ - --cache-dir tmp/html2rss-recon \ - 'https://example.com/news/' - -.agents/skills/html2rss-config/scripts/batch_recon \ - --cache-dir tmp/html2rss-recon \ - --file candidates.tsv + --cache-dir tmp/html2rss-recon --file candidates.tsv ``` -`candidates.tsv` lines: `URL`, or `slug\tURL`, or `slug\tregion\tURL` (`#` comments ok). - -**Success:** `tmp/html2rss-recon/ledger.tsv` with `BUILD` / `DEFER` / `DROP`, plus one `.html` cache file per slug. - -Verdicts: +Ledger: `tmp/html2rss-recon/ledger.tsv` + cached `.html` per slug. Dry run: `--dry-run`. -| Verdict | Meaning | -| ------- | --------------------------------------------------------------------------------------------------- | -| `BUILD` | No verified first-party feed; HTML cached for selectors (may still need Botasaurus if thin/blocked) | -| `DEFER` | Native RSS/Atom verified on the surface | -| `DROP` | Unreachable, HTTP error, HTTPS→HTTP downgrade, or error page | +| Verdict | Meaning | +| ------- | --------------------------------------------- | +| `BUILD` | No native feed; HTML cached for selectors | +| `DEFER` | Native RSS/Atom on surface | +| `DROP` | Unreachable, HTTP error, HTTPS→HTTP downgrade | -Dry plan (no network): `batch_recon --dry-run --file candidates.tsv` - -## Phase 1 — selectors from cache +## Phase 1 — selectors ```bash .agents/skills/html2rss-config/scripts/analyze_html \ --from-ledger tmp/html2rss-recon/ledger.tsv ``` -Write YAMLs only for `BUILD` rows under `lib/html2rss/configs//`. Use MCP/`check_config` only when cache analysis is insufficient. Botasaurus scrape: thin/empty/blocked HTML only — one retry max (`wait_timeout_seconds` ≤ 30, work budget), then drop. - -YAML notes: [new.md](new.md). Topics: [topics.md](topics.md). - -## Phase 2 — batched verification - -```bash -# offline validate (example: sibling CLI) -html2rss validate lib/html2rss/configs/domain/*.yml - -# parallel feed checks (Faraday group) -.agents/skills/html2rss-config/scripts/check_config domain/a.yml & -.agents/skills/html2rss-config/scripts/check_config domain/b.yml & -wait - -# one rspec boot per fetch lane -bundle exec rspec --tag fetch \ - --example 'domain/a.yml' \ - --example 'domain/b.yml' \ - spec/html2rss/configs_dynamic_spec.rb -``` - -## Phase 3 — campaign gate - -```bash -make validate -make test -.agents/skills/html2rss-config/scripts/register_botasaurus domain/bot.yml # if needed -``` - -## N=1 shortcut +Write YAML for `BUILD` rows only. YAML shape: [new.md](new.md). -Same pipeline; one URL: +## Phase 2 — verify -```bash -.agents/skills/html2rss-config/scripts/batch_recon \ - --cache-dir tmp/html2rss-recon \ - 'https://example.com/news/' -.agents/skills/html2rss-config/scripts/analyze_html \ - --from-ledger tmp/html2rss-recon/ledger.tsv -# write YAML → check_config → focused fetch → make validate/test when done -``` +Parallel `check_config`; one rspec boot per fetch lane — AGENTS.md § Quality Gate step 6. ## Repair campaigns -For broken configs, treat paths as the candidate list: diagnose with `check_config`, then the same Faraday→Botasaurus→drop loop. See [repair.md](repair.md). +Broken configs: [repair.md](repair.md) diagnose loop, then same verify gate. diff --git a/.agents/skills/html2rss-config/reference/catalog.md b/.agents/skills/html2rss-config/reference/catalog.md index a289b97c..d1db9711 100644 --- a/.agents/skills/html2rss-config/reference/catalog.md +++ b/.agents/skills/html2rss-config/reference/catalog.md @@ -41,6 +41,12 @@ Describe feed intent in `directory.title`, not the template URL. | `who.int/news.yml` | World Health Organization — News | | `apnews.com/hub.yml` | AP News — Top stories | +## Topics + +Required on every config. Vocabulary SSOT: `Html2rss::Config::Validator::DIRECTORY_TOPICS`. Prefer **1–2**; do not invent strings. + +`sports`, `energy`, `tech`, `science`, `news`, `entertainment`, `jobs`, `finance`, `security`, `travel`, `environment`, `consumer`, `civic`, `product`, `research`, `health`, `culture`, `education`, `rights`, `transport` + ## Verification ```bash diff --git a/.agents/skills/html2rss-config/reference/curation-verbs.md b/.agents/skills/html2rss-config/reference/curation-verbs.md new file mode 100644 index 00000000..2bf7a83b --- /dev/null +++ b/.agents/skills/html2rss-config/reference/curation-verbs.md @@ -0,0 +1,58 @@ +# Curation verbs (CLI + MCP) + +Seven verbs — no `_url` / `_config` suffixes. Upstream: `html2rss/CONTEXT.md` § Frozen contract. + +| Verb | Job | CLI / MCP | +| --- | --- | --- | +| inspect | Diagnostics: URL, status, alternates, surface | `inspect` | +| recon | BUILD / DEFER / DROP verdict + native_feed | `recon` | +| capture | YAML draft → test → apply | `capture` | +| validate | Schema only | `validate` | +| test | Schema + live extraction | `test` | +| apply | Ship RSS (zero items = error) | `apply` | +| scrape | Articles now, one-shot | `scrape` | + +Batch: `batch_inspect`, `batch_recon`, `batch_scrape`. CLI aliases: `feed` → `apply`, `auto` → `scrape`. + +## Three journeys + +| Goal | Path | +| --- | --- | +| Articles now | `scrape` (empty items can still be `ok` — follow `guidance`) | +| Durable YAML | `capture` → `test` → `apply` (side door: `validate` → `test` → `apply`) | +| Should we build? | `inspect` → `recon` when alternates or surface warrant it | + +**inspect ≠ recon:** inspect is cheap diagnostics; recon adds verdict + native_feed. Follow envelope `next_step` and `guidance` — do not parse scrape text as a raw item array. + +## Envelope (every MCP tool) + +`ok`, `next_step` (bare verb or `done` / `read_runtime`), `guidance`, `payload`. Resources: `html2rss://schema`, `extractors`, `strategies`, `runtime` (`botasaurus_configured` boolean only). + +## Strategy + +| Tool | `strategy: auto` behavior | +| --- | --- | +| `scrape`, `capture` | Faraday → Botasaurus fallback — **do not** retry explicit `faraday` after `auto` | +| `inspect` | Faraday only (cheap); pin `botasaurus` when you need browser rendering | + +## Configs-repo note + +`capture` → `payload.yaml` is a **draft**. Still add `registry.id`, `directory.topics`, titles, and catalog fields per [catalog.md](catalog.md) before shipping. + +## MCP in Cursor + +Namespace: `user-html2rss`. Prefer MCP when the catalog matches the server; **CLI is the fallback authority** (`html2rss …` from `../html2rss`). + +### Catalog mismatch (common after gem upgrade) + +**Symptom:** Cursor lists `inspect_url`, `capture_config`, … but calls return `-32602 Invalid params`; new names like `inspect` are “tool not found”. + +**Cause:** MCP server upgraded (bare verbs); **client `tools/list` cache stale**. + +**Fix (in order):** Settings → MCP → disable/enable `html2rss` → Reload Window → quit Cursor fully. Confirm catalog shows `inspect`, `capture`, `test`, `apply`, `scrape` (not `*_url` / `*_config`). + +**Until fixed:** use CLI or skill scripts (`html2rss_api.rb`) — same verbs, same pipeline. + +### Botasaurus + +`BOTASAURUS_SCRAPER_URL` must be on the **MCP process** (`mcp.json` env), not only your shell. If `next_step` is `read_runtime` or scrape stays empty, check `html2rss://runtime`. diff --git a/.agents/skills/html2rss-config/reference/new.md b/.agents/skills/html2rss-config/reference/new.md index ada7a4aa..0506dcdd 100644 --- a/.agents/skills/html2rss-config/reference/new.md +++ b/.agents/skills/html2rss-config/reference/new.md @@ -1,31 +1,30 @@ # Mode: new -Add one curated config. SSOT details: [AGENTS.md](../../../../AGENTS.md). +Add one curated config. Defaults and gate: [AGENTS.md](../../../../AGENTS.md). -## Steps +## Automated -1. Pick the cleanest list URL (newsroom / archive / category — not marketing homepage). Confirm with `curl -I -L` (canonical host; no HTTPS→HTTP downgrade unless you plan Botasaurus). -2. Confirm no useful first-party RSS **for that exact URL** (else drop/defer). Use `scripts/probe_rss`. Exit `3` = feed found. -3. Capture items via skill tool order (MCP → CLI → Botasaurus → Chrome). If `auto` is empty, still inspect HTML before assuming JS-only — see [pitfalls.md](pitfalls.md). -4. Write YAML under `lib/html2rss/configs//.yml`. -5. Run `scripts/check_config …` (and `--fetch` / `--botasaurus` as needed); verify real `` rows, not only the summary. `scripts/register_botasaurus` if Botasaurus-backed. -6. Handoff per skill. +```bash +.agents/skills/html2rss-config/scripts/add_config 'https://example.com/news/' --topics tech,news +``` + +Probes via `recon`, captures YAML via `capture` (draft — add catalog fields). Fallback: manual selectors. Gate: AGENTS.md. + +## Manual fallback + +When automation fails: tool order per [SKILL.md](../SKILL.md); write YAML under `lib/html2rss/configs//`; `scripts/check_config` (+ `--fetch` / `--botasaurus` as needed). ## YAML skeleton ```yaml # yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json directory: - topics: - - tech + topics: [tech] title: Example — News - summary: Short browse-card description (optional, max 160 characters). channel: url: https://example.com/news/ title: Example — News language: en - time_zone: UTC - ttl: 360 selectors: items: selector: "ARTICLE_CARD_OR_ANCHOR" @@ -37,25 +36,4 @@ selectors: extractor: href ``` -Notes: - -- Always include `directory.topics` (1–2) — see [topics.md](topics.md). -- Set `directory.title` and mirror it in `channel.title` for catalog and RSS output. -- Set `channel.language` when clear (`en` / `de` / `es` / …). Prefer a real region `time_zone` when obvious (`Africa/Johannesburg`, `Africa/Cairo`, …). -- Add `strategy: botasaurus` only when Faraday cannot produce items (or Faraday is blocked by scheme/redirect). Keep `wait_timeout_seconds` **≤ 30** (work budget; total scrape timeout defaults to 45s). -- Parameterized URLs need a `parameters:` block with `type: string` and `default`. -- Prefer item-local selectors; anchor on article URL path fragments when possible. - -## Ship bar - -Live `feed` shows repeated real articles, no nav/footer leakage, absolute URLs. Then repo `make validate` + `make test` + focused fetch. - -## Footnotes - -See [pitfalls.md](pitfalls.md) for full campaign traps. Short list: - -- **Folder name:** registrable domain only — avoid `www.` folders unless the host is uniquely `www`. -- **Archive size:** if the list HTML embeds a full multi-year archive, prefer `/latest` or paginated; note blast radius in handoff. -- **Paywall:** titles OK while bodies gated are shippable; mention risk in handoff. -- **MCP:** errored discovery or missing Botasaurus env → skip MCP, use CLI immediately. -- **Batch PR:** allowed when the user asks; still one probe→feed→fetch loop per config. +Metadata: [catalog.md](catalog.md). Ship bar: AGENTS.md Quality Gate. diff --git a/.agents/skills/html2rss-config/reference/pitfalls.md b/.agents/skills/html2rss-config/reference/pitfalls.md index 834342f2..b703af5d 100644 --- a/.agents/skills/html2rss-config/reference/pitfalls.md +++ b/.agents/skills/html2rss-config/reference/pitfalls.md @@ -1,106 +1,47 @@ -# Batch wall-time constraints +# Runtime pitfalls -Present constraints for config campaigns (N≥1). Quality-gate commands stay in [AGENTS.md](../../../../AGENTS.md). +Invariants for `new` / `repair` / batch campaigns. Quality gate: [AGENTS.md](../../../../AGENTS.md). Pipeline: [batch.md](batch.md). -## Recon +## Never guess URL paths -- Probe the **exact** intended `channel.url` (not a homepage stand-in). -- Prefer `scripts/batch_recon` over sequential `scripts/probe_rss` loops. -- Cache HTML once per candidate; author selectors from that cache. -- Treat first-party RSS on the surface as **DEFER** (no curated config unless curated value is explicit). -- Use `html2rss auto` only as optional discovery; never as proof a config is ready (AGENTS.md Auto-Source). +- Do **not** invent paths (`/news`, `/press-releases`, …) without a live HTTP 200. +- 404/403/DNS/SSL → drop or find a verified URL; do not commit guessed paths. +- Selectors must yield `items.count >= 1` via `Html2rss.apply` before shipping. -## Request strategy +## Semantic feed quality -- Faraday first. -- Botasaurus only when Faraday returns zero/blocked items and the browser shows a real list. -- Cap Botasaurus `wait_timeout_seconds` at **≤ 30** (post-boot work budget; total scrape wall defaults to 45s separately). -- One scrape retry on transient Botasaurus errors, then **DROP** with ledger evidence — no retry spirals. - -## Authoring - -- Soft budget ~3–4 minutes wall effort per BUILD site after recon. -- Minimal selectors first: `items`, `title`, `url`. Drop brittle optionals early. -- Set `enhance: false` when enhancement pulls chrome/nav/hero. - -## Verification batching - -- Offline: validate all new/changed YAML files in one pass. -- Feed checks: run several `scripts/check_config` jobs in parallel (grouped by strategy). -- Focused fetch: **one rspec boot per lane**, multiple `--example` flags: - -```bash -bundle exec rspec --tag fetch \ - --example 'domain/a.yml' \ - --example 'domain/b.yml' \ - spec/html2rss/configs_dynamic_spec.rb -``` - -Botasaurus lane: same command with `BOTASAURUS_SCRAPER_URL=http://localhost:4010`. - -- For campaigns: `make validate` and `make test` **once at the end**, not per config. -- Register every shipped Botasaurus config with `scripts/register_botasaurus`. - ---- - -# Runtime pitfalls (campaign harvest) - -Hard-won notes for `new` / `repair`. Prefer these over rediscovering the same traps. -SSOT for quality gate remains [AGENTS.md](../../../../AGENTS.md). Pipeline: [batch.md](batch.md). +- Reject feeds where all items share one URL; enforce link diversity. +- Reject generic titles ("Read more", "PDF", …) or titles under 4 characters. +- Resolve topics from `Html2rss::Config::Validator::DIRECTORY_TOPICS` at runtime. ## Probe the exact surface -- Run `scripts/probe_rss` / `scripts/batch_recon` on the **same URL** you will put in `channel.url`, not only the domain homepage. -- Example: AllAfrica homepage looked RSS-free; `https://allafrica.com/latest/` advertises a working RDF/RSS → **defer**. -- Exit `3` / ledger `DEFER` = feed found → drop/defer unless curated value is clearly higher than the native feed. - -## Coverage / gap campaigns (when the user asks) - -Still **one config quality loop at a time**. A single PR with many configs is fine if the user asked for it — do not skip probe → selectors → `check_config` → focused fetch per file. - -Prefer additions that: - -- lack usable first-party RSS -- are primary sources (IGO / regulator / think tank / dedicated newsroom list) -- avoid NA/EU-only shortlists when the ask is “global” or names a region (Africa, LATAM, …) +- Run `probe_rss` / `batch_recon` on the same URL as `channel.url`, not just the homepage. +- Exit `3` / ledger `DEFER` → native RSS found; defer unless curated value is clearly higher. -Defer WordPress/national newsrooms that already ship `/feed` or `rel=alternate`. +## Faraday vs JS -## Faraday vs “JS site” - -- `html2rss auto` returning 0 items ≠ empty HTML. **Fetch and inspect** (`curl -L` + Nokogiri, or Botasaurus `/scrape` HTML) before declaring Botasaurus-only. -- ISS Africa press/ISS Today: Faraday HTML already contained `a.card[href^=…]`; auto failed; explicit selectors shipped on Faraday. -- Prefer nested title selectors (`h6.card-subtitle`, `aria-label`) over whole-card text (dates, bylines, “PRIME”, region chrome). +- `html2rss scrape` returning 0 items ≠ empty HTML — fetch HTML before declaring Botasaurus-only. +- Prefer nested title selectors (`h6`, `aria-label`) over whole-card text. ## Redirects and schemes -- Faraday **rejects HTTPS→HTTP downgrades** (`UnsupportedUrlScheme`). Confirm with `curl -I -L` and watch `Location`. -- If the only stable surface downgrades: try Botasaurus once; if still flaky/timeout → **drop**, do not ship a lottery config (tralac lesson). +- Faraday rejects HTTPS→HTTP downgrades; confirm with `curl -I -L`. +- If only stable surface downgrades: try Botasaurus once, then drop. -## Botasaurus contract +## Botasaurus -- Health: `GET http://localhost:4010/health` (root `/` is often 404 — that is normal). -- Env: `BOTASAURUS_SCRAPER_URL=http://localhost:4010`. +- `BOTASAURUS_SCRAPER_URL=http://localhost:4010` on MCP process (`mcp.json`) and shell; health at `/health`. - **`wait_timeout_seconds` must be ≤ 30** (API validation against the work budget). Values above 30 → HTTP **422**. Do not confuse with the **45s** default total scrape wall (boot + navigate + wait). - 504 / scrape timeout → remove brittle `wait_for_selector`, retry once, then drop if still unreliable. -- Shipping `strategy: botasaurus` → always `scripts/register_botasaurus domain/file.yml`. - -## Reading `check_config` output - -- The script may print the **channel title** in the first summary lines. Confirm item quality by inspecting RSS `` rows (or Nokogiri on feed XML), not the summary alone. - -## Tool order reminders +- Shipping `strategy: botasaurus` → `scripts/register_botasaurus`. -- If `user-html2rss` MCP is `error` / discovery failed → **skip immediately** to CLI. Do not burn the timebox. -- Chrome MCP is last resort; curl + Botasaurus HTML + explicit selectors often enough. Report Chrome availability in handoff. +## MCP / CLI drift -## Selector patterns that worked +- Cursor catalog stale after gem upgrade: old `*_url` tool names, `-32602` on calls, new `inspect` “not found” — reload MCP/Cursor; use CLI until catalog shows bare verbs ([curation-verbs.md](curation-verbs.md)). +- `scrape` with `auto`: do not retry explicit `faraday` after empty auto (chain already ran). -| Pattern | Example | -| -------------------------- | ---------------------------------------------------------------------------------------------------- | -| Drupal field link | `span.field-content a[href^="/latest-news/"]` (SADC) | -| Card anchor + nested title | `a.card[href^="/iss-today/"]` + `h6.card-subtitle` (ISS) | -| Heading link | `h4.title a[href*="/News/"]` (Ahram); `h4 a[href*="lang2.html"]` (PanaPress) | -| Clean `aria-label` | `a[href*="/tea/news/"][aria-label]` + `extractor: attribute` / `attribute: aria-label` (EastAfrican) | +## Selectors -Always set `enhance: false` on items unless you have proven need. +- Anchor on article URL path fragments; use nested title / `aria-label` over card chrome. +- `enhance: false` on items unless proven necessary. diff --git a/.agents/skills/html2rss-config/reference/repair.md b/.agents/skills/html2rss-config/reference/repair.md index 3c6c6b7f..888a858a 100644 --- a/.agents/skills/html2rss-config/reference/repair.md +++ b/.agents/skills/html2rss-config/reference/repair.md @@ -1,30 +1,24 @@ # Mode: repair -Fix one existing config. SSOT: [AGENTS.md](../../../../AGENTS.md). Runtime Debugging section there is authoritative. +Fix one existing config. Runtime debugging: [AGENTS.md](../../../../AGENTS.md) § Runtime Debugging. -## Diagnose (cheapest first) +## Diagnose -1. Read the YAML; note `channel.url`, selectors, `strategy`. -2. `scripts/check_config ` (or `html2rss validate` then `html2rss feed`). -3. If zero items with Faraday: - - Retry with Botasaurus (`BOTASAURUS_SCRAPER_URL=http://localhost:4010` or `check_config … --botasaurus`). - - If Botasaurus works → keep config narrow; set `strategy: botasaurus`; run `scripts/register_botasaurus`. -4. If both request strategies fail or items are wrong → Chrome MCP snapshot; confirm item boundary / final URL after redirects. -5. Compare core `feed` vs configs-repo focused fetch when they disagree (request-strategy mismatch, not “selectors OK”). Agency/CDN sites often treat the gem UA differently than a browser — expect intermittent 403/504 in the fetch lane even when CLI `feed` looked fine; prefer Botasaurus or drop. -6. Botasaurus **422** with `wait_timeout_seconds` → value must be ≤ 20 (see [pitfalls.md](pitfalls.md)). **504** → drop wait selector or drop the config. +1. Read YAML: `channel.url`, selectors, `strategy`. +2. `scripts/check_config ` (or `html2rss validate` + `apply`). +3. Zero items on Faraday → Botasaurus (`check_config … --botasaurus`); if works → `strategy: botasaurus` + `register_botasaurus`. +4. Still wrong → Chrome MCP snapshot; confirm item boundary and post-redirect URL. +5. CLI `apply` OK but fetch spec fails → request-strategy mismatch; prefer Botasaurus or drop. +6. Botasaurus 422/504 → [pitfalls.md](pitfalls.md). ## Fix order -1. Wrong/canonical URL or locale redirect → fix `channel.url`. -2. Noisy enhancement → `enhance: false` on items (or channel). -3. Over-broad `items` → tighten to repeated article card / content link. -4. Drop weak optional fields (`description`, `published_at`, `categories`) before adding selector complexity. -5. Narrower path if the flagship page is unsalvageable. +1. Canonical URL / locale redirect → fix `channel.url`. +2. Chrome leakage → `enhance: false`. +3. Over-broad `items` → tighten to article card / content link. +4. Drop weak optionals before adding selector complexity. +5. Narrower path if flagship page is unsalvageable. -## Stop / drop +## Stop -After one tight loop (~3–4 minutes) with evidence: report **deferred** or recommend **drop** if still noisy, blocked (401/403/timeout), or first-party RSS makes the curated config low value — unless the user says keep going. - -## Done - -Same Quality Gate as [SKILL.md](../SKILL.md). Focused fetch must match the strategy you ship. Handoff: root cause, what changed, residual drift risk. +After one tight loop (~3–4 min) with evidence: defer or drop unless user says continue. Gate: AGENTS.md Quality Gate. diff --git a/.agents/skills/html2rss-config/reference/topics.md b/.agents/skills/html2rss-config/reference/topics.md index 89ef5c20..169d3630 100644 --- a/.agents/skills/html2rss-config/reference/topics.md +++ b/.agents/skills/html2rss-config/reference/topics.md @@ -1,38 +1,3 @@ # directory.topics -Required on every config. Vocabulary SSOT: `Html2rss::Config::Validator::DIRECTORY_TOPICS` (also listed in [AGENTS.md](../../../../AGENTS.md)). - -Prefer **1–2** primary topics. Do not invent strings. - -## Vocabulary - -`sports`, `energy`, `tech`, `science`, `news`, `entertainment`, `jobs`, `finance`, `security`, `travel`, `environment`, `consumer`, `civic`, `product`, `research` - -## Quick mapping - -| Surface | Typical topics | -| ---------------------------------- | ----------------------------------------- | -| AI / eng blog, OSS release notes | `tech` + `research` or `tech` + `product` | -| Company product newsroom | `tech` + `product` | -| Security advisories / cyber agency | `security` (+ `tech` or `civic`) | -| Gov / IGO / regulator press | `civic` | -| Think tank analysis | `civic` + `research` | -| Science institute / lab | `science` + `research` | -| Space agency | `science` + `research` | -| General newsroom | `news` | -| Investigative / OSINT newsroom | `news` + `civic` | -| Energy / climate agency | `energy` + `environment` | -| Markets / central bank / IMF-class | `finance` + `civic` | -| Patents / IP office | `civic` + `tech` | -| Courts / legal press | `civic` | -| Aviation / shipping regulator | `civic` | -| Telecom / spectrum regulator | `civic` + `tech` | -| Semiconductors / hardware vendor | `tech` + `product` | -| Food / agriculture agency | `environment` + `civic` | -| Jobs board | `jobs` | -| Consumer tests / recalls | `consumer` (+ `civic`) | -| Travel / local visitor news | `travel` (+ `news`) | -| Sports | `sports` | -| Film / games / music | `entertainment` | - -Ambiguous hybrid: pick audience intent, not every plausible tag. +Moved to [catalog.md § Topics](catalog.md#topics). diff --git a/.agents/skills/html2rss-config/scripts/add_config b/.agents/skills/html2rss-config/scripts/add_config new file mode 100755 index 00000000..9fddd6a9 --- /dev/null +++ b/.agents/skills/html2rss-config/scripts/add_config @@ -0,0 +1,494 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +require 'optparse' +require 'uri' +require 'fileutils' +require 'json' +require 'nokogiri' +require 'faraday' +require 'faraday/follow_redirects' +require 'public_suffix' +require_relative 'html2rss_api' + +VALID_TOPICS = Html2rss::Config::Validator::DIRECTORY_TOPICS.freeze + +USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' \ + 'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36' + +CONTAINER_CANDIDATES = [ + # Design systems (French DSFR, German Bund, UK GOV.UK, Drupal, Bootstrap) + 'div.fr-card', + 'div.fr-tile', + 'li.gem-c-document-list__item', + 'div.gem-c-document-list__item', + 'div.c-teaser', + 'div.c-card', + 'div.teaser', + 'div.teaser-card', + 'article.teaser', + 'div.views-row', + 'div.view-content > div', + 'div.item-list ul > li', + 'article', + 'div.post', + 'div.card', + 'div.news-item', + 'div.news-card', + 'div.news-list-item', + 'div.article-card', + 'div.post-card', + 'div.blog-card', + 'div.listing-item', + 'div.news-listing > div', + 'div.vf-card', + 'li.views-row', + 'li.news-item', + 'li.card', + 'li.post', + 'li.item', + 'ul.news-list > li', + 'ul.list-unstyled > li', + 'ul[class*="news"] > li', + 'ul[class*="list"] > li', + 'table tbody tr', + 'div[class*="news-item"]', + 'div[class*="news-card"]', + 'div[class*="article-card"]', + 'div[class*="post-card"]', + 'div[class*="teaser"]', + 'div[class*="card"]', + 'div[class*="post"]', + 'div[class*="article"]', + 'div[class*="item"]', + 'div[class*="entry"]', + 'div[class*="press"]', + 'div[class*="media"]', + 'li[class*="item"]', + 'li[class*="card"]', + 'a.fr-card__link', + 'a.c-teaser__link', + 'a[href^="/blog/"]:has(h2, h3, h4, p, span)', + 'a[href^="/news/"]:has(h2, h3, h4, p, span)', + 'a[href*="/news/"]:has(h2, h3, h4, p, span)', + 'a[href*="/press/"]:has(h2, h3, h4, p, span)', + 'a[href*="/articles/"]:has(h2, h3, h4, p, span)', + 'a[href*="/article/"]:has(h2, h3, h4, p, span)', + 'a[href*="/posts/"]:has(h2, h3, h4, p, span)', + 'a[href*="/noticia/"]:has(h2, h3, h4, p, span)', + 'a[href*="/comunicado/"]:has(h2, h3, h4, p, span)', + 'a[href*="/actualite/"]:has(h2, h3, h4, p, span)', + 'a[href*="/releases/"]:has(h2, h3, h4, p, span)', + 'a[href*="/story/"]:has(h2, h3, h4, p, span)', + 'a[href^="/changelog/"]', + 'a[href*="/news/"]', + 'a[href*="/press-release"]', + 'a[href*="/media-release"]' +].freeze + +def try_capture_path(url, opts, domain, slug, target_dir, target_file) + capture_opts = { enhance: false, force: opts[:force] } + capture_opts[:topics] = opts[:topics] if opts[:topics]&.any? + capture_opts[:title] = opts[:title] if opts[:title] + capture_opts[:summary] = opts[:summary] if opts[:summary] + + capture = Html2rssConfigApi.capture(url, **capture_opts) + if capture.native_feed? && !opts[:force] + return { status: :skipped, message: 'First-party RSS/Atom feed detected (use --force to override)', url: } + end + return nil unless capture.has_selectors && capture.articles_count >= 1 + + registry_id = Html2rssConfigApi.registry_id_for(domain, slug) + yaml_content = Html2rssConfigApi.inject_registry_id(capture.yaml, registry_id) + topics = (opts[:topics]&.any? ? opts[:topics] : capture.inferred_topics).select { |t| VALID_TOPICS.include?(t) } + topics = ['news'] if topics.empty? + + unless opts[:dry_run] + FileUtils.mkdir_p(target_dir) + File.write(target_file, yaml_content) + end + + rss = Html2rssConfigApi.apply_config_hash(capture.config).to_rss + first = rss.items.first + + { + status: :ok, + file: target_file, + domain:, + topics:, + title: capture.channel_title, + items_count: capture.articles_count, + sample_title: first&.title.to_s.strip.gsub(/\s+/, ' '), + sample_link: first&.link.to_s, + yaml: yaml_content, + url: + } +rescue StandardError + nil +end + +def resolve_registrable_domain(url) + uri = URI.parse(url) + host = uri.host.to_s.downcase + domain = begin + PublicSuffix.parse(host).domain + rescue StandardError + host + end + [domain, host] +end + +def generate_slug_from_url(url) + uri = URI.parse(url) + path_slug = uri.path.to_s.gsub(%r{^/|/$}, '').gsub(%r{[/_]+}, '_').downcase + path_slug = path_slug.sub(/\.(html?|php|aspx?)$/, '') + path_slug = 'news' if path_slug.empty? + path_slug +end + +def infer_topics(url, title) + text = "#{url} #{title}".downcase + inferred = [] + inferred << 'tech' if text =~ /tech|developer|software|cloud|api|compute|ai|data|model|hardware/ + inferred << 'security' if text =~ /security|vulnerab|advis|threat|breach|cve|cert|patch|cyber/ + inferred << 'science' if text =~ /science|space|astron|physics|bio|research|institut|discover|nature/ + inferred << 'energy' if text =~ /energy|wind|solar|power|grid|renewab|oil|gas|hydro/ + inferred << 'finance' if text =~ /financ|bank|invest|monetary|market|regulat|treasury|econom/ + inferred << 'civic' if text =~ /govern|policy|council|parliament|un\.org|court|treaty|public/ + inferred << 'environment' if text =~ /climat|environ|planet|sustainab|carbon|ecolog|earth/ + inferred << 'consumer' if text =~ /consum|recall|safety|rating|test|product/ + inferred << 'travel' if text =~ /travel|touris|destination|trip|flight|hotel|transit|rail/ + inferred << 'sports' if text =~ /sport|olympic|racing|football|soccer|tennis|game/ + inferred.empty? ? ['news'] : inferred.first(2) +end + +def fetch_html(url) + conn = Faraday.new(ssl: { verify: false }, request: { open_timeout: 4, timeout: 8 }) do |f| + f.response :follow_redirects, limit: 3 + f.adapter Faraday.default_adapter + end + res = conn.get(url, {}, { + 'User-Agent' => USER_AGENT, + 'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', + 'Accept-Language' => 'en-US,en;q=0.9,de;q=0.8' + }) + [res.body.to_s, res.status] +rescue StandardError + ['', 0] +end + +def find_best_selector_set(url, doc) + CONTAINER_CANDIDATES.each do |item_sel| + nodes = doc.css(item_sel) + next unless nodes.size.between?(1, 150) + + is_anchor_item = item_sel.start_with?('a[') || item_sel == 'a' || item_sel.start_with?('a.') + + title_options = if is_anchor_item + ['h1, h2, h3, h4, h5, h6', 'span[class*="title"], div[class*="title"], p[class*="title"]', 'p, span, strong', 'self'] + else + ['h1 a, h2 a, h3 a, h4 a, h5 a, h6 a', 'h1, h2, h3, h4, h5, h6', 'a.govuk-link, a.fr-card__link, a.c-teaser__link', 'a.title, a.heading, a[class*="title"], a[class*="heading"]', 'a'] + end + + url_options = if is_anchor_item + [nil] + else + ['h1 a, h2 a, h3 a, h4 a, h5 a, h6 a', 'a.govuk-link, a.fr-card__link, a.c-teaser__link', 'a.title, a.heading, a[class*="title"], a[class*="heading"]', 'a'] + end + + date_options = ['time', 'span.date, span[class*="date"], p[class*="date"]', nil] + + title_options.each do |t_sel| + url_options.each do |u_sel| + items_valid = nodes.all? do |node| + t = if t_sel == 'self' + node.text.to_s.strip + elsif t_sel + node.at_css(t_sel)&.text.to_s.strip + else + node.text.to_s.strip + end + !t.empty? && t.length >= 4 && t !~ /^(read more|more|details|link|view|pdf|click here|learn more)$/i + end + next unless items_valid + + urls = nodes.map do |node| + if u_sel.nil? + node['href'] + else + node.at_css(u_sel)&.[]('href') + end + end + next if urls.any? { |u| u.to_s.strip.empty? } + next if nodes.size > 1 && urls.uniq.size < [nodes.size * 0.6, 2].min + + date_options.each do |d_sel| + cfg = { + channel: { title: 'Test', url:, language: 'en', time_zone: 'UTC', ttl: 360 }, + selectors: { + items: { selector: item_sel, enhance: false }, + title: t_sel == 'self' ? {} : { selector: t_sel } + } + } + + if u_sel.nil? + cfg[:selectors][:url] = { extractor: 'href' } + else + cfg[:selectors][:url] = { selector: u_sel, extractor: 'href' } + end + + cfg[:selectors][:published_at] = { selector: d_sel } if d_sel + + begin + rss = Html2rssConfigApi.apply_config_hash(cfg).to_rss + count = rss.items.count + next unless count >= 1 + + valid_titles = rss.items.all? do |i| + t = i.title.to_s.strip + !t.empty? && t.length >= 4 && t !~ /^(read more|more|details|link|view|pdf|click here|learn more)$/i + end + next unless valid_titles + + valid_links = rss.items.all? { |i| !i.link.to_s.strip.empty? && i.link.to_s.start_with?('http') } + next unless valid_links + + unique_links = rss.items.map(&:link).uniq + next if count > 1 && unique_links.size < [count * 0.6, 2].min + + return { + items: item_sel, + title: t_sel == 'self' ? nil : t_sel, + url: u_sel, + date: d_sel, + count:, + sample_title: rss.items.first.title.to_s.strip.gsub(/\s+/, ' '), + sample_link: rss.items.first.link.to_s + } + rescue StandardError + next + end + end + end + end + end + nil +end + +def process_single_url(url, opts) + url = url.strip + return { status: :error, message: 'Empty URL', url: } if url.empty? + + unless opts[:force] + recon = Html2rssConfigApi.recon(url) + if recon.defer? + return { status: :skipped, message: "Native feed: #{recon.native_feed} (use --force to override)", url: } + end + if recon.drop? + return { status: :error, message: recon.notes.join('; '), url: } + end + end + + domain, _host = resolve_registrable_domain(url) + slug = generate_slug_from_url(url) + file_name = "#{slug}.yml" + target_dir = File.join('lib', 'html2rss', 'configs', domain) + target_file = File.join(target_dir, file_name) + + if File.exist?(target_file) && !opts[:force] + return { status: :skipped, message: "Config already exists at #{target_file}", url: } + end + + captured = try_capture_path(url, opts, domain, slug, target_dir, target_file) + return captured if captured + + html, status = fetch_html(url) + return { status: :error, message: "HTTP #{status} fetching URL", url: } unless (200..299).cover?(status) + + doc = Nokogiri::HTML(html) + + page_title = opts[:title] || doc.at_css('meta[property="og:site_name"]')&.[]('content') || + doc.at_css('meta[property="og:title"]')&.[]('content') || + doc.at_css('title')&.text&.strip || + domain.capitalize + page_title = page_title.gsub(/\s+/, ' ').strip + + topics = opts[:topics]&.any? ? opts[:topics] : infer_topics(url, page_title) + topics = topics.select { |t| VALID_TOPICS.include?(t) } + topics = ['news'] if topics.empty? + + summary = opts[:summary] || "#{page_title} updates and announcements." + summary = summary[0..159] + + selectors = find_best_selector_set(url, doc) + unless selectors + return { status: :error, message: 'Could not discover valid repeated article selectors on surface', url: } + end + + yaml_lines = [] + yaml_lines << '# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json' + yaml_lines << 'directory:' + yaml_lines << ' topics:' + topics.each { |t| yaml_lines << " - #{t}" } + yaml_lines << " title: #{page_title.inspect}" + yaml_lines << " summary: #{summary.inspect}" + yaml_lines << '' + yaml_lines << 'channel:' + yaml_lines << " title: #{page_title.inspect}" + yaml_lines << " url: #{url}" + yaml_lines << ' language: en' + yaml_lines << ' time_zone: UTC' + yaml_lines << ' ttl: 360' + yaml_lines << 'selectors:' + yaml_lines << ' items:' + yaml_lines << " selector: #{selectors[:items].inspect}" + yaml_lines << ' enhance: false' + yaml_lines << ' title:' + if selectors[:title] + yaml_lines << " selector: #{selectors[:title].inspect}" + end + yaml_lines << ' url:' + if selectors[:url] + yaml_lines << " selector: #{selectors[:url].inspect}" + end + yaml_lines << ' extractor: href' + if selectors[:date] + yaml_lines << ' published_at:' + yaml_lines << " selector: #{selectors[:date].inspect}" + end + yaml_lines << '' + yaml_content = yaml_lines.join("\n") + registry_id = Html2rssConfigApi.registry_id_for(domain, file_name) + yaml_content = Html2rssConfigApi.inject_registry_id(yaml_content, registry_id) + + unless opts[:dry_run] + FileUtils.mkdir_p(target_dir) + File.write(target_file, yaml_content) + end + + { + status: :ok, + file: target_file, + domain:, + topics:, + title: page_title, + items_count: selectors[:count], + sample_title: selectors[:sample_title], + sample_link: selectors[:sample_link], + yaml: yaml_content, + url: + } +end + +options = { + topics: [], + title: nil, + summary: nil, + force: false, + dry_run: false, + json: false, + file: nil, + ledger: nil +} + +parser = OptionParser.new do |opts| + opts.banner = 'Usage: add_config [options] [URL]' + + opts.on('-u', '--url URL', 'Target listing/article URL') { |v| options[:url] = v } + opts.on('-f', '--file PATH', 'Batch file (one URL per line or TSV: [slug\\t]URL[\\ttitle][\\ttopics][\\tsummary])') { |v| options[:file] = v } + opts.on('-t', '--topics TOPICS', 'Comma-separated directory topics') { |v| options[:topics] = v.split(',').map(&:strip) } + opts.on('--title TITLE', 'Directory & Channel title override') { |v| options[:title] = v } + opts.on('--summary SUMMARY', 'Directory summary (max 160 chars)') { |v| options[:summary] = v } + opts.on('--ledger PATH', 'Path to save detailed JSON ledger') { |v| options[:ledger] = v } + opts.on('--force', 'Force config creation even if native RSS probe detects feed') { options[:force] = true } + opts.on('--dry-run', 'Synthesize and test without writing to disk') { options[:dry_run] = true } + opts.on('--json', 'Output results as JSON') { options[:json] = true } + opts.on('-h', '--help', 'Show help') do + puts opts + exit 0 + end +end + +parser.parse! +options[:url] ||= ARGV[0] + +if options[:url].nil? && options[:file].nil? + puts parser + exit 1 +end + +results = [] + +if options[:file] + lines = File.readlines(options[:file]).map(&:strip).reject { |l| l.empty? || l.start_with?('#') } + puts "=== Starting Batch Ingestion of #{lines.size} Candidate Feeds from #{options[:file]} ===" + puts "Timestamp: #{Time.now.utc}" + + lines.each_with_index do |line, idx| + parts = line.split("\t").map(&:strip) + line_opts = options.dup + + url = parts.find { |p| p.start_with?('http://', 'https://') } || parts[0] + host_name = begin + URI.parse(url).host + rescue StandardError + url + end + + print format("[%03d/%03d] Probing %-45s ... ", idx + 1, lines.size, host_name) + $stdout.flush + + if parts.size >= 3 + if parts[0].start_with?('http://', 'https://') + line_opts[:title] = parts[1] unless parts[1].to_s.empty? + line_opts[:topics] = parts[2].split(',').map(&:strip) unless parts[2].to_s.empty? + line_opts[:summary] = parts[3] if parts[3] && !parts[3].empty? + else + line_opts[:topics] = parts[2].split(',').map(&:strip) unless parts[2].to_s.empty? + line_opts[:title] = parts[3] if parts[3] && !parts[3].empty? + line_opts[:summary] = parts[4] if parts[4] && !parts[4].empty? + end + end + + res = process_single_url(url, line_opts) + results << res + + if res[:status] == :ok + puts "✅ OK (#{res[:items_count]} items) -> #{res[:file]}" + elsif res[:status] == :skipped + puts "⚠️ SKIPPED: #{res[:message]}" + else + puts "❌ ERROR: #{res[:message]}" + end + end + + puts "\n=== Batch Run Summary ===" + puts "Total Processed: #{results.size}" + puts "Successes (Config Created): #{results.count { |r| r[:status] == :ok }}" + puts "Skipped (First-party RSS): #{results.count { |r| r[:status] == :skipped }}" + puts "Errors / Dropped: #{results.count { |r| r[:status] == :error }}" +elsif options[:url] + results << process_single_url(options[:url], options) +end + +if options[:ledger] + FileUtils.mkdir_p(File.dirname(options[:ledger])) + File.write(options[:ledger], JSON.pretty_generate(results)) + puts "Saved detailed ledger to #{options[:ledger]}" +end + +if options[:json] + puts JSON.pretty_generate(results) +elsif !options[:file] + results.each do |r| + if r[:status] == :ok + puts "✅ [OK #{r[:items_count]} items] #{r[:file]} (#{r[:topics].join(', ')})" + puts " Title: #{r[:title]}" + puts " Sample: #{r[:sample_title]} -> #{r[:sample_link]}" + elsif r[:status] == :skipped + puts "⚠️ [SKIPPED] #{r[:url]}: #{r[:message]}" + else + puts "❌ [ERROR] #{r[:url]}: #{r[:message]}" + end + end +end diff --git a/.agents/skills/html2rss-config/scripts/batch_recon b/.agents/skills/html2rss-config/scripts/batch_recon index da734935..33365c1d 100755 --- a/.agents/skills/html2rss-config/scripts/batch_recon +++ b/.agents/skills/html2rss-config/scripts/batch_recon @@ -1,43 +1,17 @@ #!/usr/bin/env ruby # frozen_string_literal: true -# rubocop:disable Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/MethodLength,Metrics/ParameterLists,Metrics/PerceivedComplexity - -# Parallel recon for one or many candidate URLs. -# For each candidate: follow redirects, probe first-party RSS on the exact surface, -# cache HTML once, emit a BUILD/DEFER/DROP ledger (TSV). +# Parallel recon via html2rss gem (Html2rss.batch_recon / Recon.call). +# Emits BUILD/DEFER/DROP ledger + slug-named HTML cache for analyze_html. # # Usage (from repo root): # .agents/skills/html2rss-config/scripts/batch_recon 'https://example.com/news/' -# .agents/skills/html2rss-config/scripts/batch_recon URL [URL ...] # .agents/skills/html2rss-config/scripts/batch_recon --file candidates.tsv -# -# Candidate file formats (blank lines and # comments ignored): -# https://example.com/news/ -# example.com https://example.com/news/ -# example.com LATAM https://example.com/news/ -# -# Exit 0 always after a completed run (including all-DROP). Exit 1 on usage/parse errors. require 'fileutils' -require 'net/http' -require 'openssl' require 'uri' +require_relative 'html2rss_api' -UA = 'html2rss-config-batch_recon/1.0' -FEED_PATHS = %w[ - /feed - /feed.xml - /rss - /rss.xml - /atom.xml - /index.xml - /news/rss - /news/feed - /blog/feed - /blog/rss.xml -].freeze -FEED_CT = %w[rss+xml atom+xml application/rss application/atom].freeze USAGE = <<~USAGE Usage: batch_recon [options] URL [URL ...] batch_recon [options] --file candidates.tsv @@ -49,8 +23,6 @@ USAGE = <<~USAGE --dry-run Print candidates only; no network -h, --help Show this help USAGE -ASSET_EXT = /\.(svg|png|jpg|jpeg|gif|ico|css|js)(\?|$)/i -FEED_ACCEPT = 'application/rss+xml, application/atom+xml, application/xml, text/xml, */*' def usage!(msg = nil) warn(msg) if msg @@ -94,10 +66,8 @@ end def parse_args(argv) opts = default_opts index = 0 - while index < argv.size - index = apply_flag(opts, argv, index) - index += 1 - end + index = apply_flag(opts, argv, index) + index += 1 while index < argv.size opts[:ledger] ||= File.join(opts[:cache_dir], 'ledger.tsv') opts end @@ -123,7 +93,7 @@ def candidate_parts(cols) when 2 then [cols[0], '-', cols[1]] when 3 then [cols[0], cols[1], cols[2]] else - raise "Bad candidate line (want URL or slug[\\tregion]\\turl): #{cols.join("\t").inspect}" + raise "Bad candidate line (want URL or slug\\tregion\\turl): #{cols.join("\t").inspect}" end end @@ -160,235 +130,6 @@ rescue RuntimeError => error usage!(error.message) end -def http_client(uri, insecure: false) - http = Net::HTTP.new(uri.host, uri.port) - http.use_ssl = uri.scheme == 'https' - http.open_timeout = 10 - http.read_timeout = 20 - http.verify_mode = insecure ? OpenSSL::SSL::VERIFY_NONE : OpenSSL::SSL::VERIFY_PEER - http -end - -def request(uri, accept: nil, insecure: false) - req = Net::HTTP::Get.new(uri) - req['User-Agent'] = UA - req['Accept'] = accept || 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' - http_client(uri, insecure: insecure).request(req) -end - -def empty_follow(attrs) - { - chain: attrs.fetch(:chain), - final_url: attrs.fetch(:current), - status: attrs[:status], - body: attrs[:body], - error: attrs[:error], - ssl_note: attrs[:ssl_note], - insecure: attrs.fetch(:insecure, false) - } -end - -def follow_ssl_retry(url, max, error) - retried = follow(url, insecure: true, max: max) - retried[:ssl_note] = "ssl_insecure_retry: #{error.message[0, 80]}" - retried -end - -def handle_follow_response(res, chain, current) - status = res.code.to_i - chain << "#{status}:#{current}" - if res.is_a?(Net::HTTPRedirection) && res['location'] - return [:redirect, URI.join(current, res['location']).to_s, status, nil] - end - - [:done, current, status, res.body.to_s] -end - -def follow(url, insecure: false, max: 8) - chain = [] - current = url - body = nil - status = nil - ssl_note = nil - - max.times do - begin - res = request(URI(current), insecure: insecure) - rescue OpenSSL::SSL::SSLError => error - return follow_ssl_retry(url, max, error) unless insecure - - return empty_follow(chain: chain, current: current, error: "SSL: #{error.message}", ssl_note: 'ssl_fail') - rescue Net::HTTPExceptions => error - code = error.response&.code&.to_i - chain << "#{code}:#{current}" if code - return empty_follow(chain: chain, current: current, status: code, body: error.response&.body.to_s) - end - - state, current, status, body = handle_follow_response(res, chain, current) - break if state == :done - end - - empty_follow( - chain: chain, - current: current, - status: status, - body: body, - ssl_note: ssl_note, - insecure: insecure - ) -rescue StandardError => error - empty_follow(chain: chain, current: current || url, error: "#{error.class}: #{error.message}") -end - -def feed_content_type?(content_type) - FEED_CT.any? { |marker| content_type.include?(marker) } -end - -def feed_body?(sample) - sample.include?(']/i) -end - -def real_feed?(res, body, url) - return false unless res.is_a?(Net::HTTPSuccess) - return false if url.match?(ASSET_EXT) - - content_type = res['content-type'].to_s.downcase - blocked = %w[image svg javascript text/css] - return false if blocked.any? { |part| content_type.include?(part) } - - sample = body.to_s[0, 4000] - feed_content_type?(content_type) || feed_body?(sample) -end - -def fetch_once(url) - request(URI(url), accept: FEED_ACCEPT) -end - -def fetch_feed_candidate(url) - res = fetch_once(url) - body = res.body.to_s - if res.is_a?(Net::HTTPRedirection) && res['location'] - url = URI.join(url, res['location']).to_s - res = fetch_once(url) - body = res.body.to_s - end - [res, body, url] -rescue StandardError - [nil, '', url] -end - -def link_attr(tag, name) - tag[/\b#{name}\s*=\s*["']([^"']+)["']/i, 1].to_s -end - -def alternate_feed_url?(rel, type, href) - return false if href.empty? - return false unless rel.split(/\s+/).include?('alternate') || type.match?(/rss|atom/) - return false unless type.match?(/rss|atom|xml/) || href.match?(/rss|atom|feed/i) - - true -end - -def absolute_href(page_url, href) - URI.join(page_url, href).to_s -rescue StandardError - nil -end - -def alternate_feed_hrefs(html, page_url) - html.to_s.scan(/]*>/i).filter_map do |tag| - rel = link_attr(tag, 'rel').downcase - type = link_attr(tag, 'type').downcase - href = link_attr(tag, 'href') - next unless alternate_feed_url?(rel, type, href) - - abs = absolute_href(page_url, href) - next if abs.nil? || abs.match?(ASSET_EXT) - - abs - end.uniq -end - -def origin_of(page_url) - uri = URI(page_url) - "#{uri.scheme}://#{uri.host}" -rescue StandardError - nil -end - -def collect_alternate_feeds(html, page_url) - alternate_feed_hrefs(html, page_url).filter_map do |abs| - res, body, final = fetch_feed_candidate(abs) - next unless res && real_feed?(res, body, final) - - final - end -end - -def collect_path_feeds(origin) - found = [] - FEED_PATHS.each do |path| - res, body, final = fetch_feed_candidate("#{origin}#{path}") - next unless res && real_feed?(res, body, final) - - found << final - break - end - found -end - -def probe_feeds(page_url, html) - found = collect_alternate_feeds(html, page_url) - return found.uniq if found.any? - - origin = origin_of(page_url) - return [] unless origin - - collect_path_feeds(origin).uniq -end - -def https_to_http?(start_url, final_url) - URI(start_url).scheme == 'https' && URI(final_url).scheme == 'http' -rescue StandardError - false -end - -def verdict_notes(result) - note = [] - note << result[:ssl_note] if result[:ssl_note] - note << 'insecure_ssl_fetch' if result[:insecure] - note -end - -def early_verdict(start_url, result, note) - body = result[:body].to_s - return ['DROP', note + ["unreachable: #{result[:error]}"], body] if result[:error] - return ['DROP', note + ["http_#{result[:status] || 'nil'}"], body] if result[:status].nil? || result[:status] >= 400 - return ['DROP', note + ['https_to_http_downgrade'], body] if https_to_http?(start_url, result[:final_url]) - - nil -end - -def html_verdict(result, note, body) - feeds = probe_feeds(result[:final_url], body) - return ['DEFER', note + ["native_rss=#{feeds.first}"], body] if feeds.any? - return ['DROP', note + ['error_page'], body] if body.include?('ErrorPage') || result[:final_url].include?('ErrorPage') - return ['BUILD', note + ['thin_html_needs_botasaurus'], body] if body.strip.empty? || body.bytesize < 800 - if body.match?(/cf-browser-verification|just a moment|access denied|captcha/i) && body.bytesize < 50_000 - return ['BUILD', note + ['possibly_blocked_html'], body] - end - - ['BUILD', note + ["html_bytes=#{body.bytesize}"], body] -end - -def verdict_for(start_url, result) - note = verdict_notes(result) - early = early_verdict(start_url, result, note) - return early if early - - html_verdict(result, note, result[:body].to_s) -end - def print_dry_run(opts, candidates) puts %w[slug region url].join("\t") candidates.each { |cand| puts [cand[:slug], cand[:region], cand[:url]].join("\t") } @@ -397,26 +138,32 @@ def print_dry_run(opts, candidates) puts "count=#{candidates.size}" end -def recon_row(cand, result, verdict, notes, body, cache_dir) - html_path = File.join(cache_dir, "#{cand[:slug]}.html") - File.write(html_path, body.to_s) +def redirect_chain(result) + req = result.requested_url.to_s + fin = result.final_url.to_s + req == fin ? [] : [req, fin] +end + +def recon_row(cand, result, cache_dir) + verdict = result.verdict.to_s.upcase + html_path = Html2rssConfigApi.copy_recon_cache_to_slug(cand[:url], cache_dir, cand[:slug]) + notes = result.notes.join('; ') [ cand[:slug], cand[:region], verdict, cand[:url], - result[:final_url], + result.final_url.to_s, html_path, - notes.join('; '), - (result[:chain] || []).join(' > ') + notes, + redirect_chain(result).join(' > ') ] end def process_candidate(cand, cache_dir) - result = follow(cand[:url]) - verdict, notes, body = verdict_for(cand[:url], result) - row = recon_row(cand, result, verdict, notes, body, cache_dir) - warn([verdict, cand[:slug], result[:final_url], notes.join('; ')].join("\t")) + result = Html2rssConfigApi.recon(cand[:url], cache_dir: cache_dir) + row = recon_row(cand, result, cache_dir) + warn([row[2], cand[:slug], result.final_url.to_s, row[6]].join("\t")) row end @@ -464,5 +211,3 @@ end rows = run_recon(opts, candidates) write_ledger(opts[:ledger], rows) summarize(opts, rows) - -# rubocop:enable Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/MethodLength,Metrics/ParameterLists,Metrics/PerceivedComplexity diff --git a/.agents/skills/html2rss-config/scripts/check_config b/.agents/skills/html2rss-config/scripts/check_config index 0bb00da9..4e75f4d0 100755 --- a/.agents/skills/html2rss-config/scripts/check_config +++ b/.agents/skills/html2rss-config/scripts/check_config @@ -1,12 +1,13 @@ #!/usr/bin/env ruby # frozen_string_literal: true -# Validate + feed (+ optional focused fetch) for one curated config. +# validate → test (+ optional focused fetch). Mirrors MCP golden path steps. # Usage (from repo root): # .agents/skills/html2rss-config/scripts/check_config lib/html2rss/configs/domain/file.yml # .agents/skills/html2rss-config/scripts/check_config domain/file.yml --fetch # .agents/skills/html2rss-config/scripts/check_config domain/file.yml --fetch --botasaurus +require 'json' require 'open3' require 'pathname' @@ -19,7 +20,7 @@ end def repo_root Pathname.new(__dir__).ascend.find { |p| (p / 'lib/html2rss/configs').directory? } || - abort('Could not find html2rss-configs repo root') + abort('Could not find html2rss-configs repo root (expected lib/html2rss/configs/)') end def resolve_config(arg, root) @@ -33,8 +34,10 @@ def resolve_config(arg, root) found.expand_path end -def feed_example_name(config_path, root) +def config_example_name(config_path, root) config_path.relative_path_from(root / 'lib/html2rss/configs').to_s +rescue StandardError + config_path.basename.to_s end def invoke_entry(entry, sibling) @@ -54,7 +57,6 @@ def sibling_html2rss_invocation(root) MSG end -# Prefer PATH `html2rss`, then sibling ../html2rss (exe via bundle, then bin). def resolve_html2rss_invocation(root) return ['html2rss'] unless `command -v html2rss 2>/dev/null`.strip.empty? @@ -71,10 +73,6 @@ def run!(*cmd, env: {}, quiet: false, chdir: nil) out end -def count_items(rss) - rss.scan(/]/i).size -end - usage! if ARGV.empty? || ARGV.include?('-h') || ARGV.include?('--help') fetch = ARGV.delete('--fetch') @@ -83,7 +81,7 @@ usage! unless ARGV.size == 1 root = repo_root config = resolve_config(ARGV.fetch(0), root) -example = feed_example_name(config, root) +example = config_example_name(config, root) cli = resolve_html2rss_invocation(root) cli_chdir = cli.first == 'bundle' ? (root.parent / 'html2rss').to_s : nil env = {} @@ -91,17 +89,21 @@ env['BOTASAURUS_SCRAPER_URL'] = ENV.fetch('BOTASAURUS_SCRAPER_URL', 'http://loca Dir.chdir(root) do run!(*cli, 'validate', config.to_s, env: env, chdir: cli_chdir) - rss = run!(*cli, 'feed', config.to_s, env: env, quiet: true, chdir: cli_chdir) - items = count_items(rss) + test_json = run!(*cli, 'test', config.to_s, '--json', env: env, quiet: true, chdir: cli_chdir) + result = JSON.parse(test_json) + abort("Test failed: #{result['error_message'] || 'unknown'}") unless result['success'] + + items = result['item_count'].to_i puts "items #{items}" - abort('Feed produced zero elements') if items.zero? + abort('Test reported zero items') if items.zero? - titles = rss.scan(%r{(.*?)}m).flatten.drop(1).first(5) - titles.each { |t| puts " - #{t.gsub(//, '').strip}" } + result.fetch('sample_items', []).first(5).each do |item| + puts " - #{item['title']}" + end if fetch - cmd = ['bundle', 'exec', 'rspec', '--tag', 'fetch', '--example', example, - 'spec/html2rss/configs_dynamic_spec.rb'] + spec_file = (root / 'spec/html2rss/configs_dynamic_spec.rb').file? ? 'spec/html2rss/configs_dynamic_spec.rb' : 'test/configs_dynamic_spec.rb' + cmd = ['bundle', 'exec', 'rspec', '--tag', 'fetch', '--example', example, spec_file] run!(*cmd, env: env) end end diff --git a/.agents/skills/html2rss-config/scripts/html2rss_api.rb b/.agents/skills/html2rss-config/scripts/html2rss_api.rb new file mode 100644 index 00000000..d2880d1d --- /dev/null +++ b/.agents/skills/html2rss-config/scripts/html2rss_api.rb @@ -0,0 +1,57 @@ +# frozen_string_literal: true + +# Shared html2rss gem facades (same verbs as MCP/CLI). +# Usage: require_relative 'html2rss_api' from skill scripts. + +require 'bundler/setup' +require 'digest' +require 'fileutils' +require 'uri' +require 'yaml' +require 'html2rss' + +module Html2rssConfigApi + module_function + + def recon(url, cache_dir: nil, **) + Html2rss.recon(url, cache_dir:, **) + end + + def capture(url, **) + Html2rss.capture(url, **) + end + + def test_config(path, **) + Html2rss.test(path, **) + end + + def apply_config_hash(config) + Html2rss.apply(config) + end + + def recon_html_cache_path(url, cache_dir) + host = URI(url.to_s).host.to_s.delete_prefix('www.') + host = 'snapshot' if host.empty? + digest = Digest::SHA256.hexdigest(url.to_s)[0, 12] + File.join(cache_dir, "#{host}-#{digest}.html") + end + + def copy_recon_cache_to_slug(url, cache_dir, slug) + src = recon_html_cache_path(url, cache_dir) + dest = File.join(cache_dir, "#{slug}.html") + FileUtils.cp(src, dest) if File.file?(src) + File.file?(dest) ? dest : src + end + + def inject_registry_id(yaml_content, registry_id) + doc = YAML.safe_load(yaml_content, permitted_classes: [Symbol], symbolize_names: true) + doc[:registry] = { id: registry_id } + modeline, = yaml_content.lines.first&.match(%r{\A(# yaml-language-server:.*)\n}i) + body = Html2rss::Config.to_yaml(doc.transform_keys(&:to_sym)) + modeline ? "#{modeline}\n#{body}" : body + end + + def registry_id_for(domain, slug) + "#{domain}/#{slug.sub(/\.yml\z/, '')}" + end +end diff --git a/.agents/skills/html2rss-config/scripts/probe_rss b/.agents/skills/html2rss-config/scripts/probe_rss index 2e8a4e16..0a4cb7a4 100755 --- a/.agents/skills/html2rss-config/scripts/probe_rss +++ b/.agents/skills/html2rss-config/scripts/probe_rss @@ -1,182 +1,26 @@ #!/usr/bin/env ruby # frozen_string_literal: true -# Probe first-party RSS/Atom for a page or origin URL. -# 1) Fetch the page and parse feed hints. -# 2) Fall back to common feed path guesses (+ optional extra paths). +# First-party RSS probe via html2rss recon (Syndication::Discovery). # -# Exit 0 = no feed found (curated config may add value). -# Exit 3 = feed found (print URL; consider dropping per AGENTS.md). -# Exit 1 = usage / request error. -# -# Notes for agents: exit 3 is intentional success-for-"has feed". Under `set -e`, -# check explicitly (`probe_rss …; status=$?`) — do not treat 3 as a shell failure. +# Exit 0 = no native feed on surface (curated config may add value). +# Exit 3 = native feed found (print URL; consider drop per AGENTS.md). # # Usage: # .agents/skills/html2rss-config/scripts/probe_rss 'https://example.com/news/' -# .agents/skills/html2rss-config/scripts/probe_rss 'https://example.com' /feed /rss.xml - -require 'net/http' -require 'uri' - -UA = 'html2rss-config-probe_rss/1.0' -DEFAULT_PATHS = %w[ - /feed - /feed.xml - /rss - /rss.xml - /atom.xml - /index.xml - /news/rss - /news/feed - /blog/feed - /blog/rss.xml -].freeze -FEED_CT_MARKERS = %w[xml rss atom].freeze -FEED_BODY_MARKERS = %w[]*>/i -HREF_RE = /\bhref\s*=\s*["']([^"']+)["']/i -TYPE_RE = /\btype\s*=\s*["']([^"']+)["']/i -REL_RE = /\brel\s*=\s*["']([^"']+)["']/i - -def usage! - warn <<~USAGE - Usage: probe_rss [extra/path ...] - USAGE - exit 1 -end - -def http_client(uri) - http = Net::HTTP.new(uri.host, uri.port) - http.use_ssl = uri.scheme == 'https' - http.open_timeout = 8 - http.read_timeout = 12 - http -end - -def get_request(uri) - req = Net::HTTP::Get.new(uri) - req['User-Agent'] = UA - req['Accept'] = 'application/rss+xml, application/atom+xml, application/xml, text/xml, text/html, */*' - req -end - -def follow_redirect(url, res) - loc = URI.join(url, res['location']) - [http_client(loc).request(get_request(loc)), loc.to_s] -end - -def fetch(url) - uri = URI(url) - res = http_client(uri).request(get_request(uri)) - 4.times do - break unless res.is_a?(Net::HTTPRedirection) - - res, url = follow_redirect(url, res) - end - [res, url] -rescue StandardError => error - [error, url] -end - -def ok_response?(res) - res.respond_to?(:code) && res.code.to_i.between?(200, 299) -end - -def feedish?(res) - return false unless ok_response?(res) - - ct = (res['content-type'] || '').downcase - return true if FEED_CT_MARKERS.any? { |marker| ct.include?(marker) } - - body = res.body.to_s[0, 800].downcase - FEED_BODY_MARKERS.any? { |marker| body.include?(marker) } -end - -def page_dir_paths(page) - path = URI(page).path - return [] if path.nil? || path.empty? || path == '/' - - dir = path.sub(%r{/[^/]*$}, '/') - ["#{dir}feed", "#{dir}rss.xml", "#{dir}atom.xml"] -end -def absolute_url(base, path) - return path if path.start_with?('http') +require_relative 'html2rss_api' - origin = URI.join(base, '/').to_s.chomp('/') - return "#{origin}#{path}" if path.start_with?('/') - - URI.join("#{origin}/", path).to_s -rescue StandardError - origin = base.to_s.sub(%r{/[^/]*$}, '') - origin = base.to_s if origin.empty? - "#{origin.chomp('/')}#{path.start_with?('/') ? path : "/#{path}"}" -end - -def syndication_type?(type) - return false if type.nil? || type.empty? - - t = type.downcase - t.include?('rss+xml') || t.include?('atom+xml') || t.include?('rss') || t.include?('atom') -end - -def feed_like_href?(href) - href.match?(/rss|atom|feed|\.xml/i) -end - -def alternate_rel?(tag) - tag[REL_RE, 1].to_s.downcase.split(/\s+/).include?('alternate') -end - -def href_from_alternate_link(tag, page_url) - return unless alternate_rel?(tag) - - href = tag[HREF_RE, 1] - return if href.nil? || href.empty? - return unless syndication_type?(tag[TYPE_RE, 1]) || feed_like_href?(href) - - absolute_url(page_url, href.strip) +url = ARGV.fetch(0) do + warn 'Usage: probe_rss URL' + exit 2 end -def alternate_feed_hrefs(html, page_url) - return [] if html.nil? || html.empty? - - html.scan(LINK_TAG_RE).filter_map { |tag| href_from_alternate_link(tag, page_url) }.uniq -end - -def first_feedish_url(urls) - urls.lazy.filter_map do |url| - res, final = fetch(url) - final if feedish?(res) - end.first -end - -def page_fetch_context(page) - page_res, final_page = fetch(page) - page_url = ok_response?(page_res) ? final_page : page - html = ok_response?(page_res) ? page_res.body.to_s : '' - [page_url, html] -end - -def find_feed(page, extra) - page_url, html = page_fetch_context(page) - hit = first_feedish_url(alternate_feed_hrefs(html, page_url)) - return hit if hit - - paths = (page_dir_paths(page_url) + DEFAULT_PATHS + extra).uniq - first_feedish_url(paths.map { |path| absolute_url(page_url, path) }) -end - -usage! if ARGV.empty? || ARGV.include?('-h') || ARGV.include?('--help') - -page = ARGV.fetch(0) -hit = find_feed(page, ARGV.drop(1)) +result = Html2rssConfigApi.recon(url) -if hit - puts "HAS_RSS\t#{hit}" +if result.native_feed? + puts result.native_feed exit 3 end -puts "NO_RSS\t#{page}" exit 0 diff --git a/.agents/skills/html2rss-config/scripts/register_botasaurus b/.agents/skills/html2rss-config/scripts/register_botasaurus index 7fe2e57c..9afe3574 100755 --- a/.agents/skills/html2rss-config/scripts/register_botasaurus +++ b/.agents/skills/html2rss-config/scripts/register_botasaurus @@ -17,14 +17,15 @@ end def repo_root Pathname.new(__dir__).ascend.find { |p| (p / 'spec/support/botasaurus_fetch_configs.rb').file? } || - abort('Could not find html2rss-configs repo root') + abort('Could not find html2rss-configs repo root (expected spec/support/botasaurus_fetch_configs.rb)') end def normalize_name(arg, root) rel = arg.to_s.split('lib/html2rss/configs/', 2).last.delete_prefix('/').delete_prefix('./') abort('Expected something like domain/file.yml') unless rel.match?(%r{\A[^/]+/.+\.yml\z}) - warn "Warning: missing #{root / 'lib/html2rss/configs' / rel}" unless (root / 'lib/html2rss/configs' / rel).file? + target = root / 'lib/html2rss/configs' / rel + warn "Warning: missing #{target}" unless target.file? rel end diff --git a/AGENTS.md b/AGENTS.md index 373b9d38..e4dd3e13 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -69,6 +69,18 @@ Useful patterns: - Keep selectors item-local when possible. - Do not add complexity to recover weak optional fields. +## html2rss MCP + +Prefer `user-html2rss` when the tool catalog shows **bare verbs** (`inspect`, `capture`, …). Same words as CLI and [curation-verbs.md](.agents/skills/html2rss-config/reference/curation-verbs.md). + +**Journeys:** scrape (now) · capture → test → apply (YAML) · inspect → recon (build?). Follow each result’s `next_step` / `guidance`; `capture` YAML is a draft until catalog fields are added. + +**Setup:** `BOTASAURUS_SCRAPER_URL` on the MCP process (`mcp.json`), not only your shell. Read `html2rss://runtime` when `next_step` is `read_runtime`. + +**After gem upgrade:** If old names (`inspect_url`, `capture_config`) still appear in Cursor or calls return `-32602`, the server may be fine — **reload MCP / Cursor** until the catalog matches. Until then, use CLI (`html2rss …` from `../html2rss`) or skill scripts — authoritative fallback. + +**Strategy:** `scrape`/`capture` + `auto` run Faraday → Botasaurus (don’t retry `faraday` after `auto`). `inspect` + `auto` uses Faraday only; pin `botasaurus` for JS-heavy inspect. + ## Chrome MCP Use Chrome MCP when the static HTML is unclear, the page is hydrated, or Faraday fetch returns zero items while the browser shows a valid list. @@ -85,7 +97,7 @@ If Chrome MCP is unavailable (`Transport closed` or page-lock errors), do this r 1. Kill stale Chrome MCP processes (`pkill -9 -f 'chrome-devtools-mcp|Chrome for Testing'`). 2. Retry Chrome MCP once before continuing. -3. If still unavailable, continue with `curl -I -L`, runtime `feed`, and HTML inspection in a temporary file. +3. If still unavailable, continue with `curl -I -L`, `html2rss apply`, and HTML inspection in a temporary file. 4. Explicitly report Chrome MCP outage in the final handoff. ## Botasaurus @@ -116,7 +128,7 @@ Assume the `html2rss` CLI is available on `PATH` when working against the siblin 2. Inspect the DOM in Chrome MCP before writing selectors. 3. Create the YAML with the schema modeline and minimal selectors. 4. Validate the single file with the core CLI. -5. Generate a live feed with the core CLI. +5. Generate a live feed with `html2rss apply`. 6. Tighten selectors until the feed output is clean. 7. Run repo validation and non-fetch tests. 8. Run the appropriate fetch lane: @@ -138,9 +150,11 @@ html2rss validate /abs/path/to/config.yml ```bash cd ../html2rss -html2rss feed /abs/path/to/config.yml +html2rss apply /abs/path/to/config.yml ``` +(`feed` is a CLI alias for `apply`.) + 3. Catalog serialization for changed configs: ```bash @@ -190,7 +204,7 @@ curl -I -L -s https://example.com | sed -n '1,20p' ``` - compare behavior in both runtimes: - - core repo (`../html2rss`) via `html2rss feed` + - core repo (`../html2rss`) via `html2rss apply` - configs repo fetch lane (`bundle exec rspec --tag fetch --example ...`) - if selectors are valid in core but fetch lane still returns zero items, treat this as request-strategy/runtime mismatch, not selector success. - in that case: prefer Botasaurus-backed verification if available; otherwise mark as downgraded/deferred with evidence. @@ -200,7 +214,7 @@ curl -I -L -s https://example.com | sed -n '1,20p' Use the core CLI as the authority for single-config debugging. The quickest loop is: 1. `validate` -2. `feed` +2. `apply` 3. inspect the RSS for zero items, nav/footer leakage, duplicates, relative URLs, or noisy descriptions 4. adjust selectors 5. rerun @@ -214,13 +228,13 @@ Additional high-value checks: - Remove optional fields first when quality drops (`categories`, synthetic IDs, weak descriptions) before adding selector complexity. - Set `enhance: false` early if enhancement starts pulling nav/hero/market widgets. -## Auto-Source +## Scrape (reconnaissance) -Use `auto` for reconnaissance, not as proof that a config is ready. +Use `scrape` for reconnaissance, not as proof that a config is ready. CLI alias: `auto`. ```bash cd ../html2rss -html2rss auto 'https://example.com' +html2rss scrape 'https://example.com' ``` Use it to: @@ -229,7 +243,7 @@ Use it to: - compare Faraday and Botasaurus behavior quickly - decide whether a site belongs in the curated set at all -Do not ship raw auto-sourced output without manual tightening. +Do not ship raw scrape output without manual tightening into a curated config. ## Drop Or Downgrade @@ -257,3 +271,16 @@ When finishing config work, report: - residual risks, especially selector drift, localization dependence, or Botasaurus dependence - whether Chrome MCP was available during validation - whether focused fetch specs matched core runtime behavior + +## Contributor notes + +Config workflow router: [.agents/skills/html2rss-config/SKILL.md](.agents/skills/html2rss-config/SKILL.md) (links to mode refs; does not duplicate this gate). + +Stale CLI/MCP verb drift check: + +```bash +rg -n 'html2rss (feed|auto)\b|capture_config|inspect_url|scrape_url|test_config|apply_config|batch_scrape_urls|batch_inspect_urls' \ + --glob '!**/configs/**' --glob '!lib/html2rss/configs/**' AGENTS.md README.md .agents/ spec/ +``` + +Exclude `bin/validate_configs` (repo script name). Prefer `apply` / `scrape` and bare MCP verb names in docs. diff --git a/README.md b/README.md index 782a7f7c..e74bb22d 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ The `type` field specifies the parameter type (currently only `string` is suppor Notes: - Only configs that use `%s` placeholders need a `parameters` section. -- Callers can still override those defaults at runtime with `html2rss feed ... --params ...`. +- Callers can still override those defaults at runtime with `html2rss apply ... --params ...` (`feed` is a CLI alias). - Dynamic substitution applies to `channel` and `headers`; selectors are not parameterized by this feature. ## Validation