chore: test against html2rss default strategy and migrate tooling to HTTPX

.agents/skills/html2rss-config/reference/curation-verbs.md

@@ -32,8 +32,8 @@ Batch: `batch_inspect`, `batch_recon`, `batch_scrape`. CLI aliases: `feed` → `
 
 | Tool | `strategy: auto` behavior |
 | --- | --- |
-| `scrape`, `capture` | Faraday → Botasaurus fallback — **do not** retry explicit `faraday` after `auto` |
-| `inspect` | Faraday only (cheap); pin `botasaurus` when you need browser rendering |
+| `scrape`, `capture` | Default (HTTPX) → Botasaurus fallback — **do not** retry explicit `default` after `auto` |
+| `inspect` | Default only (cheap); pin `botasaurus` when you need browser rendering |
 
 ## Configs-repo note
 

.agents/skills/html2rss-config/reference/pitfalls.md

@@ -19,14 +19,14 @@ Invariants for `new` / `repair` / batch campaigns. Quality gate: [AGENTS.md](../
 - Run `probe_rss` / `batch_recon` on the same URL as `channel.url`, not just the homepage.
 - Exit `3` / ledger `DEFER` → native RSS found; defer unless curated value is clearly higher.
 
-## Faraday vs JS
+## Static HTTP vs JS
 
 - `html2rss scrape` returning 0 items ≠ empty HTML — fetch HTML before declaring Botasaurus-only.
 - Prefer nested title selectors (`h6`, `aria-label`) over whole-card text.
 
 ## Redirects and schemes
 
-- Faraday rejects HTTPS→HTTP downgrades; confirm with `curl -I -L`.
+- Default HTTP strategy rejects HTTPS→HTTP downgrades; confirm with `curl -I -L`.
 - If only stable surface downgrades: try Botasaurus once, then drop.
 
 ## Botasaurus
@@ -39,7 +39,7 @@ Invariants for `new` / `repair` / batch campaigns. Quality gate: [AGENTS.md](../
 ## MCP / CLI drift
 
 - Cursor catalog stale after gem upgrade: old `*_url` tool names, `-32602` on calls, new `inspect` “not found” — reload MCP/Cursor; use CLI until catalog shows bare verbs ([curation-verbs.md](curation-verbs.md)).
-- `scrape` with `auto`: do not retry explicit `faraday` after empty auto (chain already ran).
+- `scrape` with `auto`: do not retry explicit `default` after empty auto (chain already ran).
 
 ## Selectors
 

.agents/skills/html2rss-config/reference/repair.md

@@ -6,7 +6,7 @@ Fix one existing config. Runtime debugging: [AGENTS.md](../../../../AGENTS.md) 
 
 1. Read YAML: `channel.url`, selectors, `strategy`.
 2. `scripts/check_config <path>` (or `html2rss validate` + `apply`).
-3. Zero items on Faraday → Botasaurus (`check_config … --botasaurus`); if works → `strategy: botasaurus` + `register_botasaurus`.
+3. Zero items on default strategy → Botasaurus (`check_config … --botasaurus`); if works → `strategy: botasaurus` + `register_botasaurus`.
 4. Still wrong → Chrome MCP snapshot; confirm item boundary and post-redirect URL.
 5. CLI `apply` OK but fetch spec fails → request-strategy mismatch; prefer Botasaurus or drop.
 6. Botasaurus 422/504 → [pitfalls.md](pitfalls.md).

.agents/skills/html2rss-config/scripts/add_config

@@ -6,8 +6,8 @@ require 'uri'
 require 'fileutils'
 require 'json'
 require 'nokogiri'
-require 'faraday'
-require 'faraday/follow_redirects'
+require 'httpx'
+require 'httpx/plugins/follow_redirects'
 require 'public_suffix'
 require_relative 'html2rss_api'
 
@@ -163,15 +163,18 @@ def infer_topics(url, title)
 end
 
 def fetch_html(url)
-  conn = Faraday.new(ssl: { verify: false }, request: { open_timeout: 4, timeout: 8 }) do |f|
-    f.response :follow_redirects, limit: 3
-    f.adapter Faraday.default_adapter
-  end
-  res = conn.get(url, {}, {
-    'User-Agent' => USER_AGENT,
-    'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
-    'Accept-Language' => 'en-US,en;q=0.9,de;q=0.8'
+  session = HTTPX.plugin(:follow_redirects).with(
+    ssl: { verify_mode: OpenSSL::SSL::VERIFY_NONE },
+    timeout: { connect_timeout: 4, request_timeout: 8 },
+    max_redirects: 3
+  )
+  res = session.get(url, headers: {
+    'user-agent' => USER_AGENT,
+    'accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
+    'accept-language' => 'en-US,en;q=0.9,de;q=0.8'
   })
+  return ['', 0] if res.is_a?(HTTPX::ErrorResponse)
+
   [res.body.to_s, res.status]
 rescue StandardError
   ['', 0]

AGENTS.md

@@ -79,11 +79,11 @@ Prefer `user-html2rss` when the tool catalog shows **bare verbs** (`inspect`, `c
 
 **After gem upgrade:** If old names (`inspect_url`, `capture_config`) still appear in Cursor or calls return `-32602`, the server may be fine — **reload MCP / Cursor** until the catalog matches. Until then, use CLI (`html2rss …` from `../html2rss`) or skill scripts — authoritative fallback.
 
-**Strategy:** `scrape`/`capture` + `auto` run Faraday → Botasaurus (don’t retry `faraday` after `auto`). `inspect` + `auto` uses Faraday only; pin `botasaurus` for JS-heavy inspect.
+**Strategy:** `scrape`/`capture` + `auto` run default (HTTPX) → Botasaurus (don’t retry `default` after `auto`). `inspect` + `auto` uses default only; pin `botasaurus` for JS-heavy inspect.
 
 ## Chrome MCP
 
-Use Chrome MCP when the static HTML is unclear, the page is hydrated, or Faraday fetch returns zero items while the browser shows a valid list.
+Use Chrome MCP when the static HTML is unclear, the page is hydrated, or default fetch returns zero items while the browser shows a valid list.
 
 Recommended sequence:
 
@@ -105,7 +105,7 @@ If Chrome MCP is unavailable (`Transport closed` or page-lock errors), do this r
 Use Botasaurus when:
 
 - the page is JS-rendered
-- Faraday fetch returns zero items but Chrome shows a valid repeated list
+- default fetch returns zero items but Chrome shows a valid repeated list
 - the site is bot-sensitive enough that static fetch is unreliable
 
 Local Botasaurus notes:
@@ -132,7 +132,7 @@ Assume the `html2rss` CLI is available on `PATH` when working against the siblin
 6. Tighten selectors until the feed output is clean.
 7. Run repo validation and non-fetch tests.
 8. Run the appropriate fetch lane:
-   - plain fetch for static or Faraday-backed configs
+   - plain fetch for static or default-backed configs
    - Botasaurus fetch for JS-heavy or Botasaurus-backed configs
 
 ## Quality Gate
@@ -175,7 +175,7 @@ make test
 
 6. Focused fetch verification:
 
-- Faraday-backed candidate:
+- Default-backed candidate:
 
 ```bash
 bundle exec rspec --tag fetch --example 'example.com/feed.yml' spec/html2rss/configs_dynamic_spec.rb
@@ -219,7 +219,7 @@ Use the core CLI as the authority for single-config debugging. The quickest loop
 4. adjust selectors
 5. rerun
 
-If Botasaurus works but Faraday does not, keep the config narrow and classify it as Botasaurus-backed instead of trying to rescue it with brittle tweaks.
+If Botasaurus works but default strategy does not, keep the config narrow and classify it as Botasaurus-backed instead of trying to rescue it with brittle tweaks.
 
 Additional high-value checks:
 
@@ -240,7 +240,7 @@ html2rss scrape 'https://example.com'
 Use it to:
 
 - discover likely repeated item selectors
-- compare Faraday and Botasaurus behavior quickly
+- compare default and Botasaurus behavior quickly
 - decide whether a site belongs in the curated set at all
 
 Do not ship raw scrape output without manual tightening into a curated config.

Gemfile

@@ -4,7 +4,7 @@ source 'https://rubygems.org'
 
 git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
 
-gem 'html2rss', github: 'html2rss/html2rss', ref: '9ac61d13'
+gem 'html2rss', github: 'html2rss/html2rss', ref: '4260c842'
 
 group :development do
   gem 'nokogiri'

Gemfile.lock

@@ -1,23 +1,20 @@
 GIT
   remote: https://github.com/html2rss/html2rss
-  revision: 9ac61d13b96ff86cf2a843ba25e35584134696e1
-  ref: 9ac61d13
+  revision: 4260c842a5632fbef712cf3bee3a4fddcfee3654
+  ref: 4260c842
   specs:
-    html2rss (0.28.0)
+    html2rss (0.29.1)
       addressable (~> 2.7)
       brotli
       dry-validation
-      faraday (> 2.0.1, < 3.0)
-      faraday-follow_redirects
-      faraday-gzip (~> 3)
+      httpx (~> 1.8)
       kramdown
       mcp (~> 1.2)
       mime-types (> 3.0)
       nokogiri (>= 1.10, < 2.0)
       rack (~> 3.0)
       rackup (~> 2.0)
       regexp_parser
-      reverse_markdown (~> 3.0)
       rss
       sanitize
       thor
@@ -77,18 +74,10 @@ GEM
       dry-initializer (~> 3.2)
       dry-schema (~> 1.14)
       zeitwerk (~> 2.6)
-    faraday (2.14.3)
-      faraday-net_http (>= 2.0, < 3.5)
-      json
-      logger
-    faraday-follow_redirects (0.5.0)
-      faraday (>= 1, < 3)
-    faraday-gzip (3.1.0)
-      faraday (>= 2.0, < 3)
-      zlib (~> 3.0)
-    faraday-net_http (3.4.4)
-      net-http (~> 0.5)
     hana (1.3.7)
+    http-2 (1.2.2)
+    httpx (1.8.3)
+      http-2 (>= 1.2.0)
     json (2.21.2)
     json_schemer (2.5.0)
       bigdecimal
@@ -106,8 +95,6 @@ GEM
       logger
       mime-types-data (~> 3.2025, >= 3.2025.0507)
     mime-types-data (3.2026.0701)
-    net-http (0.9.1)
-      uri (>= 0.11.1)
     nokogiri (1.19.4-arm64-darwin)
       racc (~> 1.4)
     nokogiri (1.19.4-x86_64-darwin)
@@ -126,8 +113,6 @@ GEM
       rack (>= 3)
     rainbow (3.1.1)
     regexp_parser (2.12.0)
-    reverse_markdown (3.0.2)
-      nokogiri
     rexml (3.4.4)
     rspec (3.13.2)
       rspec-core (~> 3.13.0)
@@ -174,10 +159,8 @@ GEM
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
     unicode-display_width (2.6.0)
-    uri (1.1.1)
     webrick (1.9.2)
     zeitwerk (2.8.3)
-    zlib (3.2.3)
 
 PLATFORMS
   arm64-darwin-23

lib/html2rss/configs/asml.com/press-releases.yml

@@ -16,6 +16,6 @@ selectors:
     selector: 'a.related-content-card[href*="/en/news/press-releases/20"]'
     enhance: false
   title:
-    selector: span.related-content-card-title
+    selector: .related-content-card-title
   url:
     extractor: href

spec/bin/validate_configs_spec.rb

@@ -20,16 +20,28 @@
     end
   end
 
-  def with_temp_config
+  it 'fails validation when config violates schema' do
+    with_temp_config('channel: {}') do |dir|
+      expect { run_script(dir) }.to raise_error(an_instance_of(SystemExit).and(having_attributes(status: 1)))
+    end
+  end
+
+  it 'fails validation when YAML is malformed' do
+    with_temp_config('invalid: [yaml: {') do |dir|
+      expect { run_script(dir) }.to raise_error(an_instance_of(SystemExit).and(having_attributes(status: 1)))
+    end
+  end
+
+  def with_temp_config(content = valid_config)
     Dir.mktmpdir do |dir|
-      write_config(dir)
+      write_config(dir, content)
       yield dir
     end
   end
 
-  def write_config(dir)
+  def write_config(dir, content = valid_config)
     FileUtils.mkdir_p(File.join(dir, File.dirname(config_path)))
-    File.write(File.join(dir, config_path), valid_config)
+    File.write(File.join(dir, config_path), content)
   end
 
   def run_script(dir)