From 46abcf77e02815667ffd87c294efff1c041191e2 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 20 Sep 2026 11:33:14 +0200 Subject: [PATCH] docs(web): document selector refinement recovery --- .../guides/refine-selectors.mdx | 29 +++++++++++ .../reference/env-variables.mdx | 51 ++++++++++--------- 2 files changed, 55 insertions(+), 25 deletions(-) create mode 100644 src/content/docs/web-application/guides/refine-selectors.mdx diff --git a/src/content/docs/web-application/guides/refine-selectors.mdx b/src/content/docs/web-application/guides/refine-selectors.mdx new file mode 100644 index 00000000..983d35f8 --- /dev/null +++ b/src/content/docs/web-application/guides/refine-selectors.mdx @@ -0,0 +1,29 @@ +--- +title: "Refine a generated feed" +description: "Recover a generated feed when it finds the wrong list, misses fields, or starts from the wrong page." +--- + +Use **Refine selectors** after automatic feed generation when the result is empty, follows the wrong list of links, or misses the title, link, or published date. Refinement is a recovery step for one generated feed, not a YAML authoring workflow. + +## Before You Start + +Your `html2rss-web` instance must have `STUDIO_ENABLED=true`, and you must have entered a valid access token. Live suggestions and previews also require `AUTO_SOURCE_ENABLED=true`. + +If refinement is unavailable, the result page does not show the action. Opening an old refinement link on an instance where the studio is disabled returns you to feed creation. + +## Refine the Result + +1. Generate a feed from a listing page and open **Refine selectors** from the result. +2. Check the page URL at the top. If you started from the wrong listing, edit it first; the studio requests a new suggestion for that page. +3. Check **Items selector**. It should identify the repeated article or entry container, such as `article` or `.post`. +4. Use the suggested selector when it identifies the correct list. Enable enhanced extraction only when the page needs additional semantic field detection. +5. Open a missing field to set a selector for its title, link, or published date. +6. Check the sample items, then choose **Save and generate**. + +The result is successful when the sample contains the intended entries and the generated feed preserves their useful title and link. + +## Keep the Recovery Focused + +Prefer changing the listing URL before adding more selectors. A category, newsroom, changelog, or archive page usually gives better results than a generic homepage. + +**View as YAML** shows the current selectors fragment for inspection. It is not a complete custom-feed config and does not publish a Feed Directory entry. For a reviewable, version-controlled scraping rule, use [Creating Custom Feeds](/creating-custom-feeds/). diff --git a/src/content/docs/web-application/reference/env-variables.mdx b/src/content/docs/web-application/reference/env-variables.mdx index 82c9712d..15535d13 100644 --- a/src/content/docs/web-application/reference/env-variables.mdx +++ b/src/content/docs/web-application/reference/env-variables.mdx @@ -5,31 +5,32 @@ description: "Configuration reference for html2rss-web environment variables." ## Supported ENV variables -| Name | Description | -| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `HTML2RSS_SECRET_KEY` | required in production; development/test gets a temporary default | -| `HTML2RSS_ACCESS_TOKEN` | token for the main feed-creation flow; paste the same value into the web UI when prompted | -| `HEALTH_CHECK_TOKEN` | optional bearer token for authenticated `GET /api/v1/health`; not part of the first-run quickstart; use `/api/v1/health/ready` for standard readiness checks | -| `BUILD_TAG` | release metadata used in logs; published Docker images set this to the release version | -| `GIT_SHA` | deployed commit metadata used in logs; published Docker images set this to the released commit | -| `SENTRY_DSN` | optional; enables Sentry errors/logs when set | -| `BOTASAURUS_SCRAPER_URL` | optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass | -| `AUTO_SOURCE_ENABLED` | `true` by default in development/test, `false` otherwise | -| `ASYNC_FEED_REFRESH_ENABLED` | optional boolean; default `false` | -| `ASYNC_FEED_REFRESH_STALE_FACTOR` | optional integer `>= 1`; default `3` | -| `FEEDS_CACHE_MAX_SIZE` | optional integer `>= 1`; default `500`; maximum number of feeds to cache in memory before eviction | -| `RATE_LIMIT_ENABLED` | optional boolean; default `true` (except `false` in `test`); enables or disables client IP rate limiting | -| `RATE_LIMIT_MAX_REQUESTS` | optional integer `>= 1`; default `60`; maximum requests allowed per IP per window | -| `RATE_LIMIT_WINDOW_SECONDS` | optional integer `>= 1`; default `60`; rate limiter sliding window duration in seconds | -| `RETRY_AFTER_TIMEOUT_SECONDS` | optional integer `>= 1`; default `300`; Retry-After header backoff duration returned on timeouts | -| `PORT` | app listen port; compose uses `4000` | -| `RACK_ENV` | Rack environment; quickstart uses `development`, production examples use `production` | -| `REQUEST_TIMEOUT_SECONDS` | optional; total web-request timeout in seconds; default `55` | -| `WEB_CONCURRENCY` | optional integer `>= 1`; default `2` in production (`1` in development); number of web worker processes | -| `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` | optional; HTTP client cap for POST `/scrape` in seconds; default `45`; mirrors `SCRAPE_TIMEOUT_SECONDS` on botasaurus-scrape-api | -| `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` | optional; maximum `wait_timeout_seconds` in feed YAML; default `30`; mirrors `SCRAPE_WORK_TIMEOUT_SECONDS` on botasaurus-scrape-api | -| `SCRAPE_WORK_TIMEOUT_SECONDS` | optional on botasaurus-scrape-api; post-boot navigate, selector wait, and scroll budget in seconds; default `30`; set on the scraper service in compose | -| `HTML2RSS_TOTAL_TIMEOUT_SECONDS` | optional; internal gem feed-build timeout in seconds; default `50` | +| Name | Description | +| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `HTML2RSS_SECRET_KEY` | required in production; development/test gets a temporary default | +| `HTML2RSS_ACCESS_TOKEN` | token for the main feed-creation flow; paste the same value into the web UI when prompted | +| `HEALTH_CHECK_TOKEN` | optional bearer token for authenticated `GET /api/v1/health`; not part of the first-run quickstart; use `/api/v1/health/ready` for standard readiness checks | +| `BUILD_TAG` | release metadata used in logs; published Docker images set this to the release version | +| `GIT_SHA` | deployed commit metadata used in logs; published Docker images set this to the released commit | +| `SENTRY_DSN` | optional; enables Sentry errors/logs when set | +| `BOTASAURUS_SCRAPER_URL` | optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass | +| `AUTO_SOURCE_ENABLED` | `true` by default in development/test, `false` otherwise | +| `STUDIO_ENABLED` | `true` by default in development/test, `false` otherwise; enables token-holder selector refinement; live preview and suggestions also require `AUTO_SOURCE_ENABLED=true` | +| `ASYNC_FEED_REFRESH_ENABLED` | optional boolean; default `false` | +| `ASYNC_FEED_REFRESH_STALE_FACTOR` | optional integer `>= 1`; default `3` | +| `FEEDS_CACHE_MAX_SIZE` | optional integer `>= 1`; default `500`; maximum number of feeds to cache in memory before eviction | +| `RATE_LIMIT_ENABLED` | optional boolean; default `true` (except `false` in `test`); enables or disables client IP rate limiting | +| `RATE_LIMIT_MAX_REQUESTS` | optional integer `>= 1`; default `60`; maximum requests allowed per IP per window | +| `RATE_LIMIT_WINDOW_SECONDS` | optional integer `>= 1`; default `60`; rate limiter sliding window duration in seconds | +| `RETRY_AFTER_TIMEOUT_SECONDS` | optional integer `>= 1`; default `300`; Retry-After header backoff duration returned on timeouts | +| `PORT` | app listen port; compose uses `4000` | +| `RACK_ENV` | Rack environment; quickstart uses `development`, production examples use `production` | +| `REQUEST_TIMEOUT_SECONDS` | optional; total web-request timeout in seconds; default `55` | +| `WEB_CONCURRENCY` | optional integer `>= 1`; default `2` in production (`1` in development); number of web worker processes | +| `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` | optional; HTTP client cap for POST `/scrape` in seconds; default `45`; mirrors `SCRAPE_TIMEOUT_SECONDS` on botasaurus-scrape-api | +| `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` | optional; maximum `wait_timeout_seconds` in feed YAML; default `30`; mirrors `SCRAPE_WORK_TIMEOUT_SECONDS` on botasaurus-scrape-api | +| `SCRAPE_WORK_TIMEOUT_SECONDS` | optional on botasaurus-scrape-api; post-boot navigate, selector wait, and scroll budget in seconds; default `30`; set on the scraper service in compose | +| `HTML2RSS_TOTAL_TIMEOUT_SECONDS` | optional; internal gem feed-build timeout in seconds; default `50` | ## Timeout Chain