Skip to content

docs(web): document selector refinement recovery #1266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gildesmarais wants to merge 1 commit into
base: main
Choose a base branch
from
+55 −25
Open

docs(web): document selector refinement recovery #1266

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions src/content/docs/web-application/guides/refine-selectors.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
title: "Refine a generated feed"
description: "Recover a generated feed when it finds the wrong list, misses fields, or starts from the wrong page."
---

Use **Refine selectors** after automatic feed generation when the result is empty, follows the wrong list of links, or misses the title, link, or published date. Refinement is a recovery step for one generated feed, not a YAML authoring workflow.

## Before You Start

Your `html2rss-web` instance must have `STUDIO_ENABLED=true`, and you must have entered a valid access token. Live suggestions and previews also require `AUTO_SOURCE_ENABLED=true`.

If refinement is unavailable, the result page does not show the action. Opening an old refinement link on an instance where the studio is disabled returns you to feed creation.

## Refine the Result

1. Generate a feed from a listing page and open **Refine selectors** from the result.
2. Check the page URL at the top. If you started from the wrong listing, edit it first; the studio requests a new suggestion for that page.
3. Check **Items selector**. It should identify the repeated article or entry container, such as `article` or `.post`.
4. Use the suggested selector when it identifies the correct list. Enable enhanced extraction only when the page needs additional semantic field detection.
5. Open a missing field to set a selector for its title, link, or published date.
6. Check the sample items, then choose **Save and generate**.

The result is successful when the sample contains the intended entries and the generated feed preserves their useful title and link.

## Keep the Recovery Focused

Prefer changing the listing URL before adding more selectors. A category, newsroom, changelog, or archive page usually gives better results than a generic homepage.

**View as YAML** shows the current selectors fragment for inspection. It is not a complete custom-feed config and does not publish a Feed Directory entry. For a reviewable, version-controlled scraping rule, use [Creating Custom Feeds](/creating-custom-feeds/).
51 changes: 26 additions & 25 deletions src/content/docs/web-application/reference/env-variables.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,31 +5,32 @@ description: "Configuration reference for html2rss-web environment variables."

## Supported ENV variables

| Name | Description |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `HTML2RSS_SECRET_KEY` | required in production; development/test gets a temporary default |
| `HTML2RSS_ACCESS_TOKEN` | token for the main feed-creation flow; paste the same value into the web UI when prompted |
| `HEALTH_CHECK_TOKEN` | optional bearer token for authenticated `GET /api/v1/health`; not part of the first-run quickstart; use `/api/v1/health/ready` for standard readiness checks |
| `BUILD_TAG` | release metadata used in logs; published Docker images set this to the release version |
| `GIT_SHA` | deployed commit metadata used in logs; published Docker images set this to the released commit |
| `SENTRY_DSN` | optional; enables Sentry errors/logs when set |
| `BOTASAURUS_SCRAPER_URL` | optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass |
| `AUTO_SOURCE_ENABLED` | `true` by default in development/test, `false` otherwise |
| `ASYNC_FEED_REFRESH_ENABLED` | optional boolean; default `false` |
| `ASYNC_FEED_REFRESH_STALE_FACTOR` | optional integer `>= 1`; default `3` |
| `FEEDS_CACHE_MAX_SIZE` | optional integer `>= 1`; default `500`; maximum number of feeds to cache in memory before eviction |
| `RATE_LIMIT_ENABLED` | optional boolean; default `true` (except `false` in `test`); enables or disables client IP rate limiting |
| `RATE_LIMIT_MAX_REQUESTS` | optional integer `>= 1`; default `60`; maximum requests allowed per IP per window |
| `RATE_LIMIT_WINDOW_SECONDS` | optional integer `>= 1`; default `60`; rate limiter sliding window duration in seconds |
| `RETRY_AFTER_TIMEOUT_SECONDS` | optional integer `>= 1`; default `300`; Retry-After header backoff duration returned on timeouts |
| `PORT` | app listen port; compose uses `4000` |
| `RACK_ENV` | Rack environment; quickstart uses `development`, production examples use `production` |
| `REQUEST_TIMEOUT_SECONDS` | optional; total web-request timeout in seconds; default `55` |
| `WEB_CONCURRENCY` | optional integer `>= 1`; default `2` in production (`1` in development); number of web worker processes |
| `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` | optional; HTTP client cap for POST `/scrape` in seconds; default `45`; mirrors `SCRAPE_TIMEOUT_SECONDS` on botasaurus-scrape-api |
| `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` | optional; maximum `wait_timeout_seconds` in feed YAML; default `30`; mirrors `SCRAPE_WORK_TIMEOUT_SECONDS` on botasaurus-scrape-api |
| `SCRAPE_WORK_TIMEOUT_SECONDS` | optional on botasaurus-scrape-api; post-boot navigate, selector wait, and scroll budget in seconds; default `30`; set on the scraper service in compose |
| `HTML2RSS_TOTAL_TIMEOUT_SECONDS` | optional; internal gem feed-build timeout in seconds; default `50` |
| Name | Description |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `HTML2RSS_SECRET_KEY` | required in production; development/test gets a temporary default |
| `HTML2RSS_ACCESS_TOKEN` | token for the main feed-creation flow; paste the same value into the web UI when prompted |
| `HEALTH_CHECK_TOKEN` | optional bearer token for authenticated `GET /api/v1/health`; not part of the first-run quickstart; use `/api/v1/health/ready` for standard readiness checks |
| `BUILD_TAG` | release metadata used in logs; published Docker images set this to the release version |
| `GIT_SHA` | deployed commit metadata used in logs; published Docker images set this to the released commit |
| `SENTRY_DSN` | optional; enables Sentry errors/logs when set |
| `BOTASAURUS_SCRAPER_URL` | optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass |
| `AUTO_SOURCE_ENABLED` | `true` by default in development/test, `false` otherwise |
| `STUDIO_ENABLED` | `true` by default in development/test, `false` otherwise; enables token-holder selector refinement; live preview and suggestions also require `AUTO_SOURCE_ENABLED=true` |
| `ASYNC_FEED_REFRESH_ENABLED` | optional boolean; default `false` |
| `ASYNC_FEED_REFRESH_STALE_FACTOR` | optional integer `>= 1`; default `3` |
| `FEEDS_CACHE_MAX_SIZE` | optional integer `>= 1`; default `500`; maximum number of feeds to cache in memory before eviction |
| `RATE_LIMIT_ENABLED` | optional boolean; default `true` (except `false` in `test`); enables or disables client IP rate limiting |
| `RATE_LIMIT_MAX_REQUESTS` | optional integer `>= 1`; default `60`; maximum requests allowed per IP per window |
| `RATE_LIMIT_WINDOW_SECONDS` | optional integer `>= 1`; default `60`; rate limiter sliding window duration in seconds |
| `RETRY_AFTER_TIMEOUT_SECONDS` | optional integer `>= 1`; default `300`; Retry-After header backoff duration returned on timeouts |
| `PORT` | app listen port; compose uses `4000` |
| `RACK_ENV` | Rack environment; quickstart uses `development`, production examples use `production` |
| `REQUEST_TIMEOUT_SECONDS` | optional; total web-request timeout in seconds; default `55` |
| `WEB_CONCURRENCY` | optional integer `>= 1`; default `2` in production (`1` in development); number of web worker processes |
| `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` | optional; HTTP client cap for POST `/scrape` in seconds; default `45`; mirrors `SCRAPE_TIMEOUT_SECONDS` on botasaurus-scrape-api |
| `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` | optional; maximum `wait_timeout_seconds` in feed YAML; default `30`; mirrors `SCRAPE_WORK_TIMEOUT_SECONDS` on botasaurus-scrape-api |
| `SCRAPE_WORK_TIMEOUT_SECONDS` | optional on botasaurus-scrape-api; post-boot navigate, selector wait, and scroll budget in seconds; default `30`; set on the scraper service in compose |
| `HTML2RSS_TOTAL_TIMEOUT_SECONDS` | optional; internal gem feed-build timeout in seconds; default `50` |

## Timeout Chain

Expand Down
Loading