feat: allow refinement of feed results

.agents/plan/studio-ceremony-cut.md

@@ -0,0 +1,211 @@
+# Implementation Plan: studio-ceremony-cut
+
+Intent: inline
+Circuit breaker approved: false
+
+## Product stance
+
+Recommendation: Build Now
+
+- Persona and path: Sam’s documented SPA path stays Create → token (if needed) → Result → Copy feed URL. This tranche does not add a journey step. It removes dead ceremony and cache polish left from the prior studio cleanup.
+- Interaction budget: zero new clicks. Suggest, preview settle, and first-edit skip stay as today.
+- Concept budget: zero new concepts. Selector refinement stays on `#/result/:token`. There is still no `#/refine` route.
+- Forced challenge: PreviewPageCache becomes process-lifetime (no TTL). Same URL still replays until replaced or process exit. That is simpler and matches how operators already reason about a single-slot cache.
+- Non-goals: no gem-schema OpenAPI generation; no dropping closed candidate/`RefinedSelectorsDocument` stamps; no dropping View-as-YAML; no merging `JsonBody` back; no collapsing field setters; no removing preview settle / first-edit skip.
+
+## Intent constraints
+
+- Earned seams stay: `RefinedSelectors`, `GeneratorInput`, `StudioRequest`/`JsonBody` ownership, `selectorDraft` part-toggle, `api/http` → service → hook layering, create weighting, save-phase recovery, result-mount follow.
+- SPA suggest sends `{ url }` only. Server `hint` + auth-chain oversized-hint coverage stay.
+- `PreviewPageCache` is process-lifetime single-slot tempfile + `at_exit`. No `TTL_SECONDS`, `fresh?`, or body-equality dedupe.
+- OpenAPI closed schemas stay. Only the duplicated `post_process_*` comment block is removed.
+- `STUDIO_PREVIEW_SETTLE_MS` and `revision === 0` first-preview skip stay.
+- View-as-YAML stays. `ConfigStudio` is not split into more files.
+- `StudioState`: `issues` only on `invalid`, `message` only on `failed`; `live` stays orthogonal/flat. Public hook return shape stays stable for tests.
+- User-facing parse failure stays `COPY.unableToCompleteCreation`. `StudioRequestError` field naming stays.
+
+## Classification and architecture
+
+- Classification: `surgical` (LOC/dead-state). Inherited from the approved plan. Do not relitigate.
+- Runtime route: `typescript-dev` and a small `ruby-dev` cache/spec touch. Not Rails.
+- Architecture branch: axioms ride-along; no architecture redesign.
+- Ownership decisions:
+  - `useStudio` drops dead `enhanceTouchedReference` / `itemsTouchedReference`.
+  - `studioService` / `api/http/feeds` drop SPA `itemsSelector`; suggest body is `{ url }`.
+  - `studioService` owns local parse helpers; no new deps.
+  - `PreviewPageCache` owns process-lifetime slot + tempfile; no TTL/dedupe.
+  - `OpenapiStudioSchemas` drops duplicated comment only.
+  - `useStudio` phase-discriminates `issues`/`message`; `live` remains flat.
+
+## Acceptance
+
+- Given a suggest call from the SPA, when it posts, then the body is `{ url }` only.
+- Given a preview of the same URL twice, when the slot still holds that URL and the file exists, then the second call replays without refetch. Given process exit, when the slot still exists, then the temp file is removed. There is no TTL-based refetch.
+- Given `StudioState` phase `invalid`, when readers ask for issues, then `issues` is present. Given phase `failed`, when readers ask for the message, then `message` is present. Given other phases, those fields are absent.
+- Given a malformed studio payload, when parsing fails, then the thrown error names the field and the message stays `COPY.unableToCompleteCreation`.
+- Given the OpenAPI studio schemas file, when the duplicated `post_process_item` comment is removed, then generation and the runtime contract still pass.
+
+## Shared files
+
+These paths appear in more than one phase because each phase changes a different fact:
+
+- `frontend/src/studio/useStudio.ts` — Phase 02 deletes touched refs; Phase 06 phase-discriminates issues/message.
+- `frontend/src/studio/studioService.ts` — Phase 02 drops itemsSelector; Phase 03 compresses parsers.
+- `frontend/src/__tests__/useStudio.test.ts` — Phase 02 and Phase 06 adjust call sites/assertions.
+- `frontend/src/__tests__/studioService.test.ts` — Phase 02 and Phase 03 adjust call sites.
+
+## Conditional risks
+
+- Observability: N/A. No APM, tracing, or logging contract changes.
+- Security: authn/authz rules stay on `StudioRequest`. Suggest hint remains server-side; SPA stops sending `items_selector`. `spec/html2rss/web/api/studio_auth_chain_spec.rb` remains the security chain for oversized hint.
+- Pre-ship: Phase 07 runs `make ci-ready` inside the Dev Container. No push unless asked.
+- All `verification_gate` commands run inside the Dev Container. A host-only exit is not a pass.
+
+<!-- phase:start -->
+
+## Phase 01 — Persist carrier
+
+Write this carrier with phase DTOs (`target_files`, `read_context`, `verification_gate`, `commit_message`). Circuit breaker: false.
+
+target_files:
+
+- `.agents/plan/studio-ceremony-cut.md`
+
+read_context:
+
+- `/Users/gil/.cursor/plans/studio_ceremony_cut_aa8b5e77.plan.md`
+- `.agents/plan/studio-architecture-cleanup.md`
+
+verification_gate: `test -f .agents/plan/studio-ceremony-cut.md`
+commit_message: `docs(plan): persist studio ceremony cut carrier`
+
+<!-- phase:end -->
+
+<!-- phase:start -->
+
+## Phase 02 — Dead refs and suggest hint
+
+In `useStudio.ts`: delete `enhanceTouchedReference` / `itemsTouchedReference` and their writes. In `studioService.ts` + `api/http/feeds.ts`: drop the SPA `itemsSelector` argument; `postSuggestSelectors` sends `{ url }` only (OpenAPI already allows optional `items_selector`). Keep server `hint` + auth-chain oversized-hint test. Update unit test call sites.
+
+target_files:
+
+- `frontend/src/studio/useStudio.ts`
+- `frontend/src/studio/studioService.ts`
+- `frontend/src/api/http/feeds.ts`
+- `frontend/src/__tests__/useStudio.test.ts`
+- `frontend/src/__tests__/studioService.test.ts`
+
+read_context:
+
+- `frontend/src/components/ConfigStudio.tsx`
+- `spec/html2rss/web/api/studio_auth_chain_spec.rb`
+
+verification_gate: `make test-frontend-unit`
+commit_message: `refactor(studio): drop dead refs and SPA suggest hint`
+
+<!-- phase:end -->
+
+<!-- phase:start -->
+
+## Phase 03 — Compress studioService parsers
+
+Add small local helpers (`expectRecord` / `expectString` / `expectArray` or equivalent) in `studioService.ts` and rewrite the repetitive parse* functions to use them. Keep `StudioRequestError` field naming and user-facing `COPY.unableToCompleteCreation`. No new deps.
+
+target_files:
+
+- `frontend/src/studio/studioService.ts`
+- `frontend/src/__tests__/studioService.test.ts`
+
+read_context:
+
+- `frontend/src/journey/copy.ts`
+
+verification_gate: `make test-frontend-unit`
+commit_message: `refactor(studio): compress studioService parsers`
+
+<!-- phase:end -->
+
+<!-- phase:start -->
+
+## Phase 04 — Slim PreviewPageCache
+
+Remove TTL/fresh/dedupe/touch paths; `path_for` only checks URL match + file exists; `store` always writes a new tempfile when storable. Keep `clear!` + `at_exit`. Delete/rewrite TTL and “unchanged temp file” examples in `v1_spec.rb`; keep replay / URL-replace / empty-does-not-clobber coverage. Update TTL mention in docs if present (process-lifetime, not 300s).
+
+target_files:
+
+- `app/web/api/v1/preview_page_cache.rb`
+- `spec/html2rss/web/api/v1_spec.rb`
+- `docs/README.md`
+- `CONTEXT.md`
+
+read_context:
+
+- `app/web/api/v1/preview_feed.rb`
+
+verification_gate: `bundle exec rspec spec/html2rss/web/api/v1_spec.rb`
+commit_message: `refactor(studio): slim PreviewPageCache to process lifetime`
+
+<!-- phase:end -->
+
+<!-- phase:start -->
+
+## Phase 05 — OpenAPI comment dedupe
+
+Remove the duplicated comment on `post_process_item` in `openapi_studio_schemas.rb`. No YAML regen unless a comment somehow affects generation (it will not).
+
+target_files:
+
+- `spec/support/openapi_studio_schemas.rb`
+
+read_context:
+
+- `spec/html2rss/web/openapi_runtime_contract_spec.rb`
+
+verification_gate: `bundle exec rspec spec/html2rss/web/openapi_runtime_contract_spec.rb`
+commit_message: `chore(openapi): dedupe post_process schema comments`
+
+<!-- phase:end -->
+
+<!-- phase:start -->
+
+## Phase 06 — StudioState: issues/message only when needed
+
+Restore a phase-discriminated shape so `issues` exists only on `invalid` and `message` only on `failed` (keep `live` orthogonal/flat as today). Adjust `studioIssues` / `canSaveStudio` / `ConfigStudio` readers if needed. Keep public hook return shape stable for tests.
+
+target_files:
+
+- `frontend/src/studio/useStudio.ts`
+- `frontend/src/components/ConfigStudio.tsx`
+- `frontend/src/__tests__/useStudio.test.ts`
+- `frontend/src/__tests__/ConfigStudio.test.tsx`
+
+read_context:
+
+- `frontend/src/studio/studioService.ts`
+
+verification_gate: `make test-frontend-unit && make lint-js`
+commit_message: `refactor(studio): phase-discriminate issues and message`
+
+<!-- phase:end -->
+
+<!-- phase:start -->
+
+## Phase 07 — Final gate
+
+`make ci-ready` in the Dev Container. No push unless asked.
+
+target_files: []
+
+read_context:
+
+- `AGENTS.md`
+- `Makefile`
+
+verification_gate: `make ci-ready`
+commit_message: `chore(studio): record ceremony cut ci-ready`
+
+<!-- phase:end -->
+
+## Out of scope
+
+Gem-schema OpenAPI generation; dropping closed candidate/`RefinedSelectorsDocument` stamps; dropping YAML UI; merging `JsonBody` back; collapsing field setters into one component API; preview settle / first-edit skip removal.

.env.example

@@ -17,6 +17,12 @@ HTML2RSS_ACCESS_TOKEN=
 # Keep enabled to allow pasting any webpage URL to generate an RSS feed.
 AUTO_SOURCE_ENABLED=true
 
+# OPTIONAL: Config refinement studio (off in production unless set to true).
+# Exposes POST /api/v1/feeds/validate, /preview and /suggest_selectors. All three
+# require HTML2RSS_ACCESS_TOKEN; preview and suggest_selectors also fetch the
+# submitted URL live and need AUTO_SOURCE_ENABLED=true.
+# STUDIO_ENABLED=false
+
 # COMPANION SCRAPER:
 # URL of the companion Botasaurus scraper API service in docker-compose.
 BOTASAURUS_SCRAPER_URL=http://botasaurus:4010

AGENTS.md

@@ -58,19 +58,50 @@ See [docs/design-system.md](docs/design-system.md) for visual rules.
 
 Public feed-directory metadata for embedded and local configs.
 
-| Item | Detail |
-| --- | --- |
-| Endpoint | `GET /api/v1/configs` |
-| Flag | `CONFIG_CATALOG_ENABLED` (default `true`; set `false` to disable) |
-| Disabled response | `404` with `{ "error": "catalog_disabled" }` |
-| Embedded entries | `Html2rss::Configs::Catalog.entries` — do not re-walk YAML in the handler |
-| Local entries | `Catalog::Merge` includes `feeds.yml` feeds only when `directory.title` is set |
-| Starter feeds (UI) | Server `meta.starters` via `Catalog::Starters.pick`; frontend `selectStarterFeeds(entries, starters)` |
-| Catalog find | `findCatalogEntries` → multi-hit list under create URL; demotes `empty`/`error`; links via `catalogFeedHref` |
-| last_result | Required on every catalog entry (`ok`/`empty`/`error`/`unknown`); process-local `Feeds::LastResults` |
-| catalog_version | `2` (clients fail-closed on other versions) |
-| CORS | Route-scoped on `/api/v1/configs` only (`GET`, `OPTIONS`) |
-| Root metadata | `GET /api/v1/` exposes `instance.catalog: { enabled, url }` |
-| Contract SSOT | Request specs under `spec/html2rss/web/api/v1_spec.rb` and generated `public/openapi.yaml` |
+| Item               | Detail                                                                                                       |
+| ------------------ | ------------------------------------------------------------------------------------------------------------ |
+| Endpoint           | `GET /api/v1/configs`                                                                                        |
+| Flag               | `CONFIG_CATALOG_ENABLED` (default `true`; set `false` to disable)                                            |
+| Disabled response  | `404` with `{ "error": "catalog_disabled" }`                                                                 |
+| Embedded entries   | `Html2rss::Configs::Catalog.entries` — do not re-walk YAML in the handler                                    |
+| Local entries      | `Catalog::Merge` includes `feeds.yml` feeds only when `directory.title` is set                               |
+| Starter feeds (UI) | Server `meta.starters` via `Catalog::Starters.pick`; frontend `selectStarterFeeds(entries, starters)`        |
+| Catalog find       | `findCatalogEntries` → multi-hit list under create URL; demotes `empty`/`error`; links via `catalogFeedHref` |
+| last_result        | Required on every catalog entry (`ok`/`empty`/`error`/`unknown`); process-local `Feeds::LastResults`         |
+| catalog_version    | `2` (clients fail-closed on other versions)                                                                  |
+| CORS               | Route-scoped on `/api/v1/configs` only (`GET`, `OPTIONS`)                                                    |
+| Root metadata      | `GET /api/v1/` exposes `instance.catalog: { enabled, url }` and `instance.studio: { enabled }`              |
+| Contract SSOT      | Request specs under `spec/html2rss/web/api/v1_spec.rb` and generated `public/openapi.yaml`                   |
 
 After handler or envelope changes: `make openapi` and `make ci-ready`.
+
+## Config studio API
+
+Token-holder refinement of the selectors subtree, then generate via create.
+
+| Item              | Detail                                                                                                                                |
+| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
+| Endpoints         | `POST /api/v1/feeds/validate`, `POST /api/v1/feeds/preview`, `POST /api/v1/feeds/suggest_selectors`                                   |
+| Create            | `POST /api/v1/feeds` accepts optional `selectors` (same access token and flag as validate; URL-only create is unaffected by the flag) |
+| Flag              | `STUDIO_ENABLED` (development/test default `true`; otherwise `false`); advertised as `instance.studio.enabled` on `GET /api/v1` |
+| Disabled response | `403` when the flag is off                                                                                                            |
+| Auth              | Access token required on validate, preview, suggest, and create                                                                       |
+| Live fetch        | Preview and suggest also need `AUTO_SOURCE_ENABLED`                                                                                   |
+| Frontend          | Successful results refine inline on `#/result/:token`. There is no `#/refine` route. |
+| Contract SSOT     | Request specs under `spec/html2rss/web/api/v1_spec.rb` and generated `public/openapi.yaml`                                            |
+
+After handler or envelope changes: `make openapi` and `make ci-ready`.
+
+Do not expand hand-stamped studio schemas in `spec/support/openapi_studio_schemas.rb` without deriving from a runtime or gem owner. Full gem JSON Schema → OpenAPI stamp generation stays future work.
+
+## Generated OpenAPI SPA transport
+
+The frontend JSON API is the generated Hey API client, not hand-written `fetch`.
+
+| Item          | Detail                                                                                                                                |
+| ------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
+| Regen         | `make openapi-client` (`frontend/openapi-ts.config.ts`). Never hand-edit `frontend/src/api/generated/**`                              |
+| Value imports | Only `frontend/src/api/http/*.ts` and `src/__tests__/apiHttp*.test.ts`. Type-only imports are allowed elsewhere                       |
+| Layers        | Generated owns wire. Adapters in `frontend/src/api/http/` own transport (`unwrap` → `{ ok, status, body }`). Parsers own domain types |
+| baseUrl       | `/api/v1` same-origin. Do not change public OpenAPI `servers` or `RSpec::OpenAPI.servers`                                             |
+| Preview       | JSON Feed stays `fetch(json_public_url)`. Do not use `renderFeedByToken`                                                              |

CONTEXT.md

@@ -28,6 +28,12 @@ On create, `findCatalogEntries` matches the typed query against the catalog (URL
 ### CreateEntry remount
 Visiting create (including hashbang `#!/…` → `#/…`) bumps `createEntryKey` so the create surface remounts. Remount alone does not auto-submit; auto-submit requires `prefillUrl`.
 
+### Config Studio
+Selector refinement for a token holder when studio is enabled. On a successful result, refinement is available on demand behind demoted **Refine selectors** on `#/result/:token` (collapsed by default; one meadow; live studio output replaces the result preview after an edit). The listing URL stays the created feed’s page; **Create another feed** starts a new listing. The visual builder and YAML are one document: validate exports `channel.url` plus selectors. A successful validate echo keeps the server’s items selector and enhance flag; the client keeps the field selectors it authored. Only `studioService` unwraps the studio envelope and maps ranked `candidates` (items / title / link / published) into domain suggestion buckets. Candidates never auto-apply. A suggestion that fails on open is silent and non-blocking: the panel stays editable without ranked choices. Chips add or remove selector parts on the authored comma-joined string; suggest does not write that string, and chips do not change Enhance. Field extractors remain fixed by the draft mapper. Save posts refined selectors through Creation IO and stays on the result view.
+
+### Refined Selectors
+The allowlisted `selectors` document a token holder may attach on feed creation. Signed into the feed token and resolved on later serves. Not a full config: channel, headers, and strategy stay denied.
+
 ### Unmatched result
 A result route whose in-memory created feed token does not match the route `feedToken`. Feed Flow replaces to create without prefill (no second journey decide elsewhere).
 

Gemfile

@@ -4,8 +4,10 @@ source 'https://rubygems.org'
 
 git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
 
-gem 'html2rss', '~> 0.30'
-# gem 'html2rss', github: 'html2rss/html2rss', branch: 'master'
+# Pinned by revision, not by branch. This ref is feat/suggest-selector-candidates
+# rebased onto master, so it includes CaptureResult#candidates and the Falcon/HTTPX
+# retry fix. Bump it deliberately; do not swap it for `branch:` or a rubygems constraint.
+gem 'html2rss', github: 'html2rss/html2rss', ref: '628820dbcc55504d0fae160595c6b6bf26bd9d68'
 gem 'html2rss-configs', github: 'html2rss/html2rss-configs'
 
 # Use these instead of the two above (uncomment them) when developing locally: