feat(studio): refine feeds with ranked selector candidates - #1152
Draft
gildesmarais wants to merge 70 commits into
Draft
gildesmarais wants to merge 70 commits into
gildesmarais wants to merge 70 commits into
Conversation
Preview and validate need ValidationReport and Test::Result#validation_issues, which only exist on gem master. html2rss-configs still resolves against that pin.
Preview and serve would otherwise build different configs. One expansion owner, with the old private builder deleted, keeps those paths identical.
…ramps Refine is memory-bound: #/refine carries no token, and a visit with no in-memory URL recovers onto remounted #/create the same way an unmatched result does.
…onfigs Client config must not be able to rewrite channel.url or headers. One allowlist owns that rule and runs again when a token is decoded.
The signature gains selectors only when a token carries them, so feed URLs already in the wild keep validating.
The served feed now expands the same signed selectors preview will use, and the allowlist runs again at serve time so a decoded token cannot skip it.
Studio edits stay server-authoritative: validate reports schema issues without minting a token, preview selects Test::Result fields so the RSS body never leaves the server, and suggest returns one capture-derived selector with its evidence.
…contract Create signs an optional allowlisted fragment into the token, and the OpenAPI hook now derives issue codes and accepted keys from the runtime sets so the published client cannot drift from the rule.
`publish_allowed_keys` wrote `spec['components']` while rspec-openapi seeds `spec[:components]` with `securitySchemes` (symbol-keyed, see rspec-openapi's DefaultSchema). The string-keyed write shadowed that hash on dump, so `public/openapi.yaml` published four `BearerAuth: []` requirements against a scheme that no longer existed, and the generated frontend client dropped bearer security from createFeed, previewFeed, suggestSelectors and getHealthStatus. Reuse whichever key form the document already carries -- the same string/symbol grammar the tags and paths stamps already use -- and merge into the existing components hash instead of replacing it. Arm `openapi-lint` in `ci-ready` so this class of defect fails the gate: `openapi-verify` only diffs the regenerated file against itself and never sees a dangling security reference. Redocly was green before the studio endpoints landed, so the two publish defects those endpoints introduced are fixed here too: ValidationIssue `expected`/`actual` are JSON-ish or nil and now publish as open schemas (OAS 3.0 rejects `nullable` without a sibling `type`), and preview gains a recorded empty-extraction example so `failure_kind` and `quality_report` publish a type next to their `nullable`. The remaining Redocly error (`security-defined` on POST /feeds/validate) is real: that operation is still unauthenticated. It is fixed in the next commit.
POST /api/v1/feeds/validate was the only studio operation reachable without a token. Preview and suggest both call `Auth.authenticate` and raise `UnauthorizedError`, so an unauthenticated caller could still drive the schema validator, the YAML parser and the refined-selectors allowlist -- CPU the instance owner never granted, and a free oracle for which config shapes the instance accepts. Gate it the same way its siblings are gated. Validate needs no live fetch, so it takes the token check only, not the auto-source flag or the per-account URL allowlist. The request spec posted with no Authorization header and asserted 200; it now carries the admin token, and the shared api-error-contract example covers the 401. Regenerated `public/openapi.yaml` and the frontend client so validateFeedConfig declares bearer security and its 401 response, which also clears the last Redocly `security-defined` error under `make openapi-lint`.
`STUDIO_ENABLED` existed only in app/web/config/flags.rb, so an operator had to read the flag registry to learn the studio endpoints exist, that they are off in production, or how to turn them on. Document it next to `AUTO_SOURCE_ENABLED` in the docs flag table, .env.example, and the production compose file (the quickstart compose is left alone: RACK_ENV=development already defaults the flag on, and it lists only what it must set). State what the flag exposes rather than just naming it: validate, preview and suggest_selectors; all token-gated; 403 "Studio is disabled" when off; preview and suggest_selectors additionally require AUTO_SOURCE_ENABLED because they fetch the caller's URL live and are bounded by the account URL allowlist. The 403 branch had no request coverage. The new examples iterate `FeedRoutes::STUDIO_POSTS`, so a future studio endpoint is covered by the same gate instead of silently shipping without one. They are `openapi: false`: the flag-off shape is an operator concern, and the other ClimateControl-driven examples in this file stay out of the published contract too.
Join feat/config-studio-rb (P01–P07 and A1 fixes) so later studio UI phases share one checkout with the published contract.
Join feat/config-studio-ts (P08) so the refine route, on-ramps, and ConfigStudio shell sit on the same checkout as the studio contract.
Unwrap the studio envelope in one service and keep refinement phase, issues, and preview in a discriminated union so later panels cannot invent a second client.
Fill the refine card with the items selector, enhance toggle, one-click capture suggestion, attribute extractors, and post-processors so refinement stays on shared primitives instead of a DOM picker.
Keep YAML as the same config as the builder by posting it through validate, and surface parse failures as a syntax notice so the builder does not hydrate from broken text.
Save posts the validated selectors through the existing create path and returns to the result view, and the refine route is now part of the journey grammar and operator docs.
FeedToken::Codec.decode rebuilt the signed selectors through RefinedSelectors.from_wire, which ran Html2rss::Config.validate. That put the full gem validator behind an unauthenticated, unsigned wire document: any caller could drive it by putting a selectors subtree in a forged token. Measured 0.004ms for a plain bad token against 0.218ms for an 808-byte forged one carrying 60 selectors, a 58x amplification before a single HMAC byte is compared. Split the build so decode runs the structural allowlist only (denied keys, shape, wire cap) and the schema check moves behind verification. SourceResolver now calls from_verified_wire, which it only reaches after authorize_feed_token! has checked the signature, so the defence-in-depth re-validation is kept without exposing it pre-auth.
Codec.decode inflated whatever arrived on the token route with no cap on either the encoded input or the inflated output. Auth.extract_token's 1024 byte limit only guards the Authorization header, not the path segment, so a 26KB token expanded to 20,000,060 bytes in 55ms for an unauthenticated caller: a 769x memory amplification per request. Reject encoded tokens over MAX_ENCODED_BYTES and inflate in chunks, aborting at MAX_INFLATED_BYTES. The abort raises Zlib::DataError, which decode already rescues, so an oversized token still fails closed as 401 rather than surfacing a new error shape.
Flags.development_or_test? defaulted a missing RACK_ENV to "development", while EnvironmentValidator.non_production? treats the same state as production. A deployment that forgot the variable therefore got production secret-key enforcement but silently enabled STUDIO_ENABLED and AUTO_SOURCE_ENABLED, exposing the studio and live-fetch surfaces. Treat unset as not-development so both flags default off. Every real entrypoint sets RACK_ENV explicitly (docker-compose, bin/dev*, the dev container, and spec_helper), so no supported workflow changes.
build_context read request.params, and Rack::Request#params parses POST for form-encoded requests, which buffers the entire body. The middleware runs before RateLimiter and before every handler's MAX_BODY_BYTES gate, so a 200KB form POST to a studio route was fully read into memory and only then rejected with 400. The new routes' body limit was effectively decorative against that content type. Read the query string instead. The strategy parameter this line wants has always been a query parameter on feed URLs, so the recorded context is unchanged for every real caller.
FEED_TOKEN_ROUTE matches any single segment under /api/v1/feeds/, so the studio routes logged as /api/v1/feeds/[REDACTED]. Validate, preview, and suggest_selectors calls became indistinguishable in the audit trail, which removes the evidence needed to investigate abuse of exactly the endpoints that trigger upstream fetches. Skip redaction for segments the router mounts by name. The list is read from FeedRoutes::STUDIO_POSTS so the router stays the single owner and a future studio route cannot drift out of the audit trail.
ValidateConfig told a schema failure apart from an allowlist rejection by
testing whether the exception message contained " [", the separator in
ValidationIssue#to_s. Root keys are echoed into the rejection message, so a
caller chose which branch ran by naming a key: POST /api/v1/feeds/validate
with a yaml root key of "foo [bar" returned 200 and report.success=true for
a config the create endpoint rejects, while "channel" and "request"
correctly returned 400. Validate and execute disagreed on the same input.
Raise RefinedSelectors::SchemaInvalid carrying the report, and rescue that
type instead of parsing a message. The rescue also stops re-validating: the
report it already has is returned directly.
assess now forwards the whole body rather than only body["selectors"], so a
denied sibling key is rejected on the JSON path as it already was on the
YAML path. The frontend only ever sends {selectors} or {yaml}, so no client
request shape changes.
All three studio handlers checked Flags.studio_enabled? and Flags.auto_source_enabled? before Auth.authenticate, so an anonymous POST was answered with 403 "Studio is disabled" or 403 "Auto source feature is disabled". That hands an unauthenticated caller the deployment's feature configuration and tells them which instances are worth probing further. Authenticate first, then evaluate the flags, so an anonymous caller only ever sees 401.
items_selector was forwarded to Html2rss.capture with no length limit; a 5000 character hint reached the gem unchanged. The selector is client text handed to a parser during a live fetch, and the handler maps no gem parse failure, so anything raised there classifies as 500 rather than a client error. Cap the hint and reject an oversized one as 400.
Drives every example as a real Rack request through the full middleware stack, so a gate bypassed at the middleware or routing layer still fails here rather than passing against a unit stub of the gate. Verified red before the fixes in this branch: 14 of 29 examples failed against the pre-fix tree, covering pre-HMAC schema validation, the token wire caps, the request-context body buffering, the " [" message sniff, the denied root keys on the JSON validate path, the RACK_ENV flag default, the flag-state disclosure, and the items_selector cap. All 29 pass after. The remaining 15 are verification rather than regression: IDOR on serve, preview, and suggest, username replay, tampered signed selectors with an assertion that no fetch happened, expired tokens, and unauthenticated POSTs. They passed before and after, and are kept so the behaviour is pinned.
The lock pinned revision 84cad9e9, which predates the gsub work. Without it a client-authored `post_process: gsub` pattern reaches `Regexp.new` unbounded, so the studio's selector path had no pattern-length or star-height ceiling. Pin `ref: fc5de544`, the current html2rss master tip. That commit is the squash-merge of PR #512, so the four branch commits (ec834ff7, b114e16c, 385c77cf, c7be5c83) are not literal ancestors of it; the trees are identical instead (`git rev-parse c7be5c83^{tree} fc5de544^{tree}` both yield fa9d14e4), which is the verifiable form of "master contains them". The installed gem carries `Gsub::MAX_PATTERN_LENGTH` and `Config::IssueMapper`. `branch: 'master'` is dropped because a floating branch silently re-resolves those bounds on any later `bundle update`. The commented `gem 'html2rss', '~> 0.30'` line is dropped too: uncommenting it resolves the released 0.30.0 gem, which has neither the gsub bounds nor `Config::IssueMapper`.
`selectors_for` signed whatever `selectors` object the body carried and never read `STUDIO_ENABLED`. The three studio POSTs honoured the kill switch, but create did not, so an operator who turned the studio off could still be handed a token whose signed `c:` document ran client-authored selectors — the flag closed the door and left the window open. Create now raises the same `403 Studio is disabled` the studio POSTs raise, but only when the body actually carries `selectors`. A body without them stays the automatic path and keeps working with the flag off, so ordinary URL-only create is untouched. The gate runs before the `AUTO_SOURCE_ENABLED` check so the refusal does not depend on a second, unrelated flag. Authentication still runs first, so an anonymous caller cannot probe flag state. `require_account` collapses to one expression to stay under `Metrics/ClassLength` without disabling the cop.
…eate Two behaviours the suite asserted nowhere. `post_process: gsub` is the selector feature that carries a client-authored regexp all the way to `Regexp.new`, so the allowlist spec now proves the pattern survives minting, signing, and decoding intact, and that the gem's compile bound (nested quantifiers, 256-character ceiling) surfaces as a 400 rather than a pathological match. A denied root key travelling next to a post_process is still rejected by name, so the post_process path does not widen the allowlist. The create kill switch is HTTP behaviour, so it is covered with real Rack requests next to the existing `AUTO_SOURCE_ENABLED` example: a `selectors` body under `STUDIO_ENABLED=false` gets the same `403 Studio is disabled` as the studio POSTs, and a URL-only body under the same flag still returns 201. Checked out against the pre-fix `create_feed.rb` the first example fails (201 instead of 403) and the second passes, so the pair discriminates the fix from the regression it guards. Both are `openapi: false`: they document flag states, not the published contract.
gildesmarais
marked this pull request as draft
September 19, 2026 22:15
Replace example-inferred items-only hulls with RefinedSelectorsDocument and PreviewSampleItem so the SPA client matches the allowlisted refine payload.
Consolidate create-shell mounts and repeated hook stubs into file-local scenario helpers without dropping observable assertions.
Zero out ConfigStudio settle delays via a file-local useStudio wrap, share Studio response builders, and keep useStudio debounce proofs on deterministic fake timers without waitFor under those timers.
Extract pure API wire builders for MSW and Playwright, then cover refine URL re-suggest/preview plus empty-blur remount and studio-off hide/bounce as two deterministic browser journeys.
Point CI Vitest at one combined test:run process, pin Playwright to a single worker, and correct contributor docs for the all-suite entrypoint.
gildesmarais
marked this pull request as draft
September 20, 2026 10:43
Apply was still sending the typed items selector as the suggest hint, so a bad edit could bias capture instead of a gem-fresh determination. Post an empty hint (same as auto-prefill) and clarify the control as Use determined selector.
Authenticated live preview was stuck on the gem's three-item sample. PreviewFeed now owns a ten-item meadow and copies only safe HTTP(S) images from the RSS document.
The JSON preview parser always capped at five items and dropped images. Audience now follows the access token used at creation, so retry cannot widen a guest meadow.
Successful results kept a separate refine route that competed with Copy feed URL. Token-backed studio now edits selectors under the one result meadow, and stale refine routes return there.
Successful results keep selector controls on the result route, and #/refine stays extraction-empty recovery. The parity gate also required the preview tests to satisfy existing ESLint rules without dropping the accepted HTTP image case.
Default MSW validate now echoes request selectors so contract paths cannot pass article from a silent overwrite without clicking Use.
Pin html2rss to the capture candidates commit and replace the single selectors suggestion with ranked items/title/link/published buckets on suggest_selectors.
Map suggest candidates through studioService and render explicit per-bucket Use actions in gated refine without auto-applying into the draft.
The candidates branch now includes the Falcon HTTPX retry fix from master.
make up runs this checkout plus the scraper image on loopback. The macOS port check uses lsof so Falcon is not killed after boot.
gildesmarais
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
POST /api/v1/feeds/suggest_selectorsreturns rankedcandidatesfor items, title, link, and published. Empty buckets are HTTP 200. The SPA applies a candidate only when the user clicks Use.STUDIO_ENABLED, and live-fetch rules.html2rssis pinned to281e5e90onfeat/suggest-selector-candidates.make up/make downrun this checkout with the published Botasaurus image on127.0.0.1:4010.Why
Auto-source is not enough when the list or a field is wrong. A single silent items selector overwrote the draft and could not offer title, link, or published choices. Local development also needs the scraper without starting the published web image.
Risk
selectorsis replaced bycandidates.make upis a host command. The Dev Container still usesmake devandhost.docker.internal.Review map
spec/html2rss/web/api/v1_spec.rb— suggest returns ranked buckets and empty buckets as success.frontend/src/__tests__/ConfigStudio.test.tsxandfrontend/src/__tests__/useStudio.test.ts— suggestion does not overwrite the draft until Use.app/web/api/v1/suggest_selectors.rb— studio gates unchanged; mapsCaptureResult#candidates.frontend/src/studio/studioService.tsandfrontend/src/components/ConfigStudio.tsx— wire parse and explicit per-bucket Use.spec/support/openapi_studio_schemas.rbandpublic/openapi.yaml— candidate schema, generated client left generated.docker-compose.botasaurus.yml,Makefile, andbin/dev— local boot and macOS listen check.Validation
make openapi,make openapi-client,make ready, andmake ci-readyexiting 0.make upcommit were not run throughmake readyormake ci-ready.