Skip to content

fix(rss): hand the pipeline plain text instead of feed HTML - #151

Merged
musingfox merged 2 commits into
mainfrom
feed-html-to-text
Nov 3, 2026
Merged

fix(rss): hand the pipeline plain text instead of feed HTML#151
musingfox merged 2 commits into
mainfrom
feed-html-to-text

Conversation

@musingfox

@musingfox musingfox commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Why

An RSS article's Article.content was the feed's raw HTML, and every LLM prompt snippet (scoring, summarize, clustering, vote order) plus language detection cut its first N characters. A Substack-style post that opens with a captioned image spends thousands of characters on <picture srcset> markup before the first sentence.

On the 2026-11-03 morning issue, two of the three Stories were summarized from that markup:

  • Tidy First "Not A Copier": the body starts at character 4626 of the feed HTML, so the 1000-character snippet held only image tags.
  • ByteByteGo "Nemotron": the body starts at character 5924, after a sponsor block.

Across 234 entries from 51 feeds, 46 (one in five) got under 200 characters of text in a 1000-character window, mostly Substack and Ghost sources.

What

  • adapters/fetch/html_text.plain_text turns an entry's HTML into body text. trafilatura does the extraction, with tables kept and captions dropped. When trafilatura declines a fragment, such as a one-line teaser, html.parser strips the tags. On CPython 3.12 before 3.12.16, html.parser can assert, so that case falls back to a regex strip.
  • CloudflareRssSource and RssSource both call it, off the event loop, once per read.
  • The RSS Worker keeps storing HTML, because it reads the story image from it. RssSource still reads the image from the raw HTML before converting.
  • architecture.md §4 now records that stored_articles.content is written as plain text. Rows stored earlier keep their HTML, because INSERT OR IGNORE never rewrites them.
  • degrade.excerpt gets a docstring-only change that states its known cost on plain text: a literal Vec<String> loses <String>, and &amp; is decoded twice.

Rejected

  • pydefuddle 0.1.0 (its only release): on Substack pages it returned the "Subscribe / Sign in" chrome or the footer instead of the post.
  • Defuddle itself: it is JavaScript, and the image deliberately has no Node.
  • A regex tag strip as the fallback: it deleted prose containing < (Price < $5 and > $1 became Price $1) and leaked tags a feed had cut off mid-way.

Verification

  • scripts/check.sh: 4761 passed.
  • New tests, each red before and green after:
    • a Substack-shaped fixture opens at the first sentence, on both paths;
    • table text is kept;
    • the fallback keeps prose with bare < and >, drops a truncated tag, and drops a tag whose attribute contains >;
    • a test passes only when trafilatura is used, not a plain tag strip;
    • conversion runs off the event loop;
    • the parser-assertion fallback.
  • Two independent review rounds were run against the contract. All findings are addressed here, except two accepted low-severity edge cases in the fallback:
    • tagless a<b with no later > drops the rest of the text;
    • <script> text leaks from a single-element fragment, as it did before.

After deploy

Only articles fetched after the deploy carry text, so a rerun of an old issue shows no change. Check the next issue's Substack and Ghost summaries.

🤖 Generated with Claude Code

musingfox and others added 2 commits November 3, 2026 12:51
@musingfox @claude
build(deps): add trafilatura for feed HTML to text
5064212 
The next commit extracts an RSS entry's body text with it. Chosen over
pydefuddle 0.1.0 (its only release), which on Substack pages returned the
"Subscribe / Sign in" chrome or the footer instead of the post, and over
Defuddle itself, which is JavaScript and would bring Node back into an
image that dropped it on purpose (see the Dockerfile header).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@musingfox @claude
fix(rss): hand the pipeline plain text instead of feed HTML
2e73dcf 
Article.content of an RSS entry was the feed's raw HTML, and every prompt
snippet (scoring, summarize, clustering, vote order) and language
detection cut its first N characters. A post that opens with a captioned
image spends thousands of characters on <picture srcset> markup: on the
2026-11-03 morning issue the Tidy First and ByteByteGo stories were
summarized from image tags and a sponsor block, and across 234 entries
from 51 feeds one in five got under 200 characters of text in a
1000-character window.

Both feed adapters now pass the entry through html_text.plain_text:
trafilatura extracts the body (tables kept, captions dropped); a fragment
it declines, such as a one-line teaser, is tag-stripped with html.parser,
not a regex, so prose like "x < 3" survives and a tag a feed cut off
mid-way does not leak. On CPython 3.12 before 3.12.16 html.parser can
assert on "<![", so that one case falls back to a regex strip rather than
failing the read. The conversion runs off the event loop, one to_thread
per read, since a Worker read holds up to WORKER_ROW_CEILING rows.

The RSS Worker keeps storing HTML, which it reads the story image from;
RssSource still reads the image from the raw HTML before converting.
Rows already stored keep their HTML (INSERT OR IGNORE never rewrites
them), so only articles fetched after deploy benefit; architecture.md §4
says so. degrade.excerpt keeps stripping tags for those rows, at the cost
on plain text that its docstring now states.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@musingfox

musingfox commented Nov 3, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@coderabbitai

coderabbitai Bot commented Nov 3, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T05:15:39.135040Z 2e73dcf Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@claude

claude Bot commented Nov 3, 2026

Copy link
Copy Markdown

Reviewed the diff (HTML-to-text conversion via trafilatura for both RSS adapters, with a tag-stripping fallback). I found no correctness, data-loss or security problems worth blocking on. I didn't run the tests.

Two things to be aware of, neither blocking:

  • trafilatura.extract is a main-content extractor, so with favor_recall=True it can still drop boilerplate-looking parts of a short feed summary. The only fallback is when it returns nothing. A partially trimmed summary will pass through without any signal.
  • logging.getLogger("trafilatura").setLevel(CRITICAL) is process-wide and runs at import. It hides real trafilatura errors from every caller, not just the teaser case the comment describes.

🤖 Generated with Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 2e73dcf9a6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@musingfox
musingfox merged commit 32d6a03 into main Nov 3, 2026
3 checks passed
@musingfox
musingfox deleted the feed-html-to-text branch November 3, 2026 06:18
@musingfox musingfox mentioned this pull request Nov 3, 2026
Merged
musingfox added a commit that referenced this pull request Nov 3, 2026
@musingfox
feat(fetch): read a teaser article's own page for its body (#152)
ee6ce46 
## Why

Some feeds give one sentence per item. On the 2026-11-03 morning issue,
the pjwu.me story "別人制定的標準" was summarized from a 27-character
description, while its page holds about 1300 characters. In a
measurement that day, 17 of 51 feeds had a median under 500 characters
of text per item. #151 made feed HTML into text; this PR fills in the
text that teaser feeds never send.

## What

- `adapters/fetch/article_page.lengthen_articles` reads the page of each
RSS article whose text is shorter than `summarize_snippet_length`. It
extracts the body with trafilatura and keeps whichever text is longer.
Any failure keeps the feed text, one page at a time.
- `run_digest` calls it once, through an optional `Deps` callable. The
call comes after the cap and the vote-similarity holds, and before
`DigestPipeline`. The page text is used in that run only and is never
stored. Scoring has already read the feed text by then.
- The callable is wired for `cyris run`, including dry runs, and for a
`regenerate` rerun. `rerender` and the comparison commands get none.
- Bounds:
  - 8 pages in flight.
  - 10 s per page, covering redirects and the body.
  - 5 MiB of raw bytes per page.
- The request asks for an uncompressed page; an encoded answer is
refused.
- Refused before any request:
  - non-http(s) URLs;
  - IP literals that are not global, in any IPv4 spelling;
  - localhost names, on the first URL and on every redirect hop;
- video links (`media_links` in `keywords.json`: `/watch`, `/shorts`,
`youtu.be`).
- The User-Agent names cyris and nothing else.
- Docs: architecture.md gets a §2 direct-injection row, a §3.2
description, three §5 grade-A bounds, and a §8 exception that CLAUDE.md
mirrors.

## Owner decisions (2026-11-03)

- The call site is an optional injected callable, invoked from the run,
with all IO in adapters. The rejected alternatives are a fetch inside
the `FetchSource`, which would store page text and fetch before the cap,
and a wrapper around the `ArticleRepository`, which would hide network
IO behind a store read.
- Without an LLM, the 200-character excerpt may come from page text.
- Hostnames are not resolved, so a name that points at a private address
is still fetched. The owner accepted this risk. Resolving names safely
needs a custom transport, which the `outbound-http-honours-environment`
spec forbids.
- Paywalls are not handled.

## Verification

- `scripts/check.sh`: 4834 passed on the branch tip.
- Codex found that httpx's `follow_redirects` read each redirect's body
whole, decompressed, before any cap. `91ea0adb` follows redirects in
`_last_hop` and closes each hop unread; a test with a watched 302 body
was red before the fix.
- Two independent review rounds were run against the contract. The first
failed and its findings are fixed here: an unknown charset aborted the
whole batch, a gzip page could inflate past the cap, shorthand IPv4 and
localhost names got through, and a YouTube page's footer beat its
description.
- The second round passed, with probes against local servers: bad
charsets, gzip bombs (peak memory +1 MB, down from +617 MB), 13
private-host spellings on the first URL and on redirect hops, and the
YouTube URL shapes.

## Open, not blocking

- Two checks, the content-type gate and the raw-byte counting, have no
test that fails without them.
- The threshold ignores the lane, so skim and follow articles may be
fetched without being summarized from it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@musingfox