[musingfox]

Why

Some feeds give one sentence per item. On the 2026-11-03 morning issue, the pjwu.me story "別人制定的標準" was summarized from a 27-character description, while its page holds about 1300 characters. In a measurement that day, 17 of 51 feeds had a median under 500 characters of text per item. #151 made feed HTML into text; this PR fills in the text that teaser feeds never send.

What

• adapters/fetch/article_page.lengthen_articles reads the page of each RSS article whose text is shorter than summarize_snippet_length. It extracts the body with trafilatura and keeps whichever text is longer. Any failure keeps the feed text, one page at a time.
• run_digest calls it once, through an optional Deps callable. The call comes after the cap and the vote-similarity holds, and before DigestPipeline. The page text is used in that run only and is never stored. Scoring has already read the feed text by then.
• The callable is wired for cyris run, including dry runs, and for a regenerate rerun. rerender and the comparison commands get none.
• Bounds:
  • 8 pages in flight.
  • 10 s per page, covering redirects and the body.
  • 5 MiB of raw bytes per page.
  • The request asks for an uncompressed page; an encoded answer is refused.
• Refused before any request:
  • non-http(s) URLs;
  • IP literals that are not global, in any IPv4 spelling;
  • localhost names, on the first URL and on every redirect hop;
  • video links (media_links in keywords.json: /watch, /shorts, youtu.be).
• The User-Agent names cyris and nothing else.
• Docs: architecture.md gets a §2 direct-injection row, a §3.2 description, three §5 grade-A bounds, and a §8 exception that CLAUDE.md mirrors.

Owner decisions (2026-11-03)

• The call site is an optional injected callable, invoked from the run, with all IO in adapters. The rejected alternatives are a fetch inside the FetchSource, which would store page text and fetch before the cap, and a wrapper around the ArticleRepository, which would hide network IO behind a store read.
• Without an LLM, the 200-character excerpt may come from page text.
• Hostnames are not resolved, so a name that points at a private address is still fetched. The owner accepted this risk. Resolving names safely needs a custom transport, which the outbound-http-honours-environment spec forbids.
• Paywalls are not handled.

Verification

• scripts/check.sh: 4834 passed on the branch tip.
• Codex found that httpx's follow_redirects read each redirect's body whole, decompressed, before any cap. 91ea0adb follows redirects in _last_hop and closes each hop unread; a test with a watched 302 body was red before the fix.
• Two independent review rounds were run against the contract. The first failed and its findings are fixed here: an unknown charset aborted the whole batch, a gzip page could inflate past the cap, shorthand IPv4 and localhost names got through, and a YouTube page's footer beat its description.
• The second round passed, with probes against local servers: bad charsets, gzip bombs (peak memory +1 MB, down from +617 MB), 13 private-host spellings on the first URL and on redirect hops, and the YouTube URL shapes.

Open, not blocking

• Two checks, the content-type gate and the raw-byte counting, have no test that fails without them.
• The threshold ignores the lane, so skim and follow articles may be fetched without being summarized from it.

🤖 Generated with Claude Code

[musingfox]

@codex review

[coderabbitai]

Important

• Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

• Source mode: Auto
• Repository: musingfox/cyris
• Branch: feed-teaser-fetch
• File: .coderabbit.yaml
• Commit: 91ea0ad
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Advanced
• Run ID: e0e7aebe-63b0-4984-b754-5673538479df

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

CodeRabbit Verify

• Run CodeRabbit Verify to test the changes in this pull request.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T08:16:58.486653Z	91ea0ad	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[claude]

Reviewed the diff and the call site in run_digest. I found nothing that needs fixing before merge.

What I checked:

• Persistence: the lengthened digest_articles list is only passed to the pipeline and to the synthetic_url_count counter. Nothing saves it, so page text can't leak into stored_articles.
• Failure isolation: the per-page except Exception and the outer try in run_digest both fall back to the feed text. The failure is logged and recorded in summary["article_pages_error"], so it doesn't pass silently.
• Redirects: the request event hook runs on every redirect hop, so the private-literal check covers redirects.
• Size cap: Accept-Encoding: identity plus refusing encoded responses keeps MAX_PAGE_BYTES honest.
• Timeout: the timeout sits inside the semaphore, so it applies per page and not per batch.

Minor points, not blocking:

• Hostname-to-private-IP SSRF is left open. The PR documents this as an accepted risk.
• Page text is only kept when it is longer than the feed text. A page with a cookie banner or other boilerplate that trafilatura extracts could replace a good short summary with noise. The PR already lists the missing tests for the content-type and byte-count checks as open.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 51cc61c2ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Enforce response bounds before following redirects

When a teaser URL returns a redirect with a large or compressed body, follow_redirects=True lets HTTPX read and decompress that entire intermediate response before client.stream() yields the final response. Consequently, neither the encoding rejection nor MAX_PAGE_BYTES protects redirect bodies. I reproduced this with the installed HTTPX using a gzipped 302 response: its 6 MiB decoded body was already buffered in response.history before application code received the final response. A larger payload can exhaust the run container's memory instead of falling back to feed text. Follow redirects manually, closing intermediate responses without reading their bodies, or otherwise enforce the bounds before HTTPX buffers them.

Useful? React with 👍 / 👎.

[musingfox]

@codex review

[claude]

Reviewed the diff and the surrounding run_digest flow. I found nothing that needs a change before merge.

What I checked:

• Redirects: each hop goes through the private-address hook, and redirect bodies are closed unread.
• Size and encoding: the 5 MiB cap is counted on raw bytes with Accept-Encoding: identity, and any encoded response is refused.
• Timeout: the 10s timeout wraps the fetch but not the semaphore wait, so queueing time doesn't count against a page.
• Failures: a failed page falls back to the feed text. The outer try/except in run_digest records article_pages_error in the summary.
• Storage: the lengthened text is built after scoring and is not persisted. digest_articles only feeds the pipeline and the synthetic_url_count.

Two minor notes, neither blocking:

• The DNS-rebinding gap is documented as an accepted risk, so I didn't raise it as a finding.
• Page text from third-party sites now goes straight into summarize prompts. That is a prompt-injection surface, but only for the summary text. It may be worth a line in the architecture doc.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 91ea0adb58

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".