Optimize news costs, translation latency, and crawl budget - #87
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5e4c8eef12
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if article_pending: | ||
| return sorted(article_pending) |
There was a problem hiding this comment.
When regular editions continue publishing, every worker invocation begins with missing article text, so this early return discards the remaining text budget and dictionary_pending is never reached. Because the worker is only queued after a new edition, live market labels and explanatory-page strings can remain untranslated indefinitely even when the fresh articles consume far less than the 150-text budget; run a second pass after exporting the article translations or explicitly schedule the deferred backlog work.
Useful? React with 👍 / 👎.
| ExecStartPost=-/usr/local/bin/node scripts/news/indexnow.mjs | ||
| ExecStartPost=-/usr/local/bin/node scripts/news/websub.mjs |
There was a problem hiding this comment.
If the bounded worker times out at 90 minutes or fails after exporting one or more languages, systemd does not execute ExecStartPost; the leading - only ignores failures of the post command itself. Consequently, successfully exported localized URLs from a partial run receive neither the promised IndexNow delta nor WebSub notification, and repeated late-language failures can prevent notifications indefinitely; place these notifications in an always-run cleanup path such as ExecStopPost or a wrapper.
Useful? React with 👍 / 👎.
| const urls = activeArticles.slice(offset, offset + SITEMAP_ARTICLE_BATCH_SIZE).map((article) => { | ||
| const locales = articleLocales(article); |
There was a problem hiding this comment.
For sitemap-based hreflang, each alternate URL must have its own <url><loc> entry and repeat the complete alternate set, including itself. Mapping every article to only the English URL here—and doing the same for listing, topic, and country pages above—leaves the localized href values without reciprocal sitemap entries, so crawlers consuming the sitemap annotations cannot validate those language clusters and the localized pages are no longer directly submitted in the general sitemap.
Useful? React with 👍 / 👎.
This branch was successfully deployed
1 participant
Summary
gpt-5.6-luna, keeping medium reasoning only for research and low reasoning for translation/social selection<loc>per page with complete localizedhreflangalternates, cutting the crawl inventory by roughly one order of magnitude while retaining multilingual discoveryVerification
npm run checknpm run test:e2e(144 passed)