perf(fetch): cache ESPN scoreboard windows without the parts nothing reads - #749
Open
ChuckBuilds wants to merge 1 commit into
Open
ChuckBuilds wants to merge 1 commit into
ChuckBuilds wants to merge 1 commit into
Conversation
…reads The sports scoreboards cache their Recent/Upcoming window as the raw ESPN response, and it stays parsed in the memory tier while fresh. Measured on hdpi, most of it is never drawn: per-team stat leaders, athlete cards (featuredAthletes, probables), team and event links, headlines, video highlights and geo broadcasts. None of those keys is read by core or by any plugin in ledmatrix-plugins. BackgroundDataService now drops them from an ESPN /scoreboard response before caching and delivering it (src/common/espn_payload.py), keeping everything else. On the five hdpi windows that is 10.6MB -> 3.0MB of JSON and ~40MB -> ~12MB of parsed objects, and parsing an expired window gets 3-4x cheaper. submit_fetch_request(slim_payload=False) caches a response whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkfgXMqqwn2w4NN7LRzhxy
Contributor
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 46 seconds. View limit detailsLimit details: You’ve used the included review currently available. Review configuration: ⚙️ Run configuration
📒 Files selected for processing (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| ErrorProne | 1 high |
🟢 Metrics 20 complexity
Metric Results Complexity 20
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The sports scoreboards cache their Recent/Upcoming window (14 days back, 7 ahead) as the raw ESPN response, and while it's fresh it stays parsed in the memory tier. Most of it is never drawn.
BackgroundDataServicenow drops those parts of an ESPN/scoreboardresponse before caching and delivering it (src/common/espn_payload.py):leadersprobables,status.featuredAthletesteam.links,event.linksheadlines,highlightsgeoBroadcastsEverything else is kept, including odds, records, linescores, situation, statistics, notes, broadcasts and venue. Only those exact keys are ever removed.
submit_fetch_request(..., slim_payload=False)caches a response whole.On the four hdpi windows (MLB, college football, NHL, NFL) this takes the JSON from 10.6 MB to 3.0 MB and the parsed objects from ~40 MB to ~12 MB. Parsing an expired window, which some scoreboard paths do on the render thread, gets 3–4x cheaper.
Type of change
Test plan
main(f568fd3) and this branch rebased on 07abd87: I ranscripts/check_plugin.pyfor baseball (MLB), football (NFL, and college football withncaa_fbenabled) and hockey (NHL) at every size, seeded with the real hdpi windows, once raw and once slimmed. Records, odds and rankings were switched on. 88 PNGs, all byte-identical. They show real games (e.g. PHI–ATL Final 2–6, Stable #3 Notre Dame vs UNC).main. The only hits areleadersin f1-scoreboard and nfl-stat-leaders, which read different ESPN APIs (not/scoreboard, not via this service), and"highlights"in baseball-scoreboard, which is a config value, not a payload key.mainthen this branch. RSS was 30–40 MB lower at every 10-minute mark (306–326 vs 337–385 MB). Late frames were 4.7 vs 4.1 per 1,000, with no regression.pytest): newtest_espn_payload.pychecks that exactly the listed keys go and everything else is unchanged, that odd shapes pass through, that the URL check rejects look-alike hosts, that the service caches and delivers the same slimmed object, the opt-out, and that other URLs are untouched. 145 related tests pass.Notes for reviewer
APIHelper/store_espn_scoreboard_cache(feat(fetch): one ESPN scoreboard cache key and a max-age response cache (fetch service stage 2) #728's shared key) aren't slimmed. They're small and outside this change's scope.slim_payload=False. The module docstring says so.🤖 Generated with Claude Code