feat(fetch): count bytes on the wire as well as decoded

[ChuckBuilds]

Summary

GET /api/v3/plugins/fetch-stats reported one byte count, bytes, which is len(response.content): the decoded body. That number was read as download volume. ledpi's football scoreboard looked like "643 MB in 6 hours". ESPN gzips every scoreboard, though, so the counter overstated real traffic about 14x.

FetchService._send now also counts wire_bytes in every counter set (totals, per plugin, per host). It reads urllib3's HTTPResponse.tell(), which is the number of raw bytes taken off the socket before decompression.

• A response with no urllib3 response behind it (a test double, another adapter) is counted at its decoded size, so the counter never claims less than it can prove.
• bytes keeps its meaning.
• The snapshot schema is unchanged; this only adds a key. The test_every_counter_is_in_every_snapshot test covers the new field automatically.

Measured against ESPN on 2026-10-04, college-football/scoreboard?dates=20261003&limit=500 (54 events):

Accept-Encoding	Content-Encoding	bytes on the wire
gzip, deflate, br (what the plugins send)	gzip	63,275
identity	—	865,068

ESPN never answers with br, and requests/urllib3 decode gzip, so the 13.7x ratio is what a rig actually saves.

Where football's bytes go (investigation for this PR)

• The volume is ledpi's config, not a bug. ledpi's ncaa_fb shows every live game (show_favorite_teams_only: false). Its journal for 2026-10-03 has 107–121 live polls an hour from 11:00 to 03:00 ET, with 1–18 FBS games live the whole time, at the configured live_update_interval: 30. Each poll is the full day's scoreboard: 865 KB decoded, 63 KB on the wire. That gives 104 MB/h decoded (the 643 MB/6 h) and ~7.6 MB/h on the wire.
• ESPN offers no lighter request with the same data:
  • groups=80 returns the same 54 events.
  • status, state, live, enable and disable parameters are ignored.
  • limit>500 truncates to 25 events.
  • /scoreboard/header lists only 16 events.
  • Per-game summaries are bigger than the whole scoreboard.
• Plugin-side fix. The real waste was on favourites-only boards (the schema default): every other team's kickoff held the poll at 30 s. That is fixed in fix(football): a favourites-only board wakes only for its own kickoffs (3.18.7) ledmatrix-plugins#622: −23% requests and −27% bytes on a replayed Saturday, and the favourite's game is shown from kickoff instead of 14 minutes in.

Type of change

• Bug fix (observability)

Test plan

• Two new tests in test_fetch_service.py:

  • A response built the way requests builds one (a urllib3 HTTPResponse carrying gzip, through HTTPAdapter.build_response) counts bytes = decoded and wire_bytes = gzip size.
  • A response without urllib3 falls back to the decoded size.
  • Mutation check: with wire_bytes=len(body), the first test fails.

• Full suite on Windows against an origin/main (a74b5a2) baseline worktree: the FAILED/ERROR lists are identical (62 failed + 6 errors, all pre-existing Windows failures). The new run has 8,549 passed against the baseline's 8,547: the two new tests.

• Tested on ledpi, a real Raspberry Pi with hardware. I deployed fetch_service.py only and ran 1.08 h on Sunday 2026-10-04 (17:10–18:15 ET), with four NFL games live and ledpi's normal plugins. Per hour:

  plugin	requests	bytes (decoded)	wire_bytes	ratio
  football-scoreboard	325	33.24 MB	3.15 MB	10.5x
  baseball-scoreboard	54	5.16 MB	0.47 MB	10.9x
  hockey-scoreboard	62	8.09 MB	0.73 MB	11.1x
  soccer-scoreboard	545	6.25 MB	1.10 MB	5.7x
  total	986	52.74 MB	5.46 MB	9.7x

  ledpi is back on main and its service has been restarted.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 33 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: cb675712-5fb7-40b3-8913-87c2ebd1a2e2

📥 Commits

Reviewing files that changed from the base of the PR and between caec9f5 and b210fb6.

📒 Files selected for processing (3)

• CHANGELOG.md
• src/common/fetch_service.py
• test/test_fetch_service.py

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 12 complexity

Metric	Results
Complexity	12

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}