perf(deps): sweep raw CDN across marker probes before any anonymous Contents API call by ShaoLOWu · Pull Request #3348 · microsoft/apm

ShaoLOWu · 2026-10-22T12:09:10Z

Summary

Implements the maintainer-approved scope from #3283 (apm-scope:v1): stop spending anonymous Contents API calls during virtual-package validation and marketplace registration on public github.com sources.

1. Virtual-package validation: raw CDN first sweep

validate_virtual_package_exists previously probed marker files one at a time, and each miss on a public github.com repo escalated to an anonymous Contents API call. A Claude-plugin-layout repo (plugin.json deep in the marker list) could burn 4+ calls before the layout was found — a large share of the 60 req/h anonymous budget.

Before the per-marker loop, every marker path is now swept through the raw CDN (raw.githubusercontent.com, no rate limit) via a new raw_only pass-through on download_raw_file / download_github_file. A single hit validates the package with zero API calls.
If the sweep finds nothing, the existing per-marker probes and both fallbacks run unchanged — API-only, private, GHES/ADO/GitLab hosts keep their previous behavior.
GitHubThrottleError still propagates untouched.

2. Marketplace add: reuse the discovery response

_auto_detect_path probes candidate locations, then fetch_marketplace(force_refresh=True) immediately re-fetched the same catalog it had just downloaded.

New _auto_detect_path_with_payload returns (path, data); the command layer passes the captured payload to fetch_marketplace(discovery_payload=...), which skips the redundant network round-trip.
The sidecar cache is still written, so subsequent force_refresh/freshness semantics are identical.

Testing

New unit tests (12):

tests/unit/deps/test_virtual_package_cdn_first_sweep.py (6): sweep hits every marker raw-only; first-hit success spends zero Contents API calls; full-miss path escalates to API and keeps fallback behavior; non-public hosts skip the sweep; single-file virtual packages unaffected.
tests/unit/marketplace/test_catalog_discovery_reuse.py (6): discovery payload is reused (no second fetch); explicit force_refresh still fetches; cache file still written.

Local run (Python 3.13, Windows): all new tests pass; the pre-existing failure set (tests/unit/deps/, test_github_downloader.py, test_virtual_package_multi_install.py) is byte-identical before and after this change (13 Windows/Unix-environment failures: symlinks, GitLab/Gitea routing, /dev/null — none touched by this PR). ruff check clean on all modified files.

AI-Generated PR disclosure

This PR was authored with AI assistance (Claude-based coding agent working under human direction): issue analysis, fix design, implementation, and test authoring were agent-assisted; scope was user-approved against the maintainer's approved-scope comment on #3283 before implementation. All changes were validated locally as described above, and the human reviewed and confirmed the final diff before submission.

Fixes #3283

…ontents API call Virtual-package validation on public github.com repos spent anonymous Contents API calls per marker file before discovering the layout, and `apm marketplace add` re-fetched the catalog immediately after the discovery probe had just downloaded it. On the default 60 req/h anonymous budget a single Claude-plugin-layout install could burn 4+ calls (validation + registration combined). - validate_virtual_package_exists: probe every marker path through the raw CDN first (zero API calls) before falling back to the existing per-marker Contents API probes; fallbacks and non-public hosts are unchanged - download_raw_file: new raw_only pass-through for CDN-only probes - marketplace add: reuse the catalog response captured during path discovery instead of re-fetching it (sidecar cache semantics kept) - tests: 12 new unit tests covering sweep ordering, API-spend assertions, fallback retention, and catalog reuse Fixes microsoft#3283

Copilot

🟡 Changes recommended

Incorrect and incomplete test mocks cause assertion failures and unintended network requests.

3 open issues

What changed in this PR

This PR addresses #3283 by reducing redundant GitHub API requests during virtual-package validation and marketplace registration.

Changes:

Sweep package markers through the raw CDN before allowing Contents API fallback.
Reuse the discovered marketplace catalog while preserving cache writes.
Add regression tests and update existing discovery mocks.

File	Description
`tests/unit/marketplace/test_marketplace_commands.py`	Updates discovery mocks to return path/payload tuples.
`tests/unit/marketplace/test_catalog_discovery_reuse.py`	Tests payload reuse, refresh behavior, and cache writes.
`tests/unit/deps/test_virtual_package_cdn_first_sweep.py`	Tests CDN-first validation and fallback behavior.
`tests/unit/deps/test_github_downloader_validation.py`	Allows keyword arguments in an existing download mock.
`src/apm_cli/marketplace/client.py`	Returns discovery payloads and supports reusing them.
`src/apm_cli/deps/github_downloader.py`	Passes the `raw_only` option through downloader methods.
`src/apm_cli/deps/github_downloader_validation.py`	Adds the CDN-only marker sweep before legacy probes.
`src/apm_cli/deps/download_strategies.py`	Prevents API fallback for raw-only requests.
`src/apm_cli/commands/marketplace/__init__.py`	Passes discovered catalog data into marketplace fetching.

_{Review effort: Balanced}

+            discovered_manifest = None
        else:
-            detected_path = _auto_detect_path(probe_source)
+            detected_path, discovered_manifest = _auto_detect_path_with_payload(probe_source)


+                "uses_public_github_anonymous_first",
+                return_value=False,
+            ),
+            patch.object(downloader._strategies, "try_raw_download") as cdn_mock,


+
+        with (
+            patch.object(downloader, "download_raw_file", side_effect=wrapper),
+            patch.object(downloader._strategies, "try_raw_download", return_value=None),


ShaoLOWu · 2026-10-23T09:33:12Z

@microsoft-github-policy-service agree

ShaoLOWu · 2026-10-23T09:33:23Z

@microsoft-github-policy-service agree

…de-network sweep tests Addresses Copilot review feedback on microsoft#3348: - Retarget the traceback tests (phase3/surface) and the five wave7 patch sites from _auto_detect_path to _auto_detect_path_with_payload, returning (path, payload) tuples so discovery stays mocked instead of executing the real fetch path. - test_non_public_host_skips_sweep: use an actual non-public-host dependency and simulate legacy per-marker misses on the download strategy seam (a bare MagicMock return was accepted as file content), stubbing the directory and ls-remote fallbacks at the validation module boundary so the host boundary is checked fully offline. - test_virtual_file_probe_keeps_legacy_semantics: mock the auth callback and return controlled API 404s (matching the full-miss test) so the single-file probe never attempts real GitHub requests. Sweep test suite runtime drops from ~57s (network-dependent) to ~3s. Verified: 6/6 sweep tests green; 332 tests across the touched marketplace/deps suites pass; tests/unit/deps failure set identical to the pre-change Windows baseline (7 pre-existing symlink/routing failures); ruff clean.

Sergio Sisternes (sergio-sisternes-epam) · 2026-11-03T10:33:29Z

Thank you for implementing the scoped fix. Maintainer approval for the bounded scope is recorded in #3283, but neither the PR nor linked issue currently carries status/accepted. Under the triage rule, this PR is labelled status/deferred until the maintainer reconciles that status. This recommendation is advisory only, not merge approval.

Generated by autopilot-pr-triage-worker. This comment is AI-generated and may contain errors.

ShaoLOWu requested review from Daniel Meppiel (danielmeppiel) and Sergio Sisternes (sergio-sisternes-epam) as code owners October 22, 2026 12:09

Copilot AI balanced review requested due to automatic review settings October 22, 2026 12:09

Copilot started reviewing on behalf of ShaoLOWu October 22, 2026 12:10

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

Navigation Menu

Repository navigation

perf(deps): sweep raw CDN across marker probes before any anonymous Contents API call - #3348

perf(deps): sweep raw CDN across marker probes before any anonymous Contents API call#3348
ShaoLOWu wants to merge 2 commits into
microsoft:mainfrom
ShaoLOWu:perf/cdn-first-marker-sweep

ShaoLOWu commented Oct 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

ShaoLOWu commented Oct 23, 2026

Uh oh!

ShaoLOWu commented Oct 23, 2026

Uh oh!

Sergio Sisternes (sergio-sisternes-epam) commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ShaoLOWu commented Oct 22, 2026

Summary

1. Virtual-package validation: raw CDN first sweep

2. Marketplace add: reuse the discovery response

Testing

AI-Generated PR disclosure

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

ShaoLOWu commented Oct 23, 2026

Uh oh!

ShaoLOWu commented Oct 23, 2026

Uh oh!

Sergio Sisternes (sergio-sisternes-epam) commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants