feat(licenses): resolve requirements.txt pin licenses from PyPI and add --offline - #113
Conversation
…dd --offline Library components parsed from requirements.txt now carry the license their package declares on PyPI. For each exact == pin the scan reads pypi.org/pypi/<name>/<version>/json and writes the result into the component's licenses[] (with an aisbom:license:source=pypi property), and into licenseDeclared for SPDX 2.3 output. Unpinned requirements are skipped. Sources are tried from most to least precise: PEP 639 license_expression, an SPDX id or expression in the license field, a short table of unambiguous free-text spellings, a classifier naming exactly one license, then any other short declaration kept as a license name. License texts pasted into the field and ambiguous classifiers (BSD License) resolve to nothing. Best-effort: a PyPI outage, rate limit, unknown package or exhausted time budget costs that dependency its license and prints a warning; exit codes and model findings never change. legal_status is never derived from it. Answers are cached in ~/.aisbom/pypi_license_cache.json (30 days resolved, 24 hours unresolved, failures never cached), and an all-hit run does not rewrite it. --offline (scan, score) or AISBOM_OFFLINE=1 makes no network access of any kind: no PyPI or OSV lookup, no telemetry, no update check. Remote targets and --share are refused up front. The subprocess CLI tests now run with AISBOM_OFFLINE=1: they are out of reach of the conftest stubs and were reaching the network from the suite.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 84389683c2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| inputs: | ||
| directory: | ||
| description: 'Directory to scan for AI model artifacts.' | ||
| description: 'Directory to scan for AI model artifacts. Exact requirements.txt pins found in it (package name and version only) are looked up on pypi.org from the runner to fill in their declared licenses; set AISBOM_OFFLINE=1 in the step env to skip network lookups.' |
There was a problem hiding this comment.
When an Action user follows this description by setting AISBOM_OFFLINE=1 while share: true is configured, the new CLI rejects the combination with exit 1, but action/entrypoint.sh only propagates SCAN_EXIT == 2 and ultimately exits 0. The workflow therefore reports success despite producing no new SBOM; worse, if the configured output path already exists, the wrapper proceeds to comment on or upload that stale file. Handle the CLI's exit 1 in the Action wrapper before processing the output.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 3d5b314. action/entrypoint.sh now rejects AISBOM_OFFLINE together with share: true before the scan runs and exits 1, so the job fails and no pre-existing sbom.json is commented on or uploaded. Exit 1 is added to the entrypoint's documented exit codes. Tests cover the refused pair (no scan, no helper invocations, exit 1) and offline without share (scans normally). I kept this narrow on purpose: the wrapper not re-raising a CLI exit 1 in general (e.g. a missing directory) predates this PR, and changing it would flip the job result for existing users, so it is tracked separately.
| mapped = {_CLASSIFIERS[c] for c in classifiers if c in _CLASSIFIERS} | ||
| # Two license classifiers could mean a choice, a combination or a stale | ||
| # leftover. Joining them into an expression would assert one of those. | ||
| if len(mapped) == 1: | ||
| return ResolvedLicense(mapped.pop(), is_spdx=True) |
There was a problem hiding this comment.
When PyPI supplies one mapped classifier together with an unmapped license classifier, such as MIT plus the generic BSD classifier, the comprehension drops the BSD entry before counting and this branch emits MIT as the sole declared license. Multiple license classifiers can represent dual or alternative licensing, so this creates an incorrect compliance assertion for those packages despite the stated no-guessing behavior; determine whether there is exactly one license classifier before returning a mapped value.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 3d5b314. Every License :: classifier is now counted before mapping, except the bare License :: OSI Approved / DFSG approved category nodes, which name no license. More than one declared license classifier resolves nothing, so MIT plus the generic BSD classifier no longer becomes plain MIT. Regression tests cover MIT beside BSD License and beside Other/Proprietary License, and confirm the bare category node is not treated as a second license.
…share in the Action Classifiers: a mapped classifier beside one the table cannot map (MIT plus the generic BSD License) used to resolve to plain MIT, because unmapped entries were dropped before counting. Every License :: classifier now counts, bar the bare OSI Approved / DFSG approved category nodes, and more than one resolves nothing. Action: with AISBOM_OFFLINE=1 in the step env and share: true, the CLI refuses with exit 1, which the entrypoint does not re-raise, so the job passed and a sbom.json already in the workspace would have been commented on and uploaded. The entrypoint now rejects that pair before scanning and exits 1.
1 participant
What
Dependency components parsed from
requirements.txtnow carry the license their package declares on PyPI, and a new--offlineflag makes a scan touch no network at all.Before this, every library component had a name and version and nothing else, so
aisbom scorelost most of the Licenses dimension for information PyPI publishes, and its remediation had to say dependency licenses were "not yet resolved automatically".License resolution (default-on)
==pin, the scan readshttps://pypi.org/pypi/<name>/<version>/json. Ranges (torch>=2.0) are skipped and counted in the summary line, since they don't name the installed release.licenses[], plus anaisbom:license:source=pypiproperty. SPDX 2.3 output carries SPDX-valid values aslicenseDeclared(licenseConcludedstaysNOASSERTION). SPDX 3.0 and Markdown output skip the lookup.license_expression→ SPDX id/expression inlicense→ a short table of unambiguous free-text spellings (Apache 2.0 License→Apache-2.0) → a classifier naming exactly one license → any other short declaration kept as a license name. A license text pasted into the field (numpy ships ~46KB of it) and ambiguous classifiers (BSD License,Apache Software License) produce no license.legal_statusis never derived from it. For libraries nothing downstream turnslicenses[]into one:aisbom diffreads the model description, so upgrading reports no drift.~/.aisbom/pypi_license_cache.json: 30 days resolved, 24 hours unresolved, failures never cached. A run where every pin is a hit does not rewrite the file.--offline/AISBOM_OFFLINE=1On
scanandscore(the env var works for every command): no PyPI lookup, no OSV lookup, no telemetry, no update check.hf:///https://targets and--shareare refused with exit 1 before anything is fetched.Disclosure
README gets new "Dependency licenses from PyPI" and "Offline and air-gapped scans" sections, a Privacy entry for the PyPI lookup, and an Action data-flow note (a fourth outbound call: registry lookups).
action.yml,action/README_ACTION.mdanddocs/air-gapped-guide.mdare updated to match.Verification
poetry run pytest --cov=aisbom --cov-fail-under=85: 1376 passed, 93.61% (1269 / 93.29% at branch point).aisbom bypass-scorecard --checkpasses.getaddrinfo/connect, and asserts--offlinemakes zero attempts. With the telemetry gate mutated out it fails on three real connection attempts toapi.aisbom.io.torch==2.13.0,transformers==5.13.1,numpy==1.26.4,requests==2.32.3, a nonexistent package,scipy>=1.10), live PyPI:licenses[]+ source property on torch (full six-part SPDX expression), transformers (Apache-2.0) and requests (Apache-2.0). numpy, the unknown package and the range get nothing.aisbom diff old new: "No changes detected."fetched_atunchanged, file not rewritten).--offlinecomponents are byte-identical to the old CLI's;scan hf://… --offlineexits 1.AISBOM_OFFLINE=1.--offlinestill writes an SBOM and refuseshf://.Not in this PR