feat(facade): check repo size and enforce clone limit before cloning (#458) by Dipro-cyber · Pull Request #459 · chaoss/CollectOSS

Dipro-cyber · 2026-08-26T06:27:42Z

Description
Some git repositories are exceptionally large, which can rapidly exhaust local disk space when cloned. This PR introduces a configurable maximum repository clone size limit (max_clone_size_kb) and safety margin (clone_size_safety_margin) in CollectOSS.

Before running git clone, CollectOSS queries platform APIs (GitHub REST API or GitLab API) to obtain reported repository size stats and computes estimated_size_kb = reported_size_kb * (1 + safety_margin). If the estimated clone size exceeds max_clone_size_kb, cloning is skipped and the repository collection status is set to Failed Clone.

This PR fixes #458

Notes for Reviewers

Added max_clone_size_kb (default 0, disabled) and clone_size_safety_margin (default 0.5, 50% margin) to Facade config in collectoss/application/config.py and FacadeHelper.
Added check_repo_size_limit() and pre-clone enforcement logic in collectoss/tasks/git/util/facade_worker/facade_worker/repofetch.py.
Added comprehensive unit tests in tests/test_tasks/test_git/test_repo_size_limit.py.

Signed commits

Yes, I signed my commits.

Generative AI disclosure

This contribution was assisted or created by Generative AI tools.
- What tools were used? Google Antigravity
- How were these tools used? Code implementation, unit testing, and drafting PR description.
- Did you review these outputs before submitting this PR? Yes

MoralCode · 2026-08-26T14:49:24Z

im a little confused on the theory of how your safety margin system is meant to work. The original idea for that is because the size as github reports it is very likely not accurate to the actual size of the whole directory that you get after running git clone.

I have other branches that are very work-in-progress for refactoring a lot of CollectOSS to use pygit2 rather than shelling out to a git subprocess (for speed/performance). I think if we wanted the size limit configuration we offer to users to be an actual/comparable measure to real on-disk first-clone size, maybe we should see if that library gives us a way to query the actual clone size of a repo.

Happy to chat on slack about this too, im very curious what you find

Dipro-cyber · 2026-09-05T21:02:10Z

@MoralCode thank u for the context sir, sorry i was a bit inactive. the github api size field is from a bare repo so it can differ significantly from the actual clone size. happy to revisit the safety margin approach once the pygit2 refactor lands, since that library likely has a way to get the actual disk usage after clone. for now, should i simplify this to just use the raw github-reported size without a safety margin, and let users set the limit knowing it's an approximation? that would make the behavior more predictable while still solving the core disk space problem.

MoralCode · 2026-09-05T22:45:39Z

I think the main idea is to figure out the disk space before the clone so we can avoid cloning if its too big.

Maybe thats where the conversation in #49 could help (I.e. if we migrated CollectOSS to use bare clones)

…haoss#458) Signed-off-by: Diptesh Roy <droy88333@gmail.com>

…ent E2E worker timeout Signed-off-by: Diptesh Roy <droy88333@gmail.com>

Per MoralCode's review feedback, the safety margin concept was confusing because the GitHub/GitLab API size field is already an approximation (measured from a bare repo, not an actual clone). Adding a multiplier on top of an already-inaccurate value made the limit unpredictable. Simplified to compare the raw reported size directly against max_clone_size_kb. Users set the limit knowing it's an approximation of the bare repo size, which is the most honest and predictable behavior. Also removes clone_size_safety_margin from config and FacadeHelper. Signed-off-by: Diptesh Roy <droy88333@gmail.com>

Dipro-cyber · 2026-09-24T19:35:39Z

@MoralCode sir can u check this once, the new feature by @drkrillo has been a help here!

MoralCode · 2026-09-24T19:43:22Z

The underlying issue has had additional notes added since this PR was filed.

The additional notes Include a way to estimate the size of a checkout (rather than using a fudge factor).

This, combined with the bare size of a git repo should give a decent estimate of repo size

Could you update this PR to account for that more precise method of checking the size?

Dipro-cyber · 2026-09-24T20:02:24Z

The underlying issue has had additional notes added since this PR was filed.

The additional notes Include a way to estimate the size of a checkout (rather than using a fudge factor).

This, combined with the bare size of a git repo should give a decent estimate of repo size

Could you update this PR to account for that more precise method of checking the size?

Oh yes sure sir, I will check out the notes.

Per MoralCode's research, the GitHub API 'size' field alone underestimates by ~4% because it only measures the bare repo. Adding the working tree file size (sum of all blob sizes from /git/trees/HEAD?recursive=1) gives a much more accurate estimate of actual on-disk clone size. If the tree is truncated (very large repo), falls back to bare size only. Updated tests to cover the combined estimation logic. Signed-off-by: Diptesh Roy <droy88333@gmail.com>

Dipro-cyber · 2026-09-24T21:13:41Z

@MoralCode the new commit now uses the combined bare repo size + file tree blob sizes for github repos. falls back to bare size only if the tree is truncated. also updated tests to cover the combined logic.

MoralCode · 2026-09-24T21:16:13Z

Wdym if the tree is truncated?

Dipro-cyber · 2026-09-24T21:37:15Z

Wdym if the tree is truncated?

sir the github git tree API (/git/trees/HEAD?recursive=1) returns truncated: true for very large repos where the tree has too many entries to return in a single response. in that case we can't sum all blob sizes, so we fall back to just the bare repo size from the metadata api. for most repos this won't happen, github truncates at ~100,000 tree entries.

- Default max_clone_size_kb changed from 0 (disabled) to 5242880 KB (5 GB) as suggested by MoralCode — provides a sensible out-of-the-box limit - Reverted contributor_interface.py and tasks.py to upstream — those changes are unrelated to the repo size limit feature Signed-off-by: Diptesh Roy <droy88333@gmail.com>

Signed-off-by: Diptesh Roy <droy88333@gmail.com>

Dipro-cyber · 2026-09-26T19:31:18Z

@MoralCode sir can you give this a check once?

MoralCode

Took another look and found some additional design related things.

Its going to take me a while to get around to testing this as well, but thanks for the reminders to keep this moving!

Address MoralCode's design review: - Split check_repo_size_limit into three functions: _get_github_repo_size_kb: raises on API error or non-dict response _get_gitlab_repo_size_kb: calls raise_for_status, raises on missing field check_repo_size_limit: policy-only, calls forge-specific fetchers - Fail CLOSED: API error, malformed response, or unsupported forge with limit>0 returns (False, None) to block the clone rather than allow it - Truncated GitHub tree: partial blob data is counted even when truncated=True, since the endpoint is not paginated and partial is better than zero (per MoralCode's feedback) - Fix TypeError crash in git_repo_initialize: estimated_kb can be None when fail-closed, so the error message now handles both cases - Tests: 12 tests covering disabled limit, under/over/exact, truncated tree (blocked and allowed), API error, malformed response, unsupported forge, and GitLab path Signed-off-by: Diptesh Roy <droy88333@gmail.com>

Pre-review audit fixes: - Correct _get_github_repo_size_kb docstring: GitHub's non-recursive tree API does allow full traversal via sub-tree requests; the previous claim that 'partial is all we can get' was inaccurate. Document the actual trade-off: we intentionally use partial data as a lower-bound estimate to avoid N round-trips on large repos. - Update inline comment and truncation warning message to reflect the lower-bound framing consistently. - Add test_gitlab_repo_under_limit: GitLab repo below the configured limit should allow the clone (was previously untested). - Add test_negative_limit_treated_as_disabled: negative limit values are treated the same as 0 (disabled); make this explicit in tests. - Remove unused GitCloneError import from test file. Signed-off-by: Diptesh Roy <droy88333@gmail.com>

Dipro-cyber · 2026-09-27T14:38:23Z

@MoralCode hopefully sir no more reviews are needed after this.

MoralCode · 2026-09-27T23:58:50Z

                "run_analysis": 1,
                "run_facade_contributors": 1,
                "commit_messages": 1,
+                "max_clone_size_kb": 5242880,


We should probably include the presence of this setting in the docs. people may be surprised if their repos stop cloning if we dont document it (and ideally link to the docs page in the error message)

MoralCode · 2026-09-27T23:58:55Z

@MoralCode hopefully sir no more reviews are needed after this.

This is a relatively major new feature so I suspect there may end up being more. I still have yet to actually run this code/test it with a live instance, Ive just been giving quick code reviews in response to your requests.

including github and gitlab in this also makes it trickier to test since we dont have full gitlab support yet

Per MoralCode's request to document this setting so operators aren't surprised when repos stop cloning. - Add max_clone_size_kb section to configuration-file-reference.rst describing default (5 GB), how to disable (set to 0), and the truncated-tree lower-bound caveat - Include the docs URL in the GitCloneError message so the error log points operators directly to the configuration reference Signed-off-by: Diptesh Roy <droy88333@gmail.com>

Dipro-cyber · 2026-09-28T18:30:59Z

@MoralCode sir did the necessary changes.

MoralCode · 2026-09-30T20:09:43Z

+                docs_ref = (
+                    "See max_clone_size_kb in the configuration reference: "
+                    "https://github.com/chaoss/CollectOSS/blob/main/docs/source"
+                    "/development-guide/configuration-file-reference.rst"


can you link to the compiled docs on docs.collectoss.org?

Dipro-cyber requested a review from MoralCode as a code owner August 26, 2026 06:27

Dipro-cyber force-pushed the feat/enforce-repo-clone-size-limit-458 branch from 9ab30b8 to b0cb8fe Compare August 26, 2026 07:00

Dipro-cyber added 3 commits September 24, 2026 22:40

feat(facade): check repo size and enforce clone limit before cloning (c…

0c6b034

…haoss#458) Signed-off-by: Diptesh Roy <droy88333@gmail.com>

fix(facade): skip API lookups for non-searchable local emails to prev…

6d41dac

…ent E2E worker timeout Signed-off-by: Diptesh Roy <droy88333@gmail.com>

Dipro-cyber force-pushed the feat/enforce-repo-clone-size-limit-458 branch from 4af2829 to 4cc4d18 Compare September 24, 2026 17:10

MoralCode requested changes Sep 25, 2026

View reviewed changes

Comment thread collectoss/application/config.py Outdated

Comment thread collectoss/tasks/github/facade_github/contributor_interfaceable/contributor_interface.py Outdated

MoralCode requested changes Sep 25, 2026

View reviewed changes

Comment thread tests/test_tasks/test_git/test_repo_size_limit.py Outdated

fix: remove unrelated email validation test from repo size limit tests

1727dd1

Signed-off-by: Diptesh Roy <droy88333@gmail.com>

MoralCode requested changes Sep 26, 2026

View reviewed changes

Comment thread tests/test_tasks/test_git/test_repo_size_limit.py Outdated

Comment thread collectoss/tasks/git/util/facade_worker/facade_worker/repofetch.py Outdated

Comment thread collectoss/tasks/git/util/facade_worker/facade_worker/repofetch.py Outdated

Dipro-cyber added 2 commits September 27, 2026 20:05

Dipro-cyber force-pushed the feat/enforce-repo-clone-size-limit-458 branch from dfeb967 to 22037ae Compare September 27, 2026 14:35

MoralCode mentioned this pull request Sep 27, 2026

Check fadace disk for available space before cloning #496

Open

MoralCode reviewed Sep 27, 2026

View reviewed changes

MoralCode requested changes Sep 30, 2026

View reviewed changes

Conversation

Dipro-cyber commented Aug 26, 2026

Uh oh!

MoralCode commented Aug 26, 2026

Uh oh!

Dipro-cyber commented Sep 5, 2026

Uh oh!

MoralCode commented Sep 5, 2026

Uh oh!

Dipro-cyber commented Sep 24, 2026

Uh oh!

MoralCode commented Sep 24, 2026

Uh oh!

Dipro-cyber commented Sep 24, 2026

Uh oh!

Dipro-cyber commented Sep 24, 2026

Uh oh!

MoralCode commented Sep 24, 2026

Uh oh!

Dipro-cyber commented Sep 24, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Dipro-cyber commented Sep 26, 2026

Uh oh!

MoralCode left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Dipro-cyber commented Sep 27, 2026

Uh oh!

MoralCode Sep 27, 2026

Choose a reason for hiding this comment

Uh oh!

MoralCode commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dipro-cyber commented Sep 28, 2026

Uh oh!

MoralCode Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MoralCode commented Sep 27, 2026 •

edited

Loading