Repository navigation
infra: reduce .sources/ submodules to what the site actually needs, watch releases for the rest #355
Description
Context
.sources/ currently holds 16 git submodules, 10 of them release-pinned. During the migration this was the right call: one person, no approvals, and everything an agent needed to verify a claim was on disk. In maintenance mode the cost profile has inverted. Every pinned repo is a recurring bump PR that needs review, and most of those repos contribute nothing to the built site.
This issue proposes a three-tier model and records what has to change in AGENTS.md as a result.
What actually depends on each submodule
| Submodule | Role | Verdict |
|---|---|---|
motoko |
Synced pages under docs/languages/motoko/ and 52 build-time <motokoExamples> file includes |
Keep |
internetidentity |
scripts/sync-ii-spec.mjs writes references/internet-identity-spec.md and verifiable-credentials-spec.md |
Keep |
examples |
Build-time snippet= extraction (plugins/remark-snippet.mjs). One page today, more planned |
Keep |
icskills |
16 of the 17 symlinks in .agents/skills/ point here |
Keep |
dotskills |
.agents/skills/technical-documentation points here |
Keep |
icp-cli |
Agent verification of CLI flags | Convert |
icp-cli-recipes |
Agent verification of recipe config | Convert |
icp-cli-templates |
Agent verification of template layout | Convert |
icp-js-sdk-docs |
Agent verification of JS SDK APIs | Convert |
motoko-core |
Agent verification of mo:core APIs |
Convert |
cdk-rs |
Agent verification of ic-cdk APIs |
Convert |
candid |
Agent verification of the Candid spec | Convert |
response-verification |
Agent verification of certification crates | Convert |
chain-fusion-signer |
Agent verification | Convert |
papi |
Agent verification | Convert |
ic-pub-key |
Agent verification | Convert |
So 5 stay, 11 convert. Note that icskills and dotskills are load-bearing rather than reference material: .agents/skills/ is a directory of symlinks into them, and AGENTS.md requires loading those skills before content work. (If we would rather install skills from skills.internetcomputer.org than vendor them, that is a separate discussion and would drop the count to 3.)
Also worth noting: .sources/examples is 265 MB and is a build dependency of exactly one page right now (guides/backends/https-outcalls.mdx, 6 snippets from 2 examples). Keeping it is a bet that <CodeExample> gets used more widely, which is the plan.
Proposed model
Tier 1: vendored content (submodule). motoko, internetidentity, examples, icskills, dotskills. Their bytes reach the built site or the agent workflow, so they stay on disk and keep their bump procedure.
Tier 2: vendored verification files (fetch, not submodule). For several repos, "verify against upstream" means reading one or two specific files. icp-cli is the clearest case: the rule is "verify every flag against .sources/icp-cli/docs/reference/cli.md", which is a single file out of a 5 MB repo. A sync job can fetch just those files at a pinned release into a small cache directory and record the version. This preserves the offline-verifiable property that the no-guessing rule depends on, at a fraction of the maintenance cost.
Tier 3: release watch only. Everything else. No vendoring: a pin file plus a weekly workflow that diffs upstream through the GitHub API and opens an issue when a new release lands, following the pattern already running in dfinity/icskills (.github/workflows/sync-upstream.yml plus scripts/sync-upstream-check.sh, which writes the diff into the issue body). We already run a narrower version of this here in sync-motoko.yml and sync-ii-spec.yml.
The trigger should be releases that require a content or version bump on the docs site, not every upstream commit. icp-cli-recipes is the motivating example: it is pinned to a June commit that predates the @dfinity/static-site recipe entirely, while upstream has shipped static-site-v0.3.3 and asset-canister-v2.3.0 since. A release watch would have caught that; a submodule that nobody bumps did not.
Required AGENTS.md changes
- Source material table (line ~215): split into the three tiers, with the fetch command for tier 2 and the watch-issue workflow for tier 3.
- "Always read from
.sources/" rule: reword so it still forbids guessing and training-data recall, but points at the tier-2 cache for the repos that no longer have a submodule. - CLI docs slug rule (line ~174): currently says the slug is
1.1, derived from "themajor.minorof the pinned icp-cli release". Withicp-climoving to tier 2 or 3 there is no pinned release to derive it from, so the rule should key off the live docs-site root instead. The links themselves are already being moved to1.3in docs: bump icp-cli docs links to the 1.3 version slug #354. .agents/submodule-bumping.md: reduce to the five tier-1 repos and add the tier-2 refresh and tier-3 watch procedures.- Drop the stale reference to
docs/guides/tools/migrating-from-dfx.md, which no longer exists indocs/.
Open questions
- Tier 2 storage: a committed cache directory (reviewable diffs, slight repo growth) or fetch-on-demand by the agent (no repo growth, needs network and gives no diff to review)? I lean committed, because a diff in a bump PR is how a reviewer notices that a flag changed.
- Do we want the skills vendored at all, or installed from
skills.internetcomputer.org? .sources/icp-cliis still pinned to v1.1.0 while v1.3.0 is out. Bumping it needs the CLI-verification pass in.agents/submodule-bumping.md. Do that once more before converting it to tier 2/3, or convert first and do the verification pass against the fetched file?
Related
- docs: bump icp-cli docs links to the 1.3 version slug #354 bumps the icp-cli docs links to the
1.3slug. TheAGENTS.mdslug rule is deliberately left out of that PR and is item 3 above.
Activity
marc0olo commentedon Aug 26, 2026
This issue is a blocker for #279, and it owes that issue an answer
Recording the coupling so the sequence is explicit.
#279 removes the stale
<!-- Upstream: -->comments (108 of them). Its "keep as-is" criterion is "references an active submodule with a specific file path". This issue changes which submodules exist, so that criterion moves under it.Measured: only 15 of the 108 comments reference active sources at all (the other 96 point at
dfinity/portalor Learn Hub, both retired). Those 15 name 11 repos, and under the proposal here 8 of them stop being submodules:icp-cli,icp-js-sdk-docs,ic-pub-key,cdk-rs,papi,icp-cli-templates,icp-cli-recipes,chain-fusion-signer. Onlyicskills,examples, andinternet-identitysurvive.More importantly, both issues are really about the same unanswered question:
That is the missing half of the release-watch design proposed here. A watch job that says "cdk-rs v0.21 released" is only actionable if something maps the release to the pages that depend on it. The
<!-- Upstream: -->comments were an informal, unenforced attempt at exactly that mapping, and after three months 89% of them point at retired sources, which is the evidence for how well a hand-maintained per-page convention holds up.Add to this issue's scope
Decide the upstream-to-page mapping, choosing between:
grep -rl "ic-cdk" docs/). Zero maintenance, and sufficient whenever the changed API name is itself the search term.source_repo/source_ref(already declared insrc/content.config.tsand currently unused), written by the sync scripts so it is correct by construction. Only meaningful for genuinely synced trees, not for hand-written pages that merely consulted a repo.A reasonable default: (c) for synced trees plus (a) for everything else, with (b) only if the watch jobs show that grep is not enough in practice. Correctness by construction beats a convention that depends on an author remembering.
Whatever is chosen here becomes the criterion #279 executes against, and it determines whether that issue deletes all 108 comments or converts 15 of them.
AGENTS.md consequence
Add to the AGENTS.md list already in this issue: line 25 mandates the
<!-- Upstream: -->comment. If #279 deletes them while that rule stands, the next content PR adds a new one. The rule has to be rewritten with whatever mapping mechanism is chosen above.Related
validate --allfails on main; what synced trees are exempt from, which touches the same sync scripts)docs: sync certified-assets docs and make static-site the primary fro…