diff --git a/README.md b/README.md index 95c11d47..e9d70edc 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,34 @@ MultiDocumenter.make( ) ``` +### Limiting copied versions and linking to all upstream versions + +When aggregating many packages, you can copy only selected deployed versions to reduce storage size of the final website: + +```julia +MultiDocumenter.MultiDocRef( + upstream = joinpath(clonedir, "SomePackage"), + path = "SomePackage", + name = "SomePackage.jl", + giturl = "https://github.com/SomeOrg/SomePackage.jl.git", + versions = MultiDocumenter.VersionSelection( + ["stable", "dev"]; + # optional: adds a "See All Versions" entry to the version selector + all_versions_url = "https://someorg.github.io/SomePackage.jl/", + ), +) +``` + +- Only the listed version directories are copied (plus root files like `index.html` and `versions.js`). + Versions that do not exist upstream are reported and skipped; if none of them exist, all versions are copied. +- The package's `index.html` is regenerated to redirect to the **first** version that was copied, and that + version becomes the canonical one for the package, so list them in the order you want. +- `versions.js` is rewritten to include only copied versions, and Documenter's "outdated version" markers + (`DOCUMENTER_NEWEST`, `DOCUMENTER_STABLE`) are repointed at versions that were copied. +- If `all_versions_url` is set, the version selector gets a `See All Versions` entry pointing there, so that + the versions that were not copied stay reachable on the upstream site. It must be an absolute `http(s)` URL + naming the site root, and is not derived from `giturl`. + ![example](sample.png) ## Deployment diff --git a/src/MultiDocumenter.jl b/src/MultiDocumenter.jl index 896affca..39c116ed 100644 --- a/src/MultiDocumenter.jl +++ b/src/MultiDocumenter.jl @@ -14,7 +14,10 @@ end SearchConfig(index_versions = ["stable"], engine = MultiDocumenter.PageFind, lowfi = false) `index_versions` is a vector of relative paths used for generating the search index. Only -the first matching path is considered. +the first matching path is considered (except for `PageFind`, which indexes all of them). +This is one global list, so a `MultiDocRef` that publishes none of these paths -- e.g. +because its [`VersionSelection`](@ref) keeps different ones -- contributes nothing to the +search index; MultiDocumenter warns about each such ref during the build. `engine` may be `MultiDocumenter.PageFind`, `MultiDocumenter.FlexSearch`, `MultiDocumenter.Stork`, or a module that conforms to the expected API (which is currently undocumented). `lowfi = true` will try to minimize search index size. Only relevant for flexsearch. @@ -42,9 +45,69 @@ users can implement their own custom components. """ abstract type DropdownComponent end +""" + struct VersionSelection + VersionSelection(versions; all_versions_url = nothing) + +Selects which deployed versions of a [`MultiDocRef`](@ref) are copied into the aggregate, +and where the ones that are left out can still be found. + +Aggregating packages with long `gh-pages` histories can push the combined site past a host's +size limit (GitHub Pages allows 1 GB), and most of that history is rarely read. Copying only +`["stable", "dev"]` keeps the site small; `all_versions_url` then keeps the rest reachable. + +* `versions`: the version directories to copy, e.g. `["stable", "dev"]`. Entries that do not + exist upstream are reported and skipped; if none of them exist, all versions are copied + instead. The **first** version that was copied is what the ref's `index.html` redirects to + and what MultiDocumenter treats as canonical, so the order matters. +* `all_versions_url`: absolute http(s) URL of the upstream documentation site *root*, e.g. + `https://org.github.io/Pkg.jl/`. When given, a "See All Versions" entry pointing there is + added to the version selector. Use the site root rather than a specific version: + Documenter's selector tries to stay on the equivalent page, so it may append the current + page's path to this URL. + +```julia +MultiDocumenter.VersionSelection( + ["stable", "dev"]; + all_versions_url = "https://org.github.io/Pkg.jl/", +) +``` +""" +struct VersionSelection + versions::Vector{String} + all_versions_url::Union{String, Nothing} + + function VersionSelection( + versions::AbstractVector{<:AbstractString}; + all_versions_url::Union{AbstractString, Nothing} = nothing, + ) + isempty(versions) && throw( + ArgumentError("VersionSelection: `versions` must name at least one version") + ) + if all_versions_url !== nothing + # A relative URL would point back into the aggregate, which is precisely where + # the versions we did not copy are not. + startswith(all_versions_url, "http://") || + startswith(all_versions_url, "https://") || + throw( + ArgumentError( + "VersionSelection: `all_versions_url` must be an absolute http(s) URL, got $(repr(all_versions_url))" + ) + ) + end + return new( + collect(String, versions), + all_versions_url === nothing ? nothing : String(all_versions_url), + ) + end +end + +VersionSelection(version::AbstractString; kwargs...) = + VersionSelection([version]; kwargs...) + """ struct MultiDocRef <: DropdownComponent - MultiDocRef(; upstream, name, path, giturl = "", branch = "gh-pages", fix_canonical_url = true) + MultiDocRef(; upstream, name, path, giturl = "", branch = "gh-pages", fix_canonical_url = true, versions = nothing) Represents one set of docs that will get an entry in the MultiDocumenter navigation. @@ -61,6 +124,12 @@ Represents one set of docs that will get an entry in the MultiDocumenter navigat * `branch`: Git branch of `giturl` where the docs will be pulled from (defaults to `gh-pages`) * `fix_canonical_url`: this can be set to `false` to disable the canonical URL fixing for this `MultiDocRef` (see also `canonical_domain` for [`make`](@ref)). +* `versions`: a [`VersionSelection`](@ref), to copy only some of the deployed versions + instead of the whole `upstream` tree. All versions are copied by default. Root files (e.g. + `index.html`, `versions.js`) are always copied. When a selection is given, `versions.js` is + rewritten to list only the copied versions, and `DOCUMENTER_NEWEST` / `DOCUMENTER_STABLE` + are repointed at copied versions so that Documenter does not mark kept pages as outdated + (and `noindex`) or link its banner at a version that is no longer there. """ struct MultiDocRef <: DropdownComponent upstream::String @@ -69,8 +138,13 @@ struct MultiDocRef <: DropdownComponent fix_canonical_url::Bool giturl::String branch::String + versions::Union{VersionSelection, Nothing} end +# so that positional construction keeps working as it did before `versions` was added +MultiDocRef(upstream, path, name, fix_canonical_url, giturl, branch) = + MultiDocRef(upstream, path, name, fix_canonical_url, giturl, branch, nothing) + function MultiDocRef(; upstream, name, @@ -78,8 +152,9 @@ function MultiDocRef(; giturl = "", branch = "gh-pages", fix_canonical_url = true, + versions::Union{VersionSelection, Nothing} = nothing, ) - return MultiDocRef(upstream, path, name, fix_canonical_url, giturl, branch) + return MultiDocRef(upstream, path, name, fix_canonical_url, giturl, branch, versions) end """ @@ -116,7 +191,34 @@ struct BrandImage imagepath::String end +""" +Warn about any [`MultiDocRef`](@ref) that will contribute nothing to the search index. + +`SearchConfig.index_versions` is a single global list, but each ref decides for itself which +versions it publishes (see [`VersionSelection`](@ref)), so a ref can easily end up with none +of the indexed version directories in its output. Its pages are then published and linked +from the navigation, but unreachable through the search box -- which is easy to miss, since +nothing else about the build looks wrong. +""" +function warn_unindexed_refs(root, docs::Vector, index_versions::AbstractVector) + for ref in filter(x -> x isa MultiDocRef, docs) + refroot = joinpath(root, ref.path) + isdir(refroot) || continue + any(dir -> isdir(joinpath(refroot, dir)), index_versions) && continue + @warn( + "None of this package's pages will be searchable: it publishes none of the " * + "indexed versions. Add one of the versions it does publish to " * + "SearchConfig(index_versions = ...).", + path = ref.path, + index_versions, + published = filter(d -> isdir(joinpath(refroot, d)), readdir(refroot)), + ) + end + return nothing +end + function walk_outputs(f, root, docs::Vector, dirs::Vector{String}) + warn_unindexed_refs(root, docs, dirs) for ref in filter(x -> x isa MultiDocRef, docs) p = joinpath(root, ref.path) for dir in dirs @@ -345,6 +447,212 @@ function maybe_clone(docs::Vector) return nothing end +# --- VersionSelection: copy only selected version dirs and link to the full upstream site --- + +const SEE_ALL_VERSIONS_LABEL = "See All Versions" + +const GENERATED_REDIRECT_MARKER = "" + +""" +Whether `page` is a generated redirect stub rather than a rendered documentation page. + +Injecting stylesheets and scripts into these would make a page whose only job is to redirect +first fetch and run the whole aggregate front end, and would destroy the marker itself. +""" +is_generated_redirect(page::AbstractString) = + startswith(page, "") || + startswith(page, GENERATED_REDIRECT_MARKER) + +""" +Copy all root files and the listed version directories from `src` to `dst`, skipping `.git` +and any version directory not in `versions`. + +Returns the subset of `versions` that was actually copied, in the given order. Requested +versions that are missing upstream, or are symlinks with no target left (which happens on +`gh-pages` branches where old version directories were pruned by hand), are reported and +skipped rather than aborting the build. + +Everything is copied with `follow_symlinks = true`, so that e.g. `stable -> v5.5.0` becomes +a real directory and the result does not depend on paths that were not copied. +""" +function cp_select_versions(src::String, dst::String, versions::Vector{String}) + mkpath(dst) + verset = Set(versions) + copied, dangling = String[], String[] + for entry in readdir(src) + entry == ".git" && continue + full = joinpath(src, entry) + if islink(full) && !ispath(full) + # ispath() follows the link, so this is a link whose target is gone + entry in verset && push!(dangling, entry) + elseif isfile(full) + cp(full, joinpath(dst, entry); force = true, follow_symlinks = true) + elseif isdir(full) && entry in verset + cp(full, joinpath(dst, entry); force = true, follow_symlinks = true) + push!(copied, entry) + end + end + isempty(dangling) || @warn "Skipping version directories whose symlink target is missing" src dangling + # dangling ones are already reported above, with a more specific reason + missing_versions = filter(v -> v ∉ copied && v ∉ dangling, versions) + isempty(missing_versions) || + @warn "Some selected versions do not exist upstream and were skipped" src missing_versions + # preserve the caller's ordering: it decides which version the ref lands on + copiedset = Set(copied) + return filter(in(copiedset), versions) +end + +"""The version a copied version directory holds, as recorded in its `siteinfo.js`.""" +function version_of_directory(outpath::AbstractString, dir::AbstractString) + siteinfo = joinpath(outpath, dir, "siteinfo.js") + isfile(siteinfo) || return nothing + m = match(r"var\s+DOCUMENTER_CURRENT_VERSION\s*=\s*\"(.*?)\"", read(siteinfo, String)) + return isnothing(m) ? nothing : m[1] +end + +""" +Rewrite `versions.js` so that the version selector only offers `kept_versions`, and so that +`DOCUMENTER_NEWEST` names the newest version we kept. + +`DOCUMENTER_NEWEST` matters because Documenter's `warner.js` compares it against each +directory's `DOCUMENTER_CURRENT_VERSION` and, on a mismatch, marks the page `noindex` and +shows an "older version" banner. Left pointing at a version that was not copied, that +de-indexes perfectly current pages. +""" +function rewrite_versions_js(outpath::String, kept_versions::Vector{String}) + vjs = joinpath(outpath, "versions.js") + isfile(vjs) || return nothing + content = read(vjs, String) + new_list = "[\n \"" * join(kept_versions, "\",\n \"") * "\"\n]" + content = replace( + content, + r"var\s+DOC_VERSIONS\s*=\s*\[[\s\S]*?\]" => "var DOC_VERSIONS = " * new_list, + ) + newest = newest_kept_version(outpath, kept_versions) + if newest !== nothing + content = replace( + content, + r"var\s+DOCUMENTER_NEWEST\s*=\s*\"(.*?)\"" => "var DOCUMENTER_NEWEST = \"$(newest)\"", + ) + end + write(vjs, content) + return nothing +end + +""" +The highest version number among the copied version directories, or `nothing` if none of +them holds a release (e.g. only `dev` was kept, which never triggers the banner anyway). +""" +function newest_kept_version(outpath::AbstractString, kept_versions::Vector{String}) + newest = nothing + for dir in kept_versions + version = version_of_directory(outpath, dir) + version === nothing && continue + occursin(Base.VERSION_REGEX, version) || continue + parsed = tryparse(VersionNumber, version) + parsed === nothing && continue + if newest === nothing || parsed > newest[2] + newest = (version, parsed) + end + end + return newest === nothing ? nothing : newest[1] +end + +"""The copied directory that `DOCUMENTER_STABLE` should point at, if we can name one.""" +function stable_target(outpath::AbstractString, kept_versions::Vector{String}) + "stable" in kept_versions && return "stable" + newest = newest_kept_version(outpath, kept_versions) + newest === nothing && return nothing + for dir in kept_versions + version_of_directory(outpath, dir) == newest && return dir + end + return nothing +end + +""" +Point `DOCUMENTER_STABLE` in each copied `siteinfo.js` at a directory that still exists. + +Documenter's "older version" banner links to `../\$(DOCUMENTER_STABLE)`, which 404s if the +stable directory was not among the copied versions. +""" +function rewrite_stable_target!(outpath::String, kept_versions::Vector{String}) + target = stable_target(outpath, kept_versions) + target === nothing && return nothing + for dir in kept_versions + siteinfo = joinpath(outpath, dir, "siteinfo.js") + isfile(siteinfo) || continue + content = read(siteinfo, String) + new_content = replace( + content, + r"var\s+DOCUMENTER_STABLE\s*=\s*\"(.*?)\"" => "var DOCUMENTER_STABLE = \"$(target)\"", + ) + new_content == content || write(siteinfo, new_content) + end + return nothing +end + +""" +Map the output subdirectory of each version-limited `MultiDocRef` to its "See All Versions" +URL. Refs that copy all their versions get no entry: the point of the link is to reach the +versions that were left out, and there are none. +""" +function see_all_versions_urls(docs::Vector) + urls = Dict{Vector{String}, String}() + for doc in Iterators.filter(x -> x isa MultiDocRef, flatten_dropdowncomponents(docs)) + doc.versions === nothing && continue + url = doc.versions.all_versions_url + url === nothing && continue + urls[splitpath(doc.path)] = url + end + return urls +end + +""" +The "See All Versions" URL that applies to the page at `relative_path`, if any. + +At most one ref can match: two refs whose paths nest would copy over each other in +[`make_output_structure`](@ref) to begin with. +""" +function see_all_versions_url_for(urls::Dict{Vector{String}, String}, relative_path::AbstractString) + isempty(urls) && return nothing + parts = splitpath(relative_path) + for (docpath, url) in urls + n = length(docpath) + # the page has to live *below* the ref's directory, not be the directory itself + n < length(parts) && view(parts, 1:n) == docpath && return url + end + return nothing +end + +""" +Add a "See All Versions" entry to Documenter's version selector, pointing at `url`. + +Documenter's own selector code (`assets/html/js/versions.js`) only ever appends to the +selector -- it never clears it, and it matches the versions from `DOC_VERSIONS` against +existing options by their text. So an `