Skip to content

Commit fcbed7b

Browse files
GuanzhouSongCopilot
GuanzhouSong
and
Copilot
authored
Publish the full DocumentDB package set and prove the repository installs (#149)
* Publish the full DocumentDB package set and prove the repo installs v0.116-0 replaced the single `postgresql-N-documentdb` extension package with a multi-package layout (`documentdb` meta, `documentdb-N`, `documentdb-common`, `documentdb-gateway`, `documentdb-postgresql-tools`). The mirror's asset filter still matched only the extension, so 14 of the release's 22 packages were dropped: `apt install documentdb` could never have resolved, and none of the new RPMs matched at all because they carry no `rhel9-` prefix. Rebuilding from the newest release alone would also have deleted the deb11/deb12/deb13/ubuntu22 components and the whole rhel8 repository, because v0.116-0 ships Tier-1 (ubuntu24 + rhel9) only. Every host already pointed at one of those would have started failing `apt update` with "Component 'ubuntu22' is not defined". So the pool is now filled additively, per (pool, package name, arch): the newest release wins, and older releases contribute only packages no newer release provides. Nothing disappears when a release narrows its matrix, and `postgresql-16-documentdb` survives on ubuntu24/rhel9 even though v0.116-0 narrowed Tier 1 to PostgreSQL 17/18. Once a release ships every distribution again, the older ones stop contributing on their own. The metadata verifier could not have caught any of this: every file it looks for was present and correct, the repository was simply unsatisfiable. Add a smoke test that resolves `documentdb` against the repository about to be published, for both APT and DNF. It solves the dependency graph without downloading ~500 MB of PostgreSQL and PostGIS, and asserts the transaction really pulls the stack, so an empty meta package cannot pass either. Rewrite PACKAGE-INSTALL.md for the two-tier reality, and document what a first-time install actually needs: that the gateway listens on all interfaces by default and how to restrict it, how to verify an install and read its version (mongosh reports the emulated MongoDB version, not DocumentDB's), where the ports, logs and config live, the per-major systemd unit names, and how to upgrade, reset or remove. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8df9084a-ccaf-432c-b015-2ccd8893a9d8 * Never fill gaps from releases newer than a pinned version The site pins the mirrored release with the DOCUMENTDB_VERSION repository variable (currently v0.113-0). The additive fill walked every other published release regardless of age, so a pin to a release older than the v0.116-0 multi-package layout would have added a newer `documentdb` meta package on top of the pinned extension. Its `documentdb-N (>= 0.116.0)` dependency cannot be satisfied by `postgresql-N-documentdb 0.113-0`, so the published repository would have been unsatisfiable -- and the pin would have been quietly defeated. A pin means "serve this version", so only the pinned release and older ones may contribute. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8df9084a-ccaf-432c-b015-2ccd8893a9d8 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8df9084a-ccaf-432c-b015-2ccd8893a9d8
1 parent 2b92d86 commit fcbed7b

3 files changed

Lines changed: 570 additions & 140 deletions

File tree

‎.github/scripts/download_packages.sh‎

Lines changed: 235 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -73,26 +73,165 @@ generate_hashes() {
7373

7474
echo "Downloading packages from $REPO releases"
7575

76-
if [ "$DOCUMENTDB_VERSION" = "latest" ]; then
77-
if release=$(curl -fqs "https://api.github.com/repos/${REPO}/releases" | python3 -c "import sys, json; releases = json.load(sys.stdin); print(json.dumps(releases[0])) if releases else sys.exit(1)")
78-
then
79-
tag="$(echo "$release" | python3 -c "import sys, json; print(json.load(sys.stdin)['tag_name'])")"
80-
echo "Using latest release: $tag"
81-
else
82-
echo "Error: Could not fetch latest release information"
83-
exit 1
84-
fi
85-
else
86-
tag="$DOCUMENTDB_VERSION"
87-
if ! release=$(curl -fqs "https://api.github.com/repos/${REPO}/releases/tags/$tag")
88-
then
89-
echo "Error: Version $tag not found in releases"
90-
exit 1
91-
fi
92-
echo "Using specified release: $tag"
76+
# ---------------------------------------------------------------------------
77+
# Release selection
78+
#
79+
# The primary release supplies the site's release-info.json and is the version
80+
# users are told about. But a release only ships the distributions that were
81+
# in its own build matrix: v0.116-0, for example, ships Tier-1 (ubuntu24 +
82+
# rhel9) only, while v0.114-0 shipped seven distributions. Rebuilding the
83+
# repository from the primary release alone would therefore DELETE the
84+
# deb11/deb12/deb13/ubuntu22 components and the whole rhel8 repository from
85+
# documentdb.io, and every host already pointed at one of them would start
86+
# failing `apt update` with "Component 'ubuntu22' is not defined". That is a
87+
# client-visible outage, not a cosmetic regression.
88+
#
89+
# So the repository is built additively: the primary release fills every
90+
# distribution it ships, then progressively older releases are consulted ONLY
91+
# to fill distributions still missing. A distribution is claimed by the newest
92+
# release that ships it and is never overwritten by an older one. Once a
93+
# release ships every distribution again, the older ones stop contributing
94+
# by themselves - no cleanup required.
95+
# ---------------------------------------------------------------------------
96+
MAX_RELEASES="${MAX_RELEASES:-8}"
97+
98+
RELEASES_JSON=$(mktemp)
99+
if ! curl -fqs "https://api.github.com/repos/${REPO}/releases?per_page=100" > "$RELEASES_JSON"; then
100+
echo "Error: Could not fetch release list"
101+
exit 1
93102
fi
94103

104+
# Ordered list of tags to consider, newest first. Drafts and prereleases are
105+
# skipped: they are not what a repository-backed `apt install` should serve.
106+
TAG_LIST=$(DOCUMENTDB_VERSION="$DOCUMENTDB_VERSION" python3 - "$RELEASES_JSON" <<'PY'
107+
import json, os, sys
108+
109+
releases = json.load(open(sys.argv[1]))
110+
published = [r for r in releases if not r.get("draft") and not r.get("prerelease")]
111+
if not published:
112+
sys.exit("Error: no published releases found")
113+
114+
requested = os.environ.get("DOCUMENTDB_VERSION", "latest")
115+
if requested != "latest":
116+
# The API returns releases newest-first.
117+
index = next((i for i, r in enumerate(published)
118+
if r["tag_name"] == requested), None)
119+
if index is None:
120+
sys.exit(f"Error: Version {requested} not found in releases")
121+
# A pin means "serve this version". Only the pinned release and OLDER ones
122+
# may contribute: pulling gap-fillers from NEWER releases would defeat the
123+
# pin, and worse, it would mix releases that depend on each other. Pinning
124+
# to a release that predates the multi-package layout would otherwise add a
125+
# newer `documentdb` meta package whose `documentdb-N (>= X)` dependency the
126+
# pinned extension cannot satisfy - an unsatisfiable repository.
127+
ordered = published[index:]
128+
else:
129+
ordered = published
130+
131+
print("\n".join(r["tag_name"] for r in ordered))
132+
PY
133+
)
134+
135+
PRIMARY_TAG=$(printf '%s\n' "$TAG_LIST" | head -n 1)
136+
echo "Primary release: $PRIMARY_TAG"
137+
138+
# Packages already placed, keyed by "pool|name|arch". A newer release always
139+
# wins; an older release contributes only packages the newer ones do not
140+
# provide at all. That is what keeps `postgresql-16-documentdb` alive on
141+
# ubuntu24/rhel9 after v0.116-0 narrowed Tier 1 to PostgreSQL 17/18, instead of
142+
# silently deleting a package the install docs still tell people to use.
143+
SEEN_FILE=$(mktemp)
144+
145+
is_claimed() { case " $1 " in *" $2 "*) return 0 ;; *) return 1 ;; esac; }
146+
147+
# "name|arch" identity of a package file, independent of its version.
148+
# postgresql-16-documentdb_0.114-0_amd64.deb -> postgresql-16-documentdb|amd64
149+
# documentdb_0.116.0_all.deb -> documentdb|all
150+
# postgresql16-documentdb-0.114.0-1.el8.x86_64.rpm -> postgresql16-documentdb|x86_64
151+
# documentdb-17-0.116.0-1.noarch.rpm -> documentdb-17|noarch
152+
package_identity() {
153+
local f="$1"
154+
case "$f" in
155+
*.deb)
156+
printf '%s|%s' "${f%%_*}" "$(printf '%s' "$f" | sed -E 's/.*_([^_]+)\.deb$/\1/')"
157+
;;
158+
*.rpm)
159+
local base="${f%.rpm}"
160+
local arch="${base##*.}"
161+
local nvr="${base%.*}"
162+
# Drop the trailing VERSION and RELEASE fields to leave the package name.
163+
printf '%s|%s' "$(printf '%s' "$nvr" | sed -E 's/-[^-]+-[^-]+$//')" "$arch"
164+
;;
165+
esac
166+
}
167+
168+
claim_package() {
169+
# claim_package <pool> <filename> -> 0 when this is a new package for the pool
170+
local key="$1|$(package_identity "$2")"
171+
if grep -qxF "$key" "$SEEN_FILE" 2>/dev/null; then
172+
return 1
173+
fi
174+
printf '%s\n' "$key" >> "$SEEN_FILE"
175+
return 0
176+
}
177+
178+
# Map an asset filename to its APT component, or empty when it is not a
179+
# distribution-prefixed .deb. Any DocumentDB package for that distribution
180+
# matches - the extension, the gateway, the tools, documentdb-common, the
181+
# per-major stand-alone and the meta package - because the v0.116-0 packaging
182+
# redesign ships all of them and an extension-only filter would silently drop
183+
# every package that makes `apt install documentdb` resolve.
184+
deb_component_for() {
185+
case "$1" in
186+
*dbgsym*) echo "" ;;
187+
deb11-*.deb) echo "deb11" ;;
188+
deb12-*.deb) echo "deb12" ;;
189+
deb13-*.deb) echo "deb13" ;;
190+
ubuntu22.04-*.deb) echo "ubuntu22" ;;
191+
ubuntu24.04-*.deb) echo "ubuntu24" ;;
192+
*) echo "" ;;
193+
esac
194+
}
195+
196+
deb_pool_for() {
197+
case "$1" in
198+
deb11) echo "$DEB_POOL_DEB11" ;;
199+
deb12) echo "$DEB_POOL_DEB12" ;;
200+
deb13) echo "$DEB_POOL_DEB13" ;;
201+
ubuntu22) echo "$DEB_POOL_UBUNTU22" ;;
202+
ubuntu24) echo "$DEB_POOL_UBUNTU24" ;;
203+
esac
204+
}
205+
206+
# RPM naming is less uniform than DEB. The extension RPMs carry a distro
207+
# prefix (rhel9-...), the gateway carries a dist tag (....el9.x86_64.rpm), and
208+
# the meta / per-major / common / tools RPMs are noarch with NO dist tag at
209+
# all, so they cannot be routed by name. Those EL-agnostic packages are placed
210+
# in exactly the pools this release populated via its prefixed assets - never
211+
# into a pool served by a different release, where their >= dependencies on a
212+
# same-version documentdb-N would be unsatisfiable.
213+
rpm_pool_for() {
214+
case "$1" in
215+
*debuginfo*|*debugsource*) echo "" ;;
216+
rhel8-*.rpm) echo "rhel8" ;;
217+
rhel9-*.rpm) echo "rhel9" ;;
218+
*.el8.*.rpm) echo "rhel8" ;;
219+
*.el9.*.rpm) echo "rhel9" ;;
220+
*) echo "" ;;
221+
esac
222+
}
223+
95224
mkdir -p out/packages
225+
226+
for tag in $TAG_LIST; do
227+
MAX_RELEASES=$((MAX_RELEASES - 1))
228+
[ "$MAX_RELEASES" -lt 0 ] && break
229+
230+
if ! release=$(curl -fqs "https://api.github.com/repos/${REPO}/releases/tags/$tag"); then
231+
echo "::warning::Could not fetch release $tag, skipping"
232+
continue
233+
fi
234+
96235
ASSETS_FILE=$(mktemp)
97236
echo "$release" | python3 -c "
98237
import sys, json
@@ -101,73 +240,88 @@ for asset in data.get('assets', []):
101240
print(f\"{asset['name']}|{asset['browser_download_url']}\")
102241
" > "$ASSETS_FILE"
103242

104-
# Process each asset
105-
while IFS='|' read -r filename download_url
106-
do
107-
if [ -z "$filename" ]; then
108-
continue
109-
fi
110-
111-
if [[ "$filename" == *.deb ]]; then
112-
wget -q -P out/packages "$download_url"
113-
114-
if [[ "$filename" =~ ^deb11-postgresql-[0-9]+-documentdb.*\.deb$ ]]; then
115-
GOT_DEB=1
116-
mkdir -p "$DEB_POOL_DEB11"
117-
clean_name=$(echo "$filename" | sed 's/^deb11-//')
118-
cp "out/packages/$filename" "$DEB_POOL_DEB11/$clean_name"
119-
sign_deb_package "$DEB_POOL_DEB11/$clean_name"
120-
elif [[ "$filename" =~ ^deb12-postgresql-[0-9]+-documentdb.*\.deb$ ]]; then
121-
GOT_DEB=1
122-
mkdir -p "$DEB_POOL_DEB12"
123-
clean_name=$(echo "$filename" | sed 's/^deb12-//')
124-
cp "out/packages/$filename" "$DEB_POOL_DEB12/$clean_name"
125-
sign_deb_package "$DEB_POOL_DEB12/$clean_name"
126-
elif [[ "$filename" =~ ^deb13-postgresql-[0-9]+-documentdb.*\.deb$ ]]; then
127-
GOT_DEB=1
128-
mkdir -p "$DEB_POOL_DEB13"
129-
clean_name=$(echo "$filename" | sed 's/^deb13-//')
130-
cp "out/packages/$filename" "$DEB_POOL_DEB13/$clean_name"
131-
sign_deb_package "$DEB_POOL_DEB13/$clean_name"
132-
elif [[ "$filename" =~ ^ubuntu22\.04-postgresql-[0-9]+-documentdb.*\.deb$ ]]; then
133-
GOT_DEB=1
134-
mkdir -p "$DEB_POOL_UBUNTU22"
135-
clean_name=$(echo "$filename" | sed 's/^ubuntu22\.04-//')
136-
cp "out/packages/$filename" "$DEB_POOL_UBUNTU22/$clean_name"
137-
sign_deb_package "$DEB_POOL_UBUNTU22/$clean_name"
138-
elif [[ "$filename" =~ ^ubuntu24\.04-postgresql-[0-9]+-documentdb.*\.deb$ ]]; then
243+
# First pass: which RPM pools does this release populate? Needed to route the
244+
# EL-agnostic noarch packages, which carry no distro hint in their names.
245+
serve_rpm=""
246+
while IFS='|' read -r filename _; do
247+
[ -z "$filename" ] && continue
248+
case "$filename" in
249+
*.rpm)
250+
pool=$(rpm_pool_for "$filename")
251+
if [ -n "$pool" ] && ! is_claimed "$serve_rpm" "$pool"; then
252+
serve_rpm="$serve_rpm $pool"
253+
fi ;;
254+
esac
255+
done < "$ASSETS_FILE"
256+
257+
added=0
258+
while IFS='|' read -r filename download_url; do
259+
[ -z "$filename" ] && continue
260+
261+
case "$filename" in
262+
*.deb)
263+
comp=$(deb_component_for "$filename")
264+
[ -z "$comp" ] && continue
265+
claim_package "$comp" "$filename" || continue
266+
wget -q -P out/packages "$download_url" || { echo "::warning::download failed: $filename"; continue; }
139267
GOT_DEB=1
140-
mkdir -p "$DEB_POOL_UBUNTU24"
141-
clean_name=$(echo "$filename" | sed 's/^ubuntu24\.04-//')
142-
cp "out/packages/$filename" "$DEB_POOL_UBUNTU24/$clean_name"
143-
sign_deb_package "$DEB_POOL_UBUNTU24/$clean_name"
144-
fi
145-
elif [[ "$filename" == *.rpm ]]; then
146-
echo "Processing RPM: $filename"
147-
wget -q -P out/packages "$download_url"
148-
149-
if [[ "$filename" =~ ^rhel8-postgresql[0-9]+-documentdb.*\.rpm$ ]]; then
150-
GOT_RPM=1
151-
mkdir -p "$RPM_POOL_RHEL8"
152-
clean_name=$(echo "$filename" | sed 's/^rhel8-//')
153-
echo " Adding to RHEL 8: $filename -> $clean_name"
154-
cp "out/packages/$filename" "$RPM_POOL_RHEL8/$clean_name"
155-
elif [[ "$filename" =~ ^rhel9-postgresql[0-9]+-documentdb.*\.rpm$ ]]; then
156-
GOT_RPM=1
157-
mkdir -p "$RPM_POOL_RHEL9"
158-
clean_name=$(echo "$filename" | sed 's/^rhel9-//')
159-
echo " Adding to RHEL 9: $filename -> $clean_name"
160-
cp "out/packages/$filename" "$RPM_POOL_RHEL9/$clean_name"
161-
else
162-
echo " Skipping RPM (does not match patterns): $filename"
163-
fi
164-
else
165-
wget -q -P out/packages "$download_url"
166-
fi
268+
pool=$(deb_pool_for "$comp")
269+
mkdir -p "$pool"
270+
# The distro prefix disambiguates release assets; it is not part of
271+
# the package name and must not survive into the pool.
272+
clean_name=$(echo "$filename" | sed -E 's/^(deb1[123]|ubuntu2[24]\.04)-//')
273+
cp "out/packages/$filename" "$pool/$clean_name"
274+
sign_deb_package "$pool/$clean_name"
275+
added=$((added + 1)) ;;
276+
277+
*.rpm)
278+
pool_name=$(rpm_pool_for "$filename")
279+
if [ -n "$pool_name" ]; then
280+
targets="$pool_name"
281+
else
282+
case "$filename" in
283+
*.noarch.rpm) targets="$serve_rpm" ;;
284+
*) targets="" ;;
285+
esac
286+
fi
287+
[ -z "$targets" ] && continue
288+
289+
downloaded=0
290+
clean_name=$(echo "$filename" | sed -E 's/^rhel[89]-//')
291+
for t in $targets; do
292+
claim_package "$t" "$filename" || continue
293+
case "$t" in
294+
rhel8) dest="$RPM_POOL_RHEL8" ;;
295+
rhel9) dest="$RPM_POOL_RHEL9" ;;
296+
*) continue ;;
297+
esac
298+
if [ "$downloaded" -eq 0 ]; then
299+
wget -q -P out/packages "$download_url" || { echo "::warning::download failed: $filename"; break; }
300+
downloaded=1
301+
GOT_RPM=1
302+
fi
303+
mkdir -p "$dest"
304+
echo " Adding to ${t}: $filename -> $clean_name"
305+
cp "out/packages/$filename" "$dest/$clean_name"
306+
added=$((added + 1))
307+
done ;;
308+
309+
*)
310+
# Non-package assets (SHA256SUMS, manifest.txt, ...) are mirrored only
311+
# for the primary release, which is what the site links to.
312+
[ "$tag" = "$PRIMARY_TAG" ] && wget -q -P out/packages "$download_url" ;;
313+
esac
167314
done < "$ASSETS_FILE"
168-
169-
rm -f "$ASSETS_FILE"
170315

316+
echo "Release $tag contributed $added package file(s)"
317+
rm -f "$ASSETS_FILE"
318+
done
319+
320+
rm -f "$RELEASES_JSON" "$SEEN_FILE"
321+
322+
# release-info.json describes the primary release only: it is the "what is the
323+
# current version" feed for the site, not an inventory of the pool.
324+
release=$(curl -fqs "https://api.github.com/repos/${REPO}/releases/tags/${PRIMARY_TAG}")
171325
echo "$release" | python3 -c "
172326
import sys, json
173327
data = json.load(sys.stdin)

0 commit comments

Comments
 (0)