diff --git a/docs/ai/workflows/publish-calibration-tree-artifacts.md b/docs/ai/workflows/publish-calibration-tree-artifacts.md index bbd80d35..84d2cfe7 100644 --- a/docs/ai/workflows/publish-calibration-tree-artifacts.md +++ b/docs/ai/workflows/publish-calibration-tree-artifacts.md @@ -57,18 +57,24 @@ would assign different immutable paths to the same release. country-specific secret is also present in GitHub Actions. 2. Publish compressed non-index files first, then `index.json.gz`. 3. Verify uploaded bytes through a consistent Blob read and decompression. -4. Add every immutable source build to `manifest.json`; change +4. For a release build, audit the uploaded build (its index and every file the + index lists) before adding its entry, so an incomplete upload never gets a + manifest entry. +5. Add every immutable source build to `manifest.json`; change `latestReleaseBuildArtifactId` only when the country's upstream pointer (`latest.json`, or the registration's `pointer_path`) still names the same - release and commit. -5. For staging publication, include only `passed`, `published`, or `completed` + release and commit, and that release did not fail in the same run. +6. For staging publication, include only `passed`, `published`, or `completed` runs from the exact webhook commit. -6. Report the source ID, build artifact ID, commit, part count, and measured sizes without reporting +7. Report the source ID, build artifact ID, commit, part count, and measured sizes without reporting credentials. Every release webhook reconciles the full enumerated release history. Every staging webhook reconciles all finalized runs at that webhook commit. A complete entry is audited and skipped; only missing entries or missing files are built. +A release or staging run that cannot be published or repaired is reported as +`failed` with its error, the remaining sources are still reconciled, and the +command then exits with an error naming every failed source. There is no scheduled reconciliation, so the next webhook repairs omissions from an earlier failed or missed publication. diff --git a/docs/calibration-tree-artifacts.md b/docs/calibration-tree-artifacts.md index e1cb1e0b..09b61a89 100644 --- a/docs/calibration-tree-artifacts.md +++ b/docs/calibration-tree-artifacts.md @@ -153,6 +153,17 @@ ineligible and is not treated as an error. The next repository webhook repeats the inventory check, so it also finds a historical release missed by an older webhook. There is no scheduled reconciliation. +A release whose build cannot be published or repaired, for example because the +schema 8 reader refuses its diagnostics, its tag moved to another commit, or a +repair's rebuilt index differs from the recorded digest, is reported as +`failed` with the error. The remaining releases are still reconciled and the +latest-release selection is still updated, unless the release the upstream +pointer names is the one that failed; the workflow then fails. A failed release +gets no new manifest entry (an entry it already had is left unchanged), unless +the manifest write was stored and only its read-back check failed ("Calibration +tree manifest verification failed."). The latest-release selection is never +moved to a failed release. + #### Microcosm release lines Microcosm publishes a release line (microcosm#823) under a constant release id: @@ -395,10 +406,17 @@ Failure meanings and recovery: - `502` from the application means GitHub rejected or did not receive the workflow dispatch; check `GITHUB_ACTIONS_DISPATCH_TOKEN` and its Actions permission. -- A failed release workflow means publication started but did not complete. - Release reconciliation stops at its first error, so the releases after a - failing one were not reconciled. Rerun that workflow after correcting the - reported error. +- A failed release workflow either stopped before reconciling any release (for + example, the manifest, the release listing or the repository's tags could + not be read, or one tag's folder or one folder's commit could not be + resolved), or printed a report in which at least one release is `failed` + or `latestReleaseFailure` says why the latest-release selection was not + updated. In the second case every release with another outcome was + reconciled, and the error names the failed releases. Every later release + workflow for that country retries them and fails again until their errors + are corrected. Rerunning does not correct a refused repair of a published + tree ("Rebuilt calibration index differs from immutable manifest entry") or + a release whose tag moved ("Released dataset … changed from … to …"). - A failed staging workflow either stopped before reconciling any run (for example, the staging repository was unavailable or listed incompatible runs), or printed a report in which at least one run is `failed`. In the @@ -438,19 +456,50 @@ Remove `--dry-run` to publish. For staging, an automated invocation also passes repository state. The GitHub workflow exposes the same `dry_run` option for an authenticated verification using Actions secrets. -Each outcome is `complete`, `published`, `repaired`, or `ineligible`, and -staging outcomes can also be `failed`. +Each outcome is `complete`, `published`, `repaired`, `ineligible`, or +`failed`. `complete` means no bundle construction or source-diagnostics download occurred. `repaired` means the publisher deterministically rebuilt an existing identity and restored only missing files. Any reconstructed index must match the digest already recorded in the manifest. -Release reconciliation stops at the first release it cannot publish or repair. -Staging reconciliation records that run as `failed`, with the error as its -`reason`, and continues with the remaining runs; after printing the report the -command exits with an error naming every failed run. A refused staging repair -therefore no longer leaves the runs after it unpublished on every later -invocation. +Release and staging reconciliation both record a source they cannot publish or +repair as `failed`, with the error as its `reason`, and continue with the +remaining sources; after printing the report the command exits with an error +naming every failed source. One release or run that cannot be published +therefore no longer leaves the sources after it unpublished on every later +invocation. Errors before the per-source loop still stop reconciliation before +any source and print no report: reading the manifest, listing releases, tags or +staging runs, the first Blob listing, and, for releases, the Hugging Face +lookups that build the release list (the folder probe and +`release_manifest.json` of each tag the manifest does not already pin at its +commit, and the commit of each folder release without a tag). +An error in one of those lookups therefore stops every release, as one failed +release did before. + +For a release, the steps that write run in this order: upload the bundle's +files (the index last), list the stored files and audit the build against its +own index, then add the manifest entry. An error at any step before the +manifest write leaves a new release without a manifest entry, and leaves a +release that already had an entry with that entry unchanged. Files a new +release uploaded before the error stay under the build's ID, which is a digest +of the country, the release ID and the source files' digests; no manifest entry +refers to them, and the next reconciliation rebuilds the same ID and checks +each existing file byte for byte before reusing it. Each manifest write reads +the stored manifest, adds one entry, and writes it back on the condition that +the stored manifest has not changed since that read (after a conflict it reads +once more and writes without the condition). A failed release makes no +manifest write, so the writes for the releases after it are the same as +without it. + +After the releases, release reconciliation points `latestReleaseBuildArtifactId` +at the release the upstream pointer names. It leaves the selection unchanged, +and records why in the report's `latestReleaseFailure`, when that release +failed in the same run, is not completely published, or the pointer cannot be +read; the command then exits with an error. A dry run reads the pointer and +reports the same `latestReleaseFailure` when the pointer cannot be read or names +a release that failed in the same run; it does not check whether the named +release is completely published, because it writes no manifest entries. `CALIBRATION_TREE_MAX_RAW_BYTES` defaults to 100,000,000 bytes per file, in addition to the fixed summary- and detail-shard limits. diff --git a/frontend/packages/calibration-tree/src/publish-calibration-tree-golden.test.ts b/frontend/packages/calibration-tree/src/publish-calibration-tree-golden.test.ts index bc35e834..ee44dd3e 100644 --- a/frontend/packages/calibration-tree/src/publish-calibration-tree-golden.test.ts +++ b/frontend/packages/calibration-tree/src/publish-calibration-tree-golden.test.ts @@ -204,15 +204,22 @@ for (const [name, description] of Object.entries(RELEASE_DESCRIPTIONS) as Array< ); } -test("a repair refuses a rebuilt index that differs from the recorded digest", async () => { +test("a repair refuses a rebuilt index that differs from the recorded digest and records it as that release's failure", async () => { const release = goldenReleaseInputs()["uk-release"]; const entry = { ...recordedReleaseEntry(release, "uk-release"), indexSha256: "0".repeat(64) }; globalThis.fetch = huggingFaceReleaseRepository(release); - await expect(withRegisteredRepositories(() => + const report = await withRegisteredRepositories(() => reconcileReleaseBuilds(release.country, "unused-blob-token", true, storeMissingIndex(release, entry)) - )).rejects.toThrow("Rebuilt calibration index differs from immutable manifest entry"); -}); + ); + + expect(report.outcomes).toEqual([{ + sourceId: release.releaseId, + status: "failed", + buildArtifactId: entry.buildArtifactId, + reason: `Rebuilt calibration index differs from immutable manifest entry ${entry.buildArtifactId}.`, + }]); +}, BUILD_TIMEOUT_MS); // Two finalized UK staging runs. The newer one, which reconciliation reaches // first, already has a manifest entry whose index has gone missing, and the diff --git a/frontend/packages/calibration-tree/src/publish-calibration-tree.test.ts b/frontend/packages/calibration-tree/src/publish-calibration-tree.test.ts index ddd93a42..47c17e23 100644 --- a/frontend/packages/calibration-tree/src/publish-calibration-tree.test.ts +++ b/frontend/packages/calibration-tree/src/publish-calibration-tree.test.ts @@ -6,6 +6,7 @@ import { parsePublisherOptions, migrateReleaseTimestamps, reconcileReleaseBuilds, + reconciliationFailureMessage, readUpstreamLatest, resolvePublicationSourceSha, manifestBuildForSource, @@ -255,23 +256,49 @@ function placeholderEntry(hfRepo = UK_REPO): CalibrationTreeManifestEntry { }; } -function memoryStore(initial: CalibrationTreeManifest) { +/** + * A manifest and Blob store in memory. It records each upload, listing, audit + * and manifest write in the order the publisher makes them. Every audit + * reports a complete build, except for the release IDs in + * `incompleteAfterUpload`, whose uploaded build audits as missing a file. + */ +function memoryStore( + initial: CalibrationTreeManifest, + options: { incompleteAfterUpload?: ReadonlySet } = {}, +) { let manifest = initial; const writes: Array<{ entry: CalibrationTreeManifestEntry; makeLatest?: boolean }> = []; const uploads: CalibrationTreeBundle[] = []; + const operations: string[] = []; return { writes, uploads, + operations, current: () => manifest, readManifest: async () => manifest, - listBlobs: async () => new Map(), - audit: async () => ({ complete: true, repairable: true, reasons: [], index: null }), + listBlobs: async () => { + operations.push("list files"); + return new Map(); + }, + audit: async (entry: CalibrationTreeManifestEntry) => { + operations.push(`audit ${entry.releaseId}`); + return entry.releaseId && options.incompleteAfterUpload?.has(entry.releaseId) + ? { + complete: false, + repairable: true, + reasons: [`missing calibration-trees/uk/${entry.buildArtifactId}/tier-1.json.gz`], + index: null, + } + : { complete: true, repairable: true, reasons: [], index: null }; + }, updateManifest: async (entry: CalibrationTreeManifestEntry, makeLatest?: boolean) => { + operations.push(`write ${entry.releaseId}${makeLatest ? " as latest" : ""}`); writes.push({ entry, makeLatest }); manifest = withCalibrationTreeManifestEntry(manifest, "uk", entry, makeLatest); return manifest; }, uploadBundle: async (bundle: CalibrationTreeBundle) => { + operations.push(`upload ${bundle.index.build.releaseId}`); uploads.push(bundle); const stored = (file: CalibrationTreeBundle["files"][number]) => ({ pathname: file.path, @@ -651,6 +678,249 @@ test("release reconciliation publishes each line cut from its folder and follows } }); +// A file the schema 8 reader refuses: its second row puts the first row's +// category under another provider. A release built from it cannot be +// published. +const REFUSED_SCHEMA_8_DIAGNOSTICS = { + schema_version: 8, + targets: [ + SCHEMA_8_DIAGNOSTICS.targets[0], + { + ...SCHEMA_8_DIAGNOSTICS.targets[0], + name: "other.target@2025", + target_name: "other.target", + hierarchy: { + ...SCHEMA_8_DIAGNOSTICS.targets[0].hierarchy, + provider: { id: "other", label: "Other provider" }, + category: { id: "provider.category", label: "Category label", provider_id: "other" }, + target: { id: "other.target", label: "Other target label" }, + }, + }, + ], +}; +const REFUSED_SCHEMA_8_ERROR = + "Schema 8 category provider.category references inconsistent providers: provider and other."; + +function lineStoreWithJuneLatest( + options: { incompleteAfterUpload?: ReadonlySet } = {}, +) { + return memoryStore( + withCalibrationTreeManifestEntry( + withCalibrationTreeManifestEntry(emptyCalibrationTreeManifest(), "uk", placeholderEntry()), + "uk", + juneEntry(), + true, + ), + options, + ); +} + +test("release reconciliation records a release it cannot build as failed and still publishes the releases after it", async () => { + // The dense tagged build (DENSE_CUT) sorts first, so on its own it used to + // stop reconciliation before the national tagged build and the + // latest-release selection. + const store = lineStoreWithJuneLatest(); + globalThis.fetch = lineRepositoryFetch([], (directory) => + directory === DENSE_LINE ? REFUSED_SCHEMA_8_DIAGNOSTICS : SCHEMA_8_DIAGNOSTICS + ); + try { + const report = await reconcileReleaseBuilds("uk", "blob-token", false, store); + const status = Object.fromEntries( + report.outcomes.map((outcome) => [outcome.sourceId, outcome.status]), + ); + expect(status).toEqual({ + [DENSE_CUT]: "failed", + [NATIONAL_CUT]: "published", + [STRAY_CUT]: "ineligible", + [NATIONAL_LINE]: "complete", + [JUNE_RELEASE]: "complete", + }); + expect(report.outcomes.find((outcome) => outcome.sourceId === DENSE_CUT)).toEqual({ + sourceId: DENSE_CUT, + status: "failed", + buildArtifactId: null, + reason: REFUSED_SCHEMA_8_ERROR, + }); + expect(report.latestReleaseFailure).toBeUndefined(); + + // Nothing is uploaded or written for the failed tagged build. The national + // tagged build is published and becomes the latest release, as + // latest-national.json names it. + expect(store.uploads.map((bundle) => bundle.index.build.releaseId)).toEqual([NATIONAL_CUT]); + expect(store.writes.map((write) => [write.entry.releaseId, write.makeLatest ?? false])) + .toEqual([[NATIONAL_CUT, false], [NATIONAL_CUT, true]]); + const builds = store.current().countries.uk!.builds; + expect(builds.map((build) => build.releaseId).sort()) + .toEqual([JUNE_RELEASE, NATIONAL_LINE, NATIONAL_CUT].sort()); + expect(store.current().countries.uk!.latestReleaseBuildArtifactId) + .toBe(builds.find((build) => build.releaseId === NATIONAL_CUT)!.buildArtifactId); + } finally { + globalThis.fetch = originalFetch; + } +}); + +test("a release whose uploaded build fails its audit gets no manifest entry, and the releases after it are still published", async () => { + const store = lineStoreWithJuneLatest({ incompleteAfterUpload: new Set([DENSE_CUT]) }); + globalThis.fetch = lineRepositoryFetch([]); + try { + const report = await reconcileReleaseBuilds("uk", "blob-token", false, store); + const dense = report.outcomes.find((outcome) => outcome.sourceId === DENSE_CUT); + expect(dense?.status).toBe("failed"); + expect(dense?.buildArtifactId).toBeNull(); + expect(dense?.reason).toMatch( + new RegExp(`^Release build ${DENSE_CUT} remains incomplete: missing calibration-trees/uk/[0-9a-f]{64}/tier-1\\.json\\.gz$`), + ); + expect(report.outcomes.find((outcome) => outcome.sourceId === NATIONAL_CUT)?.status) + .toBe("published"); + + // Each uploaded build is audited before the manifest names it, so the + // dense tagged build's files are never registered. + expect(store.operations).toEqual([ + "list files", + `upload ${DENSE_CUT}`, + "list files", + `audit ${DENSE_CUT}`, + `audit ${NATIONAL_LINE}`, + `upload ${NATIONAL_CUT}`, + "list files", + `audit ${NATIONAL_CUT}`, + `write ${NATIONAL_CUT}`, + `audit ${JUNE_RELEASE}`, + `write ${NATIONAL_CUT} as latest`, + ]); + expect(store.current().countries.uk!.builds.some((build) => build.releaseId === DENSE_CUT)) + .toBe(false); + } finally { + globalThis.fetch = originalFetch; + } +}); + +test("when the release the pointer file names fails, the latest-release selection is left as it was", async () => { + const store = lineStoreWithJuneLatest(); + globalThis.fetch = lineRepositoryFetch([], (directory) => + directory === NATIONAL_LINE ? REFUSED_SCHEMA_8_DIAGNOSTICS : SCHEMA_8_DIAGNOSTICS + ); + try { + const report = await reconcileReleaseBuilds("uk", "blob-token", false, store); + expect(report.outcomes.find((outcome) => outcome.sourceId === NATIONAL_CUT)).toEqual({ + sourceId: NATIONAL_CUT, + status: "failed", + buildArtifactId: null, + reason: REFUSED_SCHEMA_8_ERROR, + }); + expect(report.outcomes.find((outcome) => outcome.sourceId === DENSE_CUT)?.status) + .toBe("published"); + expect(report.latestReleaseFailure).toBe( + `Latest release ${NATIONAL_CUT} failed to reconcile, so the latest-release selection was not changed.`, + ); + + // Only the dense tagged build is written; the June release stays the latest. + expect(store.writes.map((write) => [write.entry.releaseId, write.makeLatest ?? false])) + .toEqual([[DENSE_CUT, false]]); + expect(store.current().countries.uk!.latestReleaseBuildArtifactId) + .toBe(juneEntry().buildArtifactId); + } finally { + globalThis.fetch = originalFetch; + } +}); + +test("the publisher prints every release outcome, then exits with an error naming each failed release", async () => { + // Two failures: the dense tagged build's file is refused, and the June release's + // stored build pins a commit its tag no longer points at. + const movedJune = { ...juneEntry(), hfCommitSha: "6".repeat(40) }; + const store = memoryStore( + withCalibrationTreeManifestEntry( + withCalibrationTreeManifestEntry(emptyCalibrationTreeManifest(), "uk", placeholderEntry()), + "uk", + movedJune, + true, + ), + ); + globalThis.fetch = lineRepositoryFetch([], (directory) => + directory === DENSE_LINE ? REFUSED_SCHEMA_8_DIAGNOSTICS : SCHEMA_8_DIAGNOSTICS + ); + const originalBlobToken = process.env.BLOB_READ_WRITE_TOKEN; + process.env.BLOB_READ_WRITE_TOKEN = "unused-blob-token"; + const printed: string[] = []; + const originalLog = console.log; + console.log = (message?: unknown) => { + printed.push(String(message)); + }; + try { + await expect( + runPublisher({ country: "uk", mode: "reconcile-releases" }, { store }), + ).rejects.toThrow( + `2 of 5 releases failed to reconcile: ${DENSE_CUT}, ${JUNE_RELEASE}. ` + + "The printed report gives each reason.", + ); + expect(printed).toHaveLength(1); + const report = JSON.parse(printed[0]) as { outcomes: Array<{ sourceId: string; status: string; reason: string | null }> }; + expect(report.outcomes.map((outcome) => [outcome.sourceId, outcome.status])).toEqual([ + [STRAY_CUT, "ineligible"], + [DENSE_CUT, "failed"], + [NATIONAL_LINE, "complete"], + [NATIONAL_CUT, "published"], + [JUNE_RELEASE, "failed"], + ]); + expect(report.outcomes.find((outcome) => outcome.sourceId === JUNE_RELEASE)?.reason).toBe( + `Released dataset ${JUNE_RELEASE} changed from ${"6".repeat(40)} to ${JUNE_SHA}.`, + ); + // The national tagged build still becomes the latest release. + expect(store.writes.map((write) => [write.entry.releaseId, write.makeLatest ?? false])) + .toEqual([[NATIONAL_CUT, false], [NATIONAL_CUT, true]]); + } finally { + console.log = originalLog; + globalThis.fetch = originalFetch; + if (originalBlobToken == null) delete process.env.BLOB_READ_WRITE_TOKEN; + else process.env.BLOB_READ_WRITE_TOKEN = originalBlobToken; + } +}); + +test("a dry run records a failed release, writes nothing and still reports the releases after it", async () => { + const store = lineStoreWithJuneLatest(); + globalThis.fetch = lineRepositoryFetch([], (directory) => + directory === DENSE_LINE ? REFUSED_SCHEMA_8_DIAGNOSTICS : SCHEMA_8_DIAGNOSTICS + ); + try { + const report = await reconcileReleaseBuilds("uk", "blob-token", true, store); + expect(report.dryRun).toBe(true); + expect(report.outcomes.find((outcome) => outcome.sourceId === DENSE_CUT)?.status) + .toBe("failed"); + expect(report.outcomes.find((outcome) => outcome.sourceId === NATIONAL_CUT)).toMatchObject({ + status: "published", + reason: "Dry run; no Blob writes performed.", + }); + expect(store.uploads).toHaveLength(0); + expect(store.writes).toHaveLength(0); + } finally { + globalThis.fetch = originalFetch; + } +}); + +test("a dry run reports the same latest-release failure as a run that writes when the release the pointer file names fails", async () => { + const refuseNationalDiagnostics = (directory: string) => + directory === NATIONAL_LINE ? REFUSED_SCHEMA_8_DIAGNOSTICS : SCHEMA_8_DIAGNOSTICS; + const expectedFailure = + `Latest release ${NATIONAL_CUT} failed to reconcile, so the latest-release selection was not changed.`; + try { + globalThis.fetch = lineRepositoryFetch([], refuseNationalDiagnostics); + const writingReport = await reconcileReleaseBuilds("uk", "blob-token", false, lineStoreWithJuneLatest()); + expect(writingReport.latestReleaseFailure).toBe(expectedFailure); + + const dryRunStore = lineStoreWithJuneLatest(); + globalThis.fetch = lineRepositoryFetch([], refuseNationalDiagnostics); + const dryRunReport = await reconcileReleaseBuilds("uk", "blob-token", true, dryRunStore); + expect(dryRunReport.outcomes.find((outcome) => outcome.sourceId === NATIONAL_CUT)?.status) + .toBe("failed"); + expect(dryRunReport.latestReleaseFailure).toBe(expectedFailure); + expect(reconciliationFailureMessage(dryRunReport)).toBe(reconciliationFailureMessage(writingReport)); + expect(dryRunStore.uploads).toHaveLength(0); + expect(dryRunStore.writes).toHaveLength(0); + } finally { + globalThis.fetch = originalFetch; + } +}); + test("the UK latest release is the cut its line pointer names", async () => { const requested: string[] = []; globalThis.fetch = lineRepositoryFetch(requested); diff --git a/frontend/packages/calibration-tree/src/publish-calibration-tree.ts b/frontend/packages/calibration-tree/src/publish-calibration-tree.ts index 6170f7bf..b101ca2e 100644 --- a/frontend/packages/calibration-tree/src/publish-calibration-tree.ts +++ b/frontend/packages/calibration-tree/src/publish-calibration-tree.ts @@ -88,7 +88,12 @@ export class CalibrationTreeArtifactSizeError extends Error {} export interface ReconciliationOutcome { sourceId: string; - // `failed`: staging only; `reason` gives the error. + // `failed`: the source could not be published or repaired; `reason` gives + // the error. A failed release gets no new manifest entry unless the + // manifest write was stored and only its read-back check failed; an entry + // it already had is left as it was, and files uploaded before the error + // stay under its build ID. A failed staging run can already have its + // entry, because staging writes the entry before it audits the upload. status: "complete" | "published" | "repaired" | "ineligible" | "failed"; buildArtifactId: string | null; reason: string | null; @@ -99,6 +104,13 @@ export interface ReconciliationReport { sourceKind: "releases" | "staging"; dryRun: boolean; outcomes: ReconciliationOutcome[]; + /** + * Release reconciliation only: why the latest-release selection could not + * be read or updated. Absent when it was updated. In a dry run, present + * only when the upstream pointer cannot be read or names a release that + * failed in this run. + */ + latestReleaseFailure?: string; } function sha256Text(value: string): string { @@ -1175,84 +1187,149 @@ export async function reconcileReleaseBuilds( "release", candidate.releaseId, ); - if (existing && existing.hfCommitSha !== candidate.hfCommitSha) { - throw new Error( - `Released dataset ${candidate.releaseId} changed from ${existing.hfCommitSha} ` + - `to ${candidate.hfCommitSha}.`, - ); - } - if (existing && releaseDirectoryOf(existing) !== candidate.releaseDirectory) { - throw new Error( - `Released dataset ${candidate.releaseId} moved from releases/${releaseDirectoryOf(existing)}/ ` + - `to releases/${candidate.releaseDirectory}/.`, - ); - } - const audit = existing ? await store.audit(existing, blobs) : null; - if (existing && audit?.complete) { - // Reconciliation validates and restores stored parts only; a stored - // entry's metadata is the timestamp migration's business. - outcomes.push({ - sourceId: candidate.releaseId, - status: "complete", - buildArtifactId: existing.buildArtifactId, - reason: null, - }); - continue; - } - if (audit && !audit.repairable) { - throw new Error( - `Release build ${candidate.releaseId} is corrupt and cannot be repaired automatically: ` + - audit.reasons.join("; "), - ); - } - const published = await publishRelease({ - country, - id: candidate.releaseId, - releaseDirectory: candidate.releaseDirectory, - expectedSha: existing?.hfCommitSha ?? candidate.hfCommitSha, - blobToken, - useExpectedShaExactly: true, - buildArtifactId: existing?.buildArtifactId, - dryRun, - expectedIndexSha256: existing?.indexSha256, - indexCreatedAt: existing ? existing.createdAt : undefined, - uploadBundle: store.uploadBundle, - }); - const entry = existing ?? releaseManifestEntry(published); - if (!dryRun) { - manifest = await store.updateManifest(entry); - blobs = await store.listBlobs(); - const verified = await store.audit(entry, blobs); - if (!verified.complete) { + // A release whose build cannot be published or repaired is reported with + // the reason, and the releases after it are still reconciled: stopping + // here would leave every later release, and the latest-release + // selection, unreconciled on every later invocation. `runPublisher` + // exits with an error once the report is printed. The manifest write is + // a release's last step, after its files are uploaded and audited, so a + // release that fails gets no new manifest entry (unless that write's own + // read-back check fails after the write was stored); a release that + // already had an entry keeps it unchanged. The latest-release selection + // is never moved to a failed release. + try { + if (existing && existing.hfCommitSha !== candidate.hfCommitSha) { + throw new Error( + `Released dataset ${candidate.releaseId} changed from ${existing.hfCommitSha} ` + + `to ${candidate.hfCommitSha}.`, + ); + } + if (existing && releaseDirectoryOf(existing) !== candidate.releaseDirectory) { throw new Error( - `Release build ${candidate.releaseId} remains incomplete: ${verified.reasons.join("; ")}`, + `Released dataset ${candidate.releaseId} moved from releases/${releaseDirectoryOf(existing)}/ ` + + `to releases/${candidate.releaseDirectory}/.`, ); } + const audit = existing ? await store.audit(existing, blobs) : null; + if (existing && audit?.complete) { + // Reconciliation validates and restores stored parts only; a stored + // entry's metadata is the timestamp migration's business. + outcomes.push({ + sourceId: candidate.releaseId, + status: "complete", + buildArtifactId: existing.buildArtifactId, + reason: null, + }); + continue; + } + if (audit && !audit.repairable) { + throw new Error( + `Release build ${candidate.releaseId} is corrupt and cannot be repaired automatically: ` + + audit.reasons.join("; "), + ); + } + const published = await publishRelease({ + country, + id: candidate.releaseId, + releaseDirectory: candidate.releaseDirectory, + expectedSha: existing?.hfCommitSha ?? candidate.hfCommitSha, + blobToken, + useExpectedShaExactly: true, + buildArtifactId: existing?.buildArtifactId, + dryRun, + expectedIndexSha256: existing?.indexSha256, + indexCreatedAt: existing ? existing.createdAt : undefined, + uploadBundle: store.uploadBundle, + }); + const entry = existing ?? releaseManifestEntry(published); + if (!dryRun) { + // The uploaded files are audited before the manifest names them, so + // an incomplete upload never becomes a manifest entry. The audit + // reads the build's own index and the Blob listing, not the + // manifest. + blobs = await store.listBlobs(); + const verified = await store.audit(entry, blobs); + if (!verified.complete) { + throw new Error( + `Release build ${candidate.releaseId} remains incomplete: ${verified.reasons.join("; ")}`, + ); + } + manifest = await store.updateManifest(entry); + } + outcomes.push({ + sourceId: candidate.releaseId, + status: existing ? "repaired" : "published", + buildArtifactId: entry.buildArtifactId, + reason: dryRun ? "Dry run; no Blob writes performed." : null, + }); + } catch (error) { + outcomes.push({ + sourceId: candidate.releaseId, + status: "failed", + buildArtifactId: existing?.buildArtifactId ?? null, + reason: error instanceof Error ? error.message : String(error), + }); } - outcomes.push({ - sourceId: candidate.releaseId, - status: existing ? "repaired" : "published", - buildArtifactId: entry.buildArtifactId, - reason: dryRun ? "Dry run; no Blob writes performed." : null, - }); } - const latest = await readUpstreamLatest(country, undefined, manifest); - const latestEntry = manifestBuildForSource( - manifest, + const failedReleaseIds = new Set( + outcomes.filter((outcome) => outcome.status === "failed").map((outcome) => outcome.sourceId), + ); + const latestReleaseFailure = await selectLatestRelease( country, - "release", - latest.releaseId, + manifest, + failedReleaseIds, + dryRun, + store, ); - if (!dryRun) { + return { + country, + sourceKind: "releases", + dryRun, + outcomes, + ...(latestReleaseFailure == null ? {} : { latestReleaseFailure }), + }; +} + +/** + * Point the country's latest-release selection at the release its upstream + * pointer names, once that release is completely published. Returns why the + * selection could not be read or updated, or null. A release that failed in + * this reconciliation is never selected, so the selection keeps naming the + * build it named before. A dry run reads the pointer and reports a release + * that failed, but neither checks nor changes the selection otherwise. + */ +async function selectLatestRelease( + country: MicrocosmCountry, + manifest: CalibrationTreeManifest, + failedReleaseIds: ReadonlySet, + dryRun: boolean, + store: CalibrationManifestStore, +): Promise { + try { + const latest = await readUpstreamLatest(country, undefined, manifest); + // Checked before the dry-run return, so a dry run reports a failed + // latest release exactly as a run that writes does. + if (failedReleaseIds.has(latest.releaseId)) { + return `Latest release ${latest.releaseId} failed to reconcile, so the latest-release selection was not changed.`; + } + // A dry run writes no manifest entries, so it cannot check whether the + // release is completely published. + if (dryRun) return null; + const latestEntry = manifestBuildForSource( + manifest, + country, + "release", + latest.releaseId, + ); if (!latestEntry || latestEntry.hfCommitSha !== latest.hfCommitSha) { - throw new Error( - `Latest release ${latest.releaseId}@${latest.hfCommitSha} is not completely published.`, - ); + return `Latest release ${latest.releaseId}@${latest.hfCommitSha} is not completely published.`; } await store.updateManifest(latestEntry, true); + return null; + } catch (error) { + return error instanceof Error ? error.message : String(error); } - return { country, sourceKind: "releases", dryRun, outcomes }; } interface TimestampMigrationOutcome { @@ -1350,15 +1427,25 @@ async function requestedReleaseSelection( }; } -/** The error a report's failed outcomes end the publisher with, if any. */ +/** + * The error a report's failed outcomes, or a latest-release selection that + * could not be updated, end the publisher with, if any. + */ export function reconciliationFailureMessage(report: ReconciliationReport): string | null { const failed = report.outcomes.filter((outcome) => outcome.status === "failed"); - if (failed.length === 0) return null; - return ( - `${failed.length} of ${report.outcomes.length} ${report.sourceKind} sources failed to ` + - `reconcile: ${failed.map((outcome) => outcome.sourceId).join(", ")}. ` + - "The printed report gives each reason." - ); + const messages: string[] = []; + if (failed.length > 0) { + const sources = report.sourceKind === "releases" ? "releases" : "staging sources"; + messages.push( + `${failed.length} of ${report.outcomes.length} ${sources} failed to ` + + `reconcile: ${failed.map((outcome) => outcome.sourceId).join(", ")}.`, + ); + } + if (report.latestReleaseFailure) { + messages.push(`The latest release was not selected: ${report.latestReleaseFailure}`); + } + if (messages.length === 0) return null; + return [...messages, "The printed report gives each reason."].join(" "); } export interface PublisherDependencies {