diff --git a/.agents/skills/hypercerts-org-sdk/SKILL.md b/.agents/skills/hypercerts-org-sdk/SKILL.md index e50a59a..a0624f9 100644 --- a/.agents/skills/hypercerts-org-sdk/SKILL.md +++ b/.agents/skills/hypercerts-org-sdk/SKILL.md @@ -3,7 +3,7 @@ name: hypercerts-org-sdk description: "Write Hypercerts and Certified records (org.hypercerts.*, app.certified.*) to a PDS from an app with @hypercerts-org/sdk: setting up the SDK context from an authenticated @atproto/lex Client, choosing the right operation and subpath, uploading blobs, and handling write errors without duplicates. Use when app code imports @hypercerts-org/sdk, or the user wants to create, link or write hypercert-related records from an app. Not for changing the SDK's own source." metadata: author: hypercerts-org - version: "3" + version: "4" --- # @hypercerts-org/sdk @@ -52,7 +52,7 @@ Make a new context when the signed-in account changes. Records are always writte | `@hypercerts-org/sdk/feature` | `createFeature`, `RecordAlreadyExistsError`, the same `Write*` errors, `FeatureInput`, `CreateFeatureResult`, `LeafletDocument` | | `@hypercerts-org/sdk/organization` | `createOrganization`, `OrganizationAlreadyExistsError`, `RecordAlreadyExistsError`, the same `Write*` errors, `OrganizationInput`, `CreateOrganizationInput`, `CreateOrganizationResult`, `LeafletDocument` | | `@hypercerts-org/sdk/blob` | `uploadBlob` | -| `@hypercerts-org/sdk/collection` | `createActivityInCollection`, `WriteError`, `WriteRejectedError`, `WriteOutcomeUnknownError`, `WriteResultMismatchError`, input/result types | +| `@hypercerts-org/sdk/collection` | `createCollection`, `createActivityInCollection`, `RecordAlreadyExistsError`, `WriteError`, `WriteRejectedError`, `WriteOutcomeUnknownError`, `WriteResultMismatchError`, input/result types | The root entry does not re-export operations. `import { createActivity } from "@hypercerts-org/sdk"` fails. @@ -67,7 +67,7 @@ Operations are grouped by domain, each under its own subpath. Read the domain's | Location (a standalone `app.certified.location`, shared by several activities) | `@hypercerts-org/sdk/location` | [references/location.md](references/location.md) | | Tag (`org.hypercerts.vocab.tag`, `org.hypercerts.workscope.tag`) | `@hypercerts-org/sdk/tag` | [references/tag.md](references/tag.md) | | Feature (`org.hypercerts.entity.feature`, with its tags and locations) | `@hypercerts-org/sdk/feature` | [references/feature.md](references/feature.md) | -| Collection (add a new Activity to an existing Collection) | `@hypercerts-org/sdk/collection` | [references/collection.md](references/collection.md) | +| Collection (create a Collection with items; add a new Activity to an existing Collection) | `@hypercerts-org/sdk/collection` | [references/collection.md](references/collection.md) | | Organization (`app.certified.actor.organization`, with its profile and locations) | `@hypercerts-org/sdk/organization` | [references/organization.md](references/organization.md) | Three things to know before opening a reference: @@ -107,7 +107,7 @@ Uploads happen before the write and are not rolled back, so a call that throws c Order of work: validate input → read `{ uri }` records → upload blobs → validate against lexicons → one write. The SDK never retries. -The three write errors are exported from `@hypercerts-org/sdk/activity`, `/actor`, `/location`, `/tag`, `/feature`, `/collection` and `/organization` and extend `WriteError`. Every `WriteError` has `attempted`: the URI and CID of every record in the request, in request order: for `createActivity` the Activity is last; for `createActivityInCollection` the Collection is last and the Activity is just before it; for `createOrganization` the organization is last. `instanceof WriteError` catches all three. +The three write errors are exported from `@hypercerts-org/sdk/activity`, `/actor`, `/location`, `/tag`, `/feature`, `/collection` and `/organization` and extend `WriteError`. Every `WriteError` has `attempted`: the URI and CID of every record in the request, in request order: for `createActivity` the Activity is last; for `createActivityInCollection` the Collection is last and the Activity is just before it; for `createCollection` the Collection is last, after the records it points at; for `createOrganization` the organization is last. `instanceof WriteError` catches all three. | Error | Meaning | What to do | | -------------------------- | ---------------------------------------------------------------------------------------------- | --------------------------------------------- | @@ -179,7 +179,8 @@ try { - `createSDKContext({ client: session })` or passing an `@atproto/api` `Agent` throws. Wrap it: `new Client(session)`. - Passing an `uploadBlob` result as `image` throws. Pass the `Blob`/`File`. -- Retrying blindly after `WriteOutcomeUnknownError` duplicates every record under new keys. The exception is a call whose records all have a fixed key and are only created if absent: `createProfile` and `createVocabularyTag`. `createOrganization` is also safe to repeat: the organization exists at most once, so a second call creates it or throws `OrganizationAlreadyExistsError`. `createFeature` and `createWorkScopeTag` are not safe to repeat. +- Retrying blindly after `WriteOutcomeUnknownError` duplicates every record under new keys. The exception is a call whose records all have a fixed key and are only created if absent: `createProfile` and `createVocabularyTag`. `createOrganization` is also safe to repeat: the organization exists at most once, so a second call creates it or throws `OrganizationAlreadyExistsError`. `createFeature`, `createCollection` and `createWorkScopeTag` are not safe to repeat. +- An item of `createCollection` is `{ activity }`, `{ feature }`, `{ collection }` or `{ uri }`, never a bare Activity input; `{ uri }` must be an Activity, Feature or Collection. - A vocabulary tag's key is `.` exactly as you pass it. A dot in either part throws a `TypeError`; the SDK does not normalise the parts. - A work-scope tag URI is not a vocabulary tag URI: each field accepts one collection, and a `{ uri }` of another throws a `TypeError`. - One write holds at most 200 records on the reference PDS. A larger call throws a `TypeError` before anything is written; the SDK does not split it. Create some records first and pass their URIs. diff --git a/.agents/skills/hypercerts-org-sdk/references/collection.md b/.agents/skills/hypercerts-org-sdk/references/collection.md index a415008..6640aac 100644 --- a/.agents/skills/hypercerts-org-sdk/references/collection.md +++ b/.agents/skills/hypercerts-org-sdk/references/collection.md @@ -1,6 +1,67 @@ # Collection domain (`@hypercerts-org/sdk/collection`) -Read [../SKILL.md](../SKILL.md) first for setup and error handling, and [activity.md](activity.md) for the `activity` input. Every example assumes `context` from `createSDKContext`. +Read [../SKILL.md](../SKILL.md) first for setup and error handling, [activity.md](activity.md) for an `activity` item, [feature.md](feature.md) for a `feature` item and [tag.md](tag.md) / [location.md](location.md) for tags and locations. Every example assumes `context` from `createSDKContext`. + +Two operations: `createCollection` creates a **new** Collection with its items; `createActivityInCollection` adds a new Activity to a Collection that **already exists**. + +## createCollection + +`createCollection(context, input)` creates an `org.hypercerts.collection` (TID key) and every new record in it, in one `applyWrites` commit: the Collection, new Activities (with their rights, locations, contributors, ...), Features, child Collections, locations and tags. All of it exists afterwards, or none does. + +```ts +import { createCollection } from "@hypercerts-org/sdk/collection"; + +const collection = await createCollection(context, { + type: "portfolio", + title: "Phobjikha valley restoration", + description: "Everything we do in the valley.", // string | Leaflet document | { uri } of any record + avatar: "https://example.org/valley.png", // URL string, or an image Blob/File to upload + location: { uri: "at://did:plc:abc123/app.certified.location/3k2a4b5c6d7e8" }, // reused + tags: [{ category: "theme", key: "forest", name: "Forest", status: "accepted" }], // created + items: [ + { activity: { title: "Reforestation", shortDescription: "Planting 5,000 trees." }, weight: "2" }, // created + { feature: { type: "zone", title: "Valley floor zone" } }, // created + { collection: { title: "Phase 2", items: [] } }, // created, can hold items too + { uri: "at://did:plc:abc123/org.hypercerts.claim.activity/3k2a4b5c6d7e8", weight: "0.5" }, // reused + ], +}); +collection.items[0]; // { uri, cid, type: "activity", created: true, locations: [], contributors: [] } +collection.items[3]; // { uri, cid, type: "activity", created: false } +``` + +- An item is **exactly one** of `{ activity }`, `{ feature }`, `{ collection }` (the data of a new record, same input as `createActivity`, `createFeature` and `createCollection`) or `{ uri }`, plus an optional `weight`. The SDK never guesses from the fields: a bare Activity (`{ title, shortDescription }`) is a `TypeError`. +- `{ uri }` must be an Activity, Feature or Collection with a DID (`at:////`). It is read for its CID and reused unchanged; it may belong to another account if public. Another record type, or a handle, is a `TypeError` naming the item. +- `weight` is a positive decimal string (`"2"`, `"0.50"`), stored exactly as you pass it. Weights need not add up to anything. +- Order is kept. Nothing is deduplicated or searched for: the same `{ uri }` twice gives two items, the same data twice gives two records. +- `location` and `tags` take new records or `{ uri }` like in `createFeature`. A new tag's key is `.`: a taken key throws `RecordAlreadyExistsError` before anything is uploaded or written, and two new tags with the same key anywhere in the call are a `TypeError` (create it first and pass its `uri`). +- `description`: a string and a Leaflet document (it needs `$type: "pub.leaflet.pages.linearDocument"`) stay inline in the Collection; `{ uri }` becomes a strong reference. The SDK never creates a document record. +- `avatar` (image up to 5 MB) and `banner` (image up to 10 MB): a URL string is stored as it is, a `Blob`/`File` is uploaded. +- The result mirrors the input: `result.items[i]`, `result.tags[i]` and `result.location` say what was created (`created: true`) or reused (`created: false`). Narrow an item on `created` and `type` to reach the nested results (`item.locations`, `item.items`, ...). +- One write holds at most 200 records, the Collection and every nested Activity, location and tag included. A bigger call throws a `TypeError` before anything is read or uploaded; it is never split. Create some records first and pass their `uri`. +- The session needs `create` on every collection the call writes to. + +### Order of work and failures + +Validate all input and count the records, read every `{ uri }` and every new tag's key, upload files, build the records, one write. A bad item, a wrong record type, too many records or an unreadable `{ uri }` fails (`TypeError` or `Error`) before any upload. A file the image limits reject, or a record the lexicon rejects (a title that is too long), is only found when that record is built, so files of other records may already be uploaded. Uploads are not rolled back if the write fails; see [Uploads are not part of the atomic write](../SKILL.md#uploads-are-not-part-of-the-atomic-write). + +Errors are as in [Handle failures](../SKILL.md#6-handle-failures). After `WriteOutcomeUnknownError`, `attempted` lists every new record, children before the records that point at them, the Collection **last**. The write is all-or-nothing, so check the Collection with `getRecord`; if it exists, everything does. **Do not retry blindly**: it would create every record again under new keys. + +### `CollectionInput` + +| Field | Type | Notes | +| ------------------------ | ---------------------------------------- | ------------------------------------------------------------------------------- | +| `title` | `string` | Required. At most 800 characters and 80 graphemes. | +| `type` | `string` | `favorites`, `project`, `portfolio`, `program`; other values accepted. | +| `shortDescription` | `string` | At most 300 graphemes. `shortDescriptionFacets` holds its rich-text facets. | +| `description` | `string \| LeafletDocument \| { uri }` | Inline text or document, or a reference to a record of any type. | +| `langs` | `string[]` | Up to three BCP-47 tags. | +| `avatar` / `banner` | `string \| Blob` | URL stored as is, or an image uploaded. | +| `location` | `LocationInput \| { uri }` | New or existing `app.certified.location`. | +| `tags` | `(VocabularyTagInput \| { uri })[]` | At most 20; vocabulary tags only (a work-scope tag URI throws). | +| `items` | `CollectionItemInput[]` | At most 1000. `{ activity \| feature \| collection \| uri, weight? }`. | +| `createdAt` | `string` | ISO 8601; defaults to the time of the call, for every new record. | + +Not supported: `signatures`, adding items to a Collection that exists (see below), changing an existing record. ## createActivityInCollection @@ -25,19 +86,19 @@ const { activity, collection } = await createActivityInCollection(context, { - The item is appended last. Everything else in the Collection (`createdAt`, existing items, unknown fields, `signatures`) is kept as it was. `signatures` is not re-signed: after the append, an earlier signature no longer matches the content. - A Collection that holds 1000 items is full: `Error`, nothing written. A Collection that does not exist or is not a valid `org.hypercerts.collection` record: `Error`, nothing written. - Nothing is deduplicated: a second call creates a second Activity and a second item. -- Not supported: creating or editing Collections, adding an existing Activity, adding to another account's Collection, nested Collections. +- Not supported: adding an existing Activity, a Feature or a child Collection to an existing Collection, and adding to another account's Collection. To create a whole new Collection, use `createCollection`. -## Concurrent writes (`InvalidSwap`) +### Concurrent writes (`InvalidSwap`) The write is conditional on the account not having changed since the Collection was read, so a concurrent change to its `items` cannot be overwritten with a stale list. When it did change you get `WriteRejectedError` with `code === "InvalidSwap"`: see [Concurrent changes](../SKILL.md#concurrent-changes-invalidswap) for the rule and the retry example. Any other `WriteRejectedError` (4xx) is not fixed by calling again. After `WriteOutcomeUnknownError`, `attempted` lists the new records and, last, the Collection: check the Activity and whether the Collection lists it before retrying, or you create a second Activity. -## Uploads are outside the commit +### Uploads are outside the commit `Blob`/`File` values are uploaded before the repository is read and are not rolled back if the write fails; see [Uploads are not part of the atomic write](../SKILL.md#uploads-are-not-part-of-the-atomic-write). A bad URI, another account's Collection, a missing, invalid or full Collection, or an invalid `activity` fails before any upload (the Collection is read once up front, then again after the latest commit for the guarded write). -## `CreateActivityInCollectionInput` +### `CreateActivityInCollectionInput` | Field | Type | Notes | | --------------- | --------------- | ------------------------------------------------------------------------------------------------------- | diff --git a/.changeset/create-collection.md b/.changeset/create-collection.md new file mode 100644 index 0000000..c2e7676 --- /dev/null +++ b/.changeset/create-collection.md @@ -0,0 +1,7 @@ +--- +"@hypercerts-org/sdk": minor +--- + +- Add `createCollection` at `@hypercerts-org/sdk/collection`: creates a Collection with its items, location and tags in one `applyWrites` commit. +- An item is `{ activity }`, `{ feature }` or `{ collection }` (new, nested to any depth) or `{ uri }` (an existing Activity, Feature or Collection, reused unchanged), with an optional `weight` kept as written. +- Failed writes throw `WriteRejectedError` (nothing created) or `WriteOutcomeUnknownError` (lists every attempted record, the Collection last); the SDK never retries. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fe4605b..a540dc4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -61,30 +61,12 @@ jobs: # Exact and renamed copies, near-misses with up to 3 differing lines, and functions with the # same structure. Test files are skipped: their setup repeats on purpose. - run: pnpm check:duplicates --reporters console,sarif --output jscpd-report - # The semantic check embeds functions with a 548 MB local model; cache it so only the first - # run downloads it. The key changes when the jscpd version (and so its default model) does. - - if: ${{ !cancelled() }} - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: ~/.cache/jscpd/models - key: jscpd-models-${{ hashFiles('pnpm-lock.yaml') }} - restore-keys: jscpd-models- - # Functions that do the same thing written differently, which the check above misses. - # Report-only for now: it flags too many false matches to block merges. - - if: ${{ !cancelled() }} - continue-on-error: true - run: pnpm check:duplicates:semantic --reporters console,sarif --output jscpd-semantic-report # Upload even when clones fail the step above, which is when the annotations are useful. - if: ${{ !cancelled() && hashFiles('jscpd-report/jscpd-report.sarif') != '' }} uses: github/codeql-action/upload-sarif@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1 with: sarif_file: jscpd-report/jscpd-report.sarif category: jscpd - - if: ${{ !cancelled() && hashFiles('jscpd-semantic-report/jscpd-report.sarif') != '' }} - uses: github/codeql-action/upload-sarif@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1 - with: - sarif_file: jscpd-semantic-report/jscpd-report.sarif - category: jscpd-semantic package: name: Package consumer check (Node ${{ matrix.node }}) diff --git a/AGENTS.md b/AGENTS.md index 92c3324..65fa37e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,7 +21,7 @@ Put detailed guides, design notes and decision records in dedicated documentatio - Create a new package only when a domain has its own setup or dependencies (for example the planned `@hypercerts-org/auth`). `@hypercerts-org/sdk` must not depend on auth helpers; it receives an already-authenticated client. - Packages are ESM-only. Every public entry point must be listed in the package's `exports` field. - Keep packages free of import-time side effects so unused operations can be tree-shaken (`sideEffects: false`). -- Helpers shared by several operations usually live in `packages/sdk/src/internal/` and are not exported. Check there, and search the package, before writing a private helper (see below for where shared code goes). `pnpm check:duplicates` (also run in CI) reports copied code, including renamed copies and functions with the same structure, and `pnpm check:duplicates:semantic` (report-only in CI because it has false matches; downloads a 548 MB model once) reports functions that do the same thing written differently. When `check:duplicates` finds a clone, share the code rather than loosening the check; review each semantic match and share the code when it is a real duplicate. +- Helpers shared by several operations usually live in `packages/sdk/src/internal/` and are not exported. Check there, and search the package, before writing a private helper (see below for where shared code goes). `pnpm check:duplicates` (also run in CI) reports copied code, including renamed copies and functions with the same structure. When `check:duplicates` finds a clone, share the code rather than loosening the check. - Put code where a reader would look for it. Code shared by several domains usually belongs in `internal/`, but use judgement: if it is really one domain's logic that another domain builds on (for example `createActivityInCollection` reusing the activity domain's `prepareActivity`), keep it in its domain and import it from there. Move code when its home stops making sense, not by rule. Avoid circular imports between domains. When `internal/` grows too large to scan, group it into subfolders by concern (for example `internal/records/`, `internal/writes/`) rather than by domain. ## Tooling @@ -80,6 +80,4 @@ pnpm check:package pnpm check:duplicates ``` -Also run `pnpm check:duplicates:semantic` and review what it reports. It does not block CI. - Add a changeset (`pnpm changeset`) for any change that affects a published package. Do not document SDK operations as available until they are implemented. diff --git a/README.md b/README.md index 7865b6a..e963fc0 100644 --- a/README.md +++ b/README.md @@ -34,19 +34,18 @@ pnpm install pnpm build ``` -| Command | What it does | -| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -| `pnpm build` | Build all packages | -| `pnpm typecheck` | Type-check all packages | -| `pnpm test` | Run the tests with Vitest | -| `pnpm test:e2e` | Run the end-to-end tests against a PDS in a local sandbox (needs setup, see [End-to-end tests](./docs/e2e-testing.md)) | -| `pnpm lint` | Lint with Biome | -| `pnpm format` | Format files with Biome | -| `pnpm format:check` | Check formatting without changing files | -| `pnpm check:package` | Build, pack and install each package in an isolated project, then check it in Node, TypeScript and a Vite browser build | -| `pnpm check:duplicates` | Check for copied code with jscpd | -| `pnpm check:duplicates:semantic` | Check for functions that do the same thing written differently (jscpd; downloads a 548 MB model once) | -| `pnpm changeset` | Record a change for the next release | +| Command | What it does | +| ----------------------- | ----------------------------------------------------------------------------------------------------------------------- | +| `pnpm build` | Build all packages | +| `pnpm typecheck` | Type-check all packages | +| `pnpm test` | Run the tests with Vitest | +| `pnpm test:e2e` | Run the end-to-end tests against a PDS in a local sandbox (needs setup, see [End-to-end tests](./docs/e2e-testing.md)) | +| `pnpm lint` | Lint with Biome | +| `pnpm format` | Format files with Biome | +| `pnpm format:check` | Check formatting without changing files | +| `pnpm check:package` | Build, pack and install each package in an isolated project, then check it in Node, TypeScript and a Vite browser build | +| `pnpm check:duplicates` | Check for copied code with jscpd | +| `pnpm changeset` | Record a change for the next release | ## Releasing diff --git a/package.json b/package.json index ee564f4..d2e8988 100644 --- a/package.json +++ b/package.json @@ -16,8 +16,7 @@ "format": "biome format --write .", "format:check": "biome format .", "check:package": "pnpm build && node scripts/check-package.mjs", - "check:duplicates": "jscpd --similarity --ignore-identifiers --ignore-literals --max-gap-lines 3 --ignore \"**/*.test.ts,**/test-utils/**\" --pattern \"{packages/sdk/src,scripts}/**\" --exit-code 1 .", - "check:duplicates:semantic": "jscpd --semantic --semantic-download --kind semantic --semantic-same-threshold 0.55 --ignore \"**/*.test.ts,**/test-utils/**\" --pattern \"{packages/sdk/src,scripts}/**\" --exit-code 1 .", + "check:duplicates": "jscpd --similarity --ignore-identifiers --ignore-literals --max-gap-lines 3 --ignore \"**/*.test.ts,**/test-utils/**,packages/sdk/src/index.ts,packages/sdk/src/*/index.ts\" --pattern \"{packages/sdk/src,scripts}/**\" --exit-code 1 .", "changeset": "changeset" }, "devDependencies": { diff --git a/packages/sdk/e2e/collection.e2e.test.ts b/packages/sdk/e2e/collection.e2e.test.ts index e494079..33a4e71 100644 --- a/packages/sdk/e2e/collection.e2e.test.ts +++ b/packages/sdk/e2e/collection.e2e.test.ts @@ -1,11 +1,21 @@ import { toDatetimeString } from "@atproto/lex"; import * as collection from "@hypercerts-org/lexicon/org/hypercerts/collection"; -import { beforeAll, describe, expect, it } from "vitest"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { createActivity, createRights } from "../src/activity/index.js"; -import { createActivityInCollection } from "../src/collection/index.js"; +import { createActivityInCollection, createCollection } from "../src/collection/index.js"; +import { createFeature } from "../src/feature/index.js"; +import { readE2eEnv } from "./env.js"; import { coordinates, rights, unique } from "./fixtures.js"; -import { expectReferencesResolve, expectStored, type Ref, snapshotCounts } from "./records.js"; -import { type Account, loginAs } from "./session.js"; +import { redirectPlcDirectory } from "./plc.js"; +import { + collections, + defined, + expectReferencesResolve, + expectStored, + type Ref, + snapshotCounts, +} from "./records.js"; +import { type Account, loginAs, type RewriteRequest } from "./session.js"; let user: Account; beforeAll(async () => { @@ -92,3 +102,206 @@ describe("createActivityInCollection", () => { await expectStored(user.client, seeded.ref); }); }); + +// A vocabulary tag's key is chosen by the caller and cannot be reused, so every run uses a new one. +const vocabularyTag = () => ({ + category: "theme", + key: unique("forest", "-"), + name: unique("Forest"), + status: "proposed", +}); + +// The items a stored Collection lists, as the strong references and weights it holds. +const itemsOf = (stored: Record) => + stored.items as { itemIdentifier: Ref; itemWeight?: string }[]; + +describe("createCollection with new records", () => { + it("stores the Collection, its nested records and a child Collection in one write, and every reference resolves", async () => { + const existing = await createActivity(user.context, base()); + const before = await snapshotCounts(user.client); + + const result = await createCollection(user.context, { + type: "portfolio", + title: unique("E2E collection"), + description: "Everything we do in the valley.", + location: coordinates, + tags: [vocabularyTag()], + items: [ + { + activity: { + ...base(), + locations: [coordinates], + // Inline: stored in the Activity, no record of its own. + contributors: [{ identity: { inline: "did:plc:bob456" }, details: { inline: "Lead" } }], + }, + weight: "2", + }, + { feature: { type: "zone", title: unique("E2E feature") } }, + { + collection: { + title: unique("E2E child collection"), + items: [{ uri: existing.uri, weight: "0.5" }], + }, + }, + ], + }); + + // The Collection and every record it points at exist with the CIDs the SDK reported. + const stored = await expectStored(user.client, result); + await expectReferencesResolve(user.client, stored); + const [activityItem, featureItem, childItem] = result.items; + expect(result.items.map(({ type, created }) => [type, created])).toEqual([ + ["activity", true], + ["feature", true], + ["collection", true], + ]); + expect(itemsOf(stored)).toEqual([ + { itemIdentifier: refOnly(defined(activityItem)), itemWeight: "2" }, + { itemIdentifier: refOnly(defined(featureItem)) }, + { itemIdentifier: refOnly(defined(childItem)) }, + ]); + + // The new Activity holds its new location by reference and its contributor inline. + const activity = await expectStored(user.client, defined(activityItem)); + expect(await expectReferencesResolve(user.client, activity)).toHaveLength(1); + expect(activity.contributors).toEqual([ + { + contributorIdentity: { + $type: "org.hypercerts.claim.activity#contributorIdentity", + identity: "did:plc:bob456", + }, + contributionDetails: { + $type: "org.hypercerts.claim.activity#contributorRole", + role: "Lead", + }, + }, + ]); + + // The child Collection lists the existing Activity at the version that was read. + const child = await expectStored(user.client, defined(childItem)); + expect(itemsOf(child)).toEqual([{ itemIdentifier: refOnly(existing), itemWeight: "0.5" }]); + + // Two Collections, one Activity, one Feature, two locations and one tag; nothing else. + expect(await snapshotCounts(user.client)).toEqual({ + ...before, + collection: before.collection + 2, + activity: before.activity + 1, + feature: before.feature + 1, + location: before.location + 2, + vocabularyTag: before.vocabularyTag + 1, + }); + }); + + it("mixes new and reused records: a new Activity at an existing location, and an existing Feature", async () => { + const seed = await createActivity(user.context, { ...base(), locations: [coordinates] }); + const location = refOnly(defined(seed.locations[0])); + const feature = await createFeature(user.context, { title: unique("E2E feature") }); + const before = await snapshotCounts(user.client); + + const result = await createCollection(user.context, { + title: unique("E2E collection"), + location: { uri: location.uri }, + items: [ + { activity: { ...base(), locations: [{ uri: location.uri }] } }, + { uri: feature.uri }, + ], + }); + + expect(result.location).toEqual({ ...location, created: false }); + const [activityItem, featureItem] = result.items; + expect(featureItem).toEqual({ ...refOnly(feature), type: "feature", created: false }); + expect(activityItem).toMatchObject({ type: "activity", created: true }); + // The new Activity points at the reused location, not a copy of it. + const activity = await expectStored(user.client, defined(activityItem)); + expect(await expectReferencesResolve(user.client, activity)).toEqual([location]); + const stored = await expectStored(user.client, result); + expect(itemsOf(stored)).toEqual([ + { itemIdentifier: refOnly(defined(activityItem)) }, + { itemIdentifier: refOnly(feature) }, + ]); + // Only the Collection and the Activity are new. + expect(await snapshotCounts(user.client)).toEqual({ + ...before, + collection: before.collection + 1, + activity: before.activity + 1, + }); + }); +}); + +describe("createCollection reusing another account's records", () => { + beforeAll(() => redirectPlcDirectory(readE2eEnv().plcUrl)); + afterAll(() => vi.unstubAllGlobals()); + + it("lists them at the version it read without writing to either account's copies", async () => { + const other = await loginAs("user2"); + expect(other.did, "SDK_E2E_USER2 must be a different account than SDK_E2E_USER1").not.toBe( + user.did, + ); + const theirs = await createActivity(other.context, { ...base(), locations: [coordinates] }); + const theirLocation = refOnly(defined(theirs.locations[0])); + const userBefore = await snapshotCounts(user.client); + const otherBefore = await snapshotCounts(other.client); + + const result = await createCollection(user.context, { + title: unique("E2E collection"), + location: { uri: theirLocation.uri }, + items: [{ uri: theirs.uri, weight: "1" }], + }); + + expect(result.location).toEqual({ ...theirLocation, created: false }); + expect(result.items).toEqual([{ ...refOnly(theirs), type: "activity", created: false }]); + const stored = await expectStored(user.client, result); + await expectReferencesResolve(user.client, stored); + expect(itemsOf(stored)).toEqual([{ itemIdentifier: refOnly(theirs), itemWeight: "1" }]); + expect(await snapshotCounts(user.client)).toEqual({ + ...userBefore, + collection: userBefore.collection + 1, + }); + expect(await snapshotCounts(other.client)).toEqual(otherBefore); + }); +}); + +describe("createCollection when the PDS rejects one record of the write", () => { + // Gives the Feature in an applyWrites request the record key of `taken`, a Feature that already + // exists, so the PDS must refuse that one create. Counts the requests it changed. + function collideFeatureWith(taken: Ref): RewriteRequest & { rewritten: number } { + const rkey = defined(taken.uri.split("/").at(-1)); + const rewrite = (path: string, init: RequestInit): RequestInit => { + if (!path.includes("com.atproto.repo.applyWrites") || typeof init.body !== "string") { + return init; + } + const body = JSON.parse(init.body) as { writes: { collection: string; rkey: string }[] }; + const write = defined(body.writes.find((w) => w.collection === collections.feature)); + write.rkey = rkey; + rewrite.rewritten += 1; + return { ...init, body: JSON.stringify(body) }; + }; + rewrite.rewritten = 0; + return rewrite; + } + + it("creates none of the records, nested ones included", async () => { + const taken = await createFeature(user.context, { title: unique("E2E feature") }); + const rewrite = collideFeatureWith(taken); + const tampered = await loginAs("user1", rewrite); + const before = await snapshotCounts(user.client); + + // The PDS answers a create on an existing record key with HTTP 500, so the outcome is unknown + // to the SDK; the counts show nothing was written. + await expect( + createCollection(tampered.context, { + title: unique("E2E collection"), + location: coordinates, + items: [ + { activity: { ...base(), locations: [coordinates] } }, + { feature: { title: unique("E2E feature") } }, + { collection: { title: unique("E2E child collection") } }, + ], + }), + ).rejects.toMatchObject({ name: "WriteOutcomeUnknownError" }); + + expect(rewrite.rewritten, "the applyWrites request was not changed").toBe(1); + expect(await snapshotCounts(user.client)).toEqual(before); + await expectStored(user.client, taken); + }); +}); diff --git a/packages/sdk/src/activity/build-activity.ts b/packages/sdk/src/activity/build-activity.ts index 37dfdc5..bbdf031 100644 --- a/packages/sdk/src/activity/build-activity.ts +++ b/packages/sdk/src/activity/build-activity.ts @@ -16,22 +16,18 @@ import { mediaValue } from "../internal/media.js"; import type { PreparedRecord } from "../internal/prepare-record.js"; import type { StrongRef } from "../internal/resolve-record-ref.js"; import { - assertDistinctKeys, type ExistingRecord, - flattenSlots, isInputObject, - newSlots, - prepareNew, + type NewSlot, + prepareSlots, type ReferenceOf, - readBeforeWriting, recordRef, - referenceLookup, resultOf, type Slot, type SupportingRecordRef, toSlot, } from "../internal/slots.js"; -import { assertWithinWriteLimit, submitWrites } from "../internal/submit-writes.js"; +import { submitWrites } from "../internal/submit-writes.js"; import { type LocationInput, locationKind } from "../location/build-location.js"; import { type WorkScopeTagInput, @@ -271,17 +267,18 @@ function planWorkScope(build: BuildContext, value: unknown, at: string): WorkSco }; } -// The slots the Activity points at, in the order their records are written, nested ones before the -// records that point at them. -function allSlots(plan: ActivityPlan): Slot[] { - return flattenSlots([ +// The slots of the records the Activity points at directly, in the order their records are written. +// Records that those point at are found by `flattenSlots` and are written before them. An inline +// value is stored in the Activity and is no record, so it is left out. +function supportingSlots(plan: ActivityPlan): Slot[] { + return [ ...(plan.workScope?.kind === "cel" ? plan.workScope.usedTags : []), ...(plan.rights ? [plan.rights] : []), ...plan.locations, ...plan.contributors.flatMap(({ identity, details }) => details ? [identity, details] : [identity], ), - ]); + ].filter((slot) => slot.choice.kind !== "inline"); } /** @@ -429,6 +426,73 @@ export interface PreparedActivity { readonly result: CreateActivityResult; } +/** + * An Activity whose input is validated and whose supporting records are planned, with nothing read, + * uploaded or written yet. + */ +export interface PlannedActivity { + /** + * The Activity as a record to create. The supporting records it points at are its references, + * so `flattenSlots` finds them, and they are prepared before it. + */ + readonly slot: NewSlot; + /** What the caller gets back for the Activity, once its records are prepared. */ + readonly result: ( + created: ReadonlyMap, + refOf: ReferenceOf, + ) => CreateActivityResult; +} + +/** + * Validates the input of an Activity and plans it and its supporting records, without sending any + * request: no record is read, no file uploaded, no record written. The `image` is uploaded when the + * slot is prepared (see `prepareNew`), after the records it points at have been built. + * + * Internal. `prepareActivity` plans, reads and prepares one Activity on its own. A call that + * creates several Activities (see `createCollection`) plans them all, so that it can count the + * records and read every existing one before anything is uploaded. + * + * `path` names the Activity's data in messages when it is part of a larger input, for example + * `activity` gives `activity.locations[0]`; omit it when the input is the Activity itself. + * + * Throws TypeError, naming the field, for any problem in the input. + */ +export function planActivitySlot( + build: BuildContext, + input: unknown, + path?: string, +): PlannedActivity { + const field = path ?? "input"; + const at = path === undefined ? "" : `${path}.`; + const parsed = parseInput(build, field, activityInput, input); + const plan = planActivity(build, parsed, at); + const slot: NewSlot = { + collection: activity.main.$type, + field, + choice: { + kind: "new", + key: undefined, + references: { supporting: supportingSlots(plan) }, + prepare: async (refOf) => { + const image = + parsed.image === undefined + ? undefined + : await mediaValue(build, `${at}image`, parsed.image, "image"); + return prepareChecked( + build, + path ?? "the Activity", + activity.main, + activityRecord(build, parsed, plan, image, refOf), + ); + }, + }, + }; + return { + slot, + result: (created, refOf) => ({ ...refOf(slot), ...summarise(plan, created, refOf) }), + }; +} + /** * Does everything `createActivity` does before it writes: validates the input, reads the existing * records it points at, uploads blobs and prepares every new record, the Activity last. @@ -451,33 +515,14 @@ export async function prepareActivity( options: { path?: string; otherWrites?: number } = {}, ): Promise { const { path, otherWrites = 0 } = options; - const at = path === undefined ? "" : `${path}.`; - const parsed = parseInput(build, path ?? "input", activityInput, input); - const plan = planActivity(build, parsed, at); - const slots = allSlots(plan); - const fresh = newSlots(slots); - assertDistinctKeys(build, fresh); - // The Activity itself is one more record in the write. - assertWithinWriteLimit(build.operation, fresh.length + 1 + otherWrites); - + const planned = planActivitySlot(build, input, path); // Reads first, then uploads: a URI that cannot be resolved fails before anything is uploaded. - const existing = await readBeforeWriting(build, slots, { checkKeys: true }); - const [image, created] = await Promise.all([ - parsed.image === undefined ? undefined : mediaValue(build, `${at}image`, parsed.image, "image"), - prepareNew(slots, existing), - ]); - - const refOf = referenceLookup(existing, created); - const record = await prepareChecked( - build, - "the Activity", - activity.main, - activityRecord(build, parsed, plan, image, refOf), - ); - return { - writes: [...created.values(), record], - result: { ...record.ref, ...summarise(plan, created, refOf) }, - }; + // The write limit counts the Activity itself too. + const { created, refOf } = await prepareSlots(build, [planned.slot], { + checkKeys: true, + otherWrites, + }); + return { writes: [...created.values()], result: planned.result(created, refOf) }; } function activityRecord( diff --git a/packages/sdk/src/collection/build-collection.ts b/packages/sdk/src/collection/build-collection.ts new file mode 100644 index 0000000..1d1a12d --- /dev/null +++ b/packages/sdk/src/collection/build-collection.ts @@ -0,0 +1,409 @@ +import { l } from "@atproto/lex"; +import * as strongRef from "@hypercerts-org/lexicon/com/atproto/repo/strongRef"; +import * as activity from "@hypercerts-org/lexicon/org/hypercerts/claim/activity"; +import * as collection from "@hypercerts-org/lexicon/org/hypercerts/collection"; +import * as defs from "@hypercerts-org/lexicon/org/hypercerts/defs"; +import * as feature from "@hypercerts-org/lexicon/org/hypercerts/entity/feature"; +import { + type ActivityInput, + type CreateActivityResult, + planActivitySlot, +} from "../activity/build-activity.js"; +import { + type CreateFeatureResult, + type FeatureInput, + type FeatureTree, + featureKind, + featureResult, + type LeafletDocument, +} from "../feature/build-feature.js"; +import { type BuildContext, parseInput } from "../internal/build-context.js"; +import { longDescription } from "../internal/long-description.js"; +import { + compact, + optionalMedia, + optionalText, + strictObject, + weightString, +} from "../internal/input.js"; +import { mediaTransform } from "../internal/media.js"; +import type { PreparedRecord } from "../internal/prepare-record.js"; +import { buildRecord, recordKind } from "../internal/record-kind.js"; +import type { StrongRef } from "../internal/resolve-record-ref.js"; +import { + type ExistingRecord, + isExistingRecord, + type NewSlot, + planNew, + type ReferenceOf, + recordRef, + resultOf, + type Slot, + type SupportingRecordRef, + toExistingSlot, + toSlot, +} from "../internal/slots.js"; +import { type LocationInput, locationKind } from "../location/build-location.js"; +import { + type VocabularyTagInput, + type VocabularyTagResult, + vocabularyTagKind, +} from "../tag/build-tag.js"; + +/** + * One item of a new Collection: exactly one of `activity`, `feature`, `collection` or `uri`, with an + * optional `weight`. + * + * @remarks + * The choice is explicit and never guessed from the fields: the data of a new Activity, Feature + * or Collection goes under `activity`, `feature` or `collection`, and an existing record under + * `uri`. A bare Activity (`{ title, shortDescription }`) is not an item. An existing record is + * reused as it is and never changed; it must be an Activity, a Feature or a Collection. + */ +export type CollectionItemInput = { + /** + * The weight of the item in the Collection, a positive decimal number written as a string (for + * example `"2"` or `"0.5"`). Weights need not add up to anything; consumers normalise them. The + * string is stored as written. At most 100 characters. Left out of the item when not set. + */ + weight?: string | undefined; +} & ( + | { + /** A new Activity and its supporting records. See {@link ActivityInput}. */ + activity: ActivityInput; + feature?: never; + collection?: never; + uri?: never; + } + | { + /** A new Feature and its tags and locations. See {@link FeatureInput}. */ + feature: FeatureInput; + activity?: never; + collection?: never; + uri?: never; + } + | { + /** A new child Collection, which can have items of its own. See {@link CollectionInput}. */ + collection: CollectionInput; + activity?: never; + feature?: never; + uri?: never; + } + | { + /** + * An existing Activity, Feature or Collection as `at:////`, identified + * by a DID, not a handle. It may belong to another account if it is public. + */ + uri: string; + activity?: never; + feature?: never; + collection?: never; + } +); + +/** The data of a new `org.hypercerts.collection` record and the records it needs. */ +export interface CollectionInput { + /** The display name. At most 800 characters and 80 graphemes. */ + title: string; + /** + * The kind of Collection: `favorites`, `project`, `portfolio` or `program`. Other values are + * accepted. At most 64 characters. + */ + type?: string | undefined; + /** A summary for previews, at most 3000 characters and 300 graphemes. */ + shortDescription?: string | undefined; + /** + * Rich-text annotations (mentions, links, hashtags) of `shortDescription`, as + * `app.bsky.richtext.facet` objects. Passed to the record as given; the lexicon validates them. + */ + shortDescriptionFacets?: collection.Main["shortDescriptionFacets"]; + /** + * A long description: plain text, a {@link LeafletDocument}, or `{ uri }` of an existing record + * of any type, stored as a reference to it. A string and a document are stored inline, as they + * are (a document needs its `$type`). The SDK never creates a document record. + */ + description?: string | LeafletDocument | ExistingRecord | undefined; + /** Languages of the text fields as BCP-47 tags (up to three), for example `["en"]`. */ + langs?: readonly string[] | undefined; + /** + * The avatar: an image `Blob` or `File` (JPEG, PNG or WebP, up to 5 MB) that the SDK uploads, or + * a URL string that is stored as it is. + */ + avatar?: Blob | string | undefined; + /** + * The banner behind the Collection's view: an image `Blob` or `File` (JPEG, PNG or WebP, up to + * 10 MB) that the SDK uploads, or a URL string that is stored as it is. + */ + banner?: Blob | string | undefined; + /** Where the Collection's work happened: the data of a new location or `{ uri }` of an existing one. */ + location?: LocationInput | ExistingRecord | undefined; + /** + * The vocabulary tags that classify the Collection, at most 20: each entry is the data of a new + * vocabulary tag, which gets the record key `.`, or `{ uri }` of an existing one. + */ + tags?: readonly (VocabularyTagInput | ExistingRecord)[] | undefined; + /** The items, in the order they are stored, at most 1000. See {@link CollectionItemInput}. */ + items?: readonly CollectionItemInput[] | undefined; + /** An ISO 8601 date-time. Defaults to the time of the call. */ + createdAt?: string | undefined; +} + +/** The kinds of record an item can be. */ +export type CollectionItemType = "activity" | "feature" | "collection"; + +/** + * What a {@link createCollection} call did for one item, in the order of the input: whether it + * created the record (`created`) and which kind of record it is (`type`). For a new item the + * result also holds what the call created for it: narrow on `created` and `type`. + */ +export type CreateCollectionItemResult = + | (CreateActivityResult & { type: "activity"; created: true }) + | (CreateFeatureResult & { type: "feature"; created: true }) + | (CreateCollectionResult & { type: "collection"; created: true }) + | (SupportingRecordRef & { type: CollectionItemType; created: false }); + +/** + * The result of {@link createCollection}. + * + * @remarks + * `items[i]` and `tags[i]` belong to the input entry at the same index, whether it was created or + * reused. Filter on `created` for only the records this call wrote. + */ +export interface CreateCollectionResult { + /** The URI of the new Collection. */ + uri: StrongRef["uri"]; + /** The CID of the new Collection. */ + cid: StrongRef["cid"]; + /** The location, when the input had `location`. */ + location?: SupportingRecordRef; + /** The vocabulary tags, in input order. */ + tags: readonly VocabularyTagResult[]; + /** The items, in input order. */ + items: readonly CreateCollectionItemResult[]; +} + +/** A Collection and every record it needs, planned: nothing is read, uploaded or written yet. */ +export interface PlannedCollection { + /** The Collection as a record to create. Everything it points at is a reference of the slot. */ + readonly slot: NewSlot; + /** What the caller gets back for the Collection once its records are prepared. */ + readonly result: ( + created: ReadonlyMap, + refOf: ReferenceOf, + ) => CreateCollectionResult; +} + +// The choices of `description`, `location`, `tags` and `items` are read below, where what each may +// hold is known, so the schema only says where they are. The stored value of an item is not the +// input's, so the record is built by hand from the plan and `buildRecord` only completes it. +const collectionKind = recordKind( + collection.main, + { + type: optionalText, + title: l.string(), + shortDescription: optionalText, + shortDescriptionFacets: l.optional(l.array(l.unknown())), + description: l.optional(l.unknown()), + langs: l.optional(l.array(l.string())), + avatar: optionalMedia, + banner: optionalMedia, + location: l.optional(l.unknown()), + tags: l.optional(l.array(l.unknown())), + items: l.optional(l.array(l.unknown())), + }, + { avatar: mediaTransform("image"), banner: mediaTransform("largeImage") }, +); + +const itemInput = strictObject({ + activity: l.optional(l.unknown()), + feature: l.optional(l.unknown()), + collection: l.optional(l.unknown()), + uri: l.optional(l.string()), + weight: l.optional(weightString), +}); + +const ITEM_CHOICES = ["activity", "feature", "collection", "uri"] as const; + +// The record types an existing item may have. +const ITEM_COLLECTIONS = [activity.main.$type, feature.main.$type, collection.main.$type] as const; +const ITEM_TYPES = { + [activity.main.$type]: "activity", + [feature.main.$type]: "feature", + [collection.main.$type]: "collection", +} as const satisfies Record<(typeof ITEM_COLLECTIONS)[number], CollectionItemType>; + +interface PlannedItem { + readonly slot: Slot; + readonly weight: string | undefined; + readonly result: ( + created: ReadonlyMap, + refOf: ReferenceOf, + ) => CreateCollectionItemResult; +} + +/** + * Validates the input of a Collection and plans it and every record it needs, at any depth, + * without sending any request, so every problem in the data is found before the first one. + * + * Internal. `field` names the data in messages, for example `input` gives `input.items[0].uri`. + * New child records are planned with their own record families' builders: Activities with + * `planActivitySlot`, Features with `featureKind`, child Collections with this function. + * + * Throws TypeError, naming the field, for any problem in the data. + */ +export function planCollection( + build: BuildContext, + field: string, + value: unknown, +): PlannedCollection { + // Hand-written instead of a `recordKind` with `references`: an item stores a strong reference plus + // a weight, which a reference field (a plain list of strong references) cannot express. + const plan = planFields(build, field, value); + const slot: NewSlot = { + collection: collection.main.$type, + field, + choice: { + kind: "new", + key: undefined, + references: compact({ + description: plan.description, + location: plan.location, + tags: plan.tags, + items: plan.items?.map((item) => item.slot), + }), + prepare: (refOf) => buildRecord(collectionKind, build, storedFields(plan, refOf), field), + }, + }; + return { + slot, + result: (created, refOf) => ({ + ...refOf(slot), + ...compact({ location: plan.location && recordRef(plan.location, created, refOf) }), + tags: (plan.tags ?? []).map((tag) => resultOf(tag, created, refOf)), + items: (plan.items ?? []).map((item) => item.result(created, refOf)), + }), + }; +} + +interface CollectionPlan { + readonly input: l.InferOutput; + /** The existing record of a `{ uri }` description. */ + readonly description: Slot | undefined; + readonly location: Slot | undefined; + readonly tags: readonly Slot[] | undefined; + readonly items: readonly PlannedItem[] | undefined; +} + +function planFields(build: BuildContext, field: string, value: unknown): CollectionPlan { + const input = parseInput(build, field, collectionKind.input, value); + return { + input, + description: planDescription(build, `${field}.description`, input.description), + location: + input.location === undefined + ? undefined + : toSlot(build, `${field}.location`, input.location, locationKind, false), + tags: input.tags?.map((tag, i) => + toSlot(build, `${field}.tags[${i}]`, tag, vocabularyTagKind, false), + ), + items: input.items?.map((item, i) => planItem(build, `${field}.items[${i}]`, item)), + }; +} + +// The input with each field that holds records replaced by what is stored for it. +function storedFields( + { input, description, location, tags, items }: CollectionPlan, + refOf: ReferenceOf, +) { + return { + ...input, + ...compact({ + description: descriptionValue(input.description, description, refOf), + location: location && refOf(location), + tags: tags?.map(refOf), + items: items?.map((item) => + compact({ itemIdentifier: refOf(item.slot), itemWeight: item.weight }), + ), + }), + }; +} + +// Only `{ uri }` is a record. A string and a Leaflet document are stored in the Collection itself. +function planDescription(build: BuildContext, field: string, value: unknown): Slot | undefined { + if (value === undefined) return undefined; + const description = parseInput(build, field, longDescription, value); + return isExistingRecord(description) ? toExistingSlot(build, field, description) : undefined; +} + +// The union of the lexicon's `description` has the strong reference as a member, so a reference is +// stored with its `$type`. +function descriptionValue(description: unknown, existing: Slot | undefined, refOf: ReferenceOf) { + if (existing !== undefined) return strongRef.main.build(refOf(existing)); + return typeof description === "string" + ? defs.descriptionString.build({ value: description }) + : description; +} + +function planItem(build: BuildContext, field: string, value: unknown): PlannedItem { + const item = parseInput(build, field, itemInput, value); + const given = ITEM_CHOICES.filter((choice) => item[choice] !== undefined); + if (given.length !== 1) { + throw new TypeError( + `${build.operation}: ${field} must have exactly one of activity, feature, collection or uri, but has ${given.length === 0 ? "none" : given.join(" and ")}. Use { activity }, { feature } or { collection } with the data of a new record, or { uri } of an existing one, and an optional weight.`, + ); + } + // `uri` first: it is the only choice the schema types, the rest are planned by their builders. + const target = + item.uri !== undefined + ? existingItem(build, `${field}.uri`, item.uri) + : planNewItem(build, field, item); + return { ...target, weight: item.weight }; +} + +function planNewItem( + build: BuildContext, + field: string, + item: l.InferOutput, +): Omit { + if (item.activity !== undefined) { + const planned = planActivitySlot(build, item.activity, `${field}.activity`); + return { + slot: planned.slot, + result: (created, refOf) => ({ + ...planned.result(created, refOf), + type: "activity", + created: true, + }), + }; + } + if (item.feature !== undefined) { + const slot = planNew(build, `${field}.feature`, featureKind, item.feature); + return { + slot, + result: (created, refOf) => ({ + ...featureResult(resultOf(slot, created, refOf)), + type: "feature", + created: true, + }), + }; + } + const child = planCollection(build, `${field}.collection`, item.collection); + return { + slot: child.slot, + result: (created, refOf) => ({ + ...child.result(created, refOf), + type: "collection", + created: true, + }), + }; +} + +function existingItem( + build: BuildContext, + field: string, + uri: string, +): Omit { + const slot = toExistingSlot(build, field, { uri }, ITEM_COLLECTIONS); + const type = ITEM_TYPES[slot.collection]; + return { slot, result: (_created, refOf) => ({ ...refOf(slot), type, created: false }) }; +} diff --git a/packages/sdk/src/collection/create-collection.test.ts b/packages/sdk/src/collection/create-collection.test.ts new file mode 100644 index 0000000..4c89506 --- /dev/null +++ b/packages/sdk/src/collection/create-collection.test.ts @@ -0,0 +1,472 @@ +import { describe, expect, it } from "vitest"; +import { CID, DID, json } from "../test-utils/fake-pds.js"; +import { fakeRepo, methods, type ReceivedCreate, refTo, writeOf } from "../test-utils/fake-repo.js"; +import { + type CollectionInput, + type CollectionItemInput, + createCollection, + type LeafletDocument, + RecordAlreadyExistsError, + WriteOutcomeUnknownError, + WriteRejectedError, +} from "./index.js"; + +const ACTIVITY = "org.hypercerts.claim.activity"; +const COLLECTION = "org.hypercerts.collection"; +const FEATURE = "org.hypercerts.entity.feature"; +const LOCATION = "app.certified.location"; +const RIGHTS = "org.hypercerts.claim.rights"; +const VOCAB = "org.hypercerts.vocab.tag"; +const WORKSCOPE = "org.hypercerts.workscope.tag"; +const LEAFLET = "pub.leaflet.pages.linearDocument"; +// CIDv1 (raw, sha-256) of the words "activity", "feature" and "collection": the CIDs the PDS reports +// for the existing records, so a reference that carries the wrong one is noticed. +const ACTIVITY_CID = "bafkreiagkgddx5ozaitcwf6emipmgqcuj7ybm5jfipmzvewx2iuhfwfeku"; +const FEATURE_CID = "bafkreibk2vrddf3hcvyipxna33ddsh2eph4kasdjvmgmru5jynrx3lttwu"; +const COLLECTION_CID = "bafkreih2oslojlueamdn6qn5mweaaojoetnyvrdwofm5nppyui5srrcoua"; + +const uriOf = (collection: string, rkey = "3k2a4b5c6d7e8") => `at://${DID}/${collection}/${rkey}`; +const png = () => new Blob([new Uint8Array([1, 2, 3])], { type: "image/png" }); +const activity = { title: "Trees", shortDescription: "Planting trees" }; +const tag = (category: string, key: string) => ({ category, key, name: key, status: "accepted" }); +const location = (name: string) => ({ + lpVersion: "1.0", + srs: "http://www.opengis.net/def/crs/OGC/1.3/CRS84", + locationType: "coordinate-decimal", + location: { string: "90.17,27.47" }, + name, +}); + +describe("createCollection with existing records only", () => { + const activityUri = uriOf(ACTIVITY, "3jzfcijpj2z2a"); + const featureUri = uriOf(FEATURE, "3jzfcijpj2z2b"); + const collectionUri = uriOf(COLLECTION, "3jzfcijpj2z2c"); + const existing = { + [activityUri]: ACTIVITY_CID, + [featureUri]: FEATURE_CID, + [collectionUri]: COLLECTION_CID, + }; + + it("references each record by the CID it was read with, keeping order, weights and repeats", async () => { + const { context, requests, creates } = fakeRepo({ existing }); + + const result = await createCollection(context, { + title: "Portfolio", + items: [ + { uri: featureUri, weight: "0.50" }, + { uri: activityUri, weight: "2" }, + { uri: collectionUri }, + { uri: activityUri, weight: "007" }, + ], + }); + + // Each record is read once, nothing is uploaded, and the Collection is the only write. + expect(methods(requests)).toEqual([ + ...Array(3).fill("com.atproto.repo.getRecord"), + "com.atproto.repo.applyWrites", + ]); + expect(creates).toHaveLength(1); + expect(creates[0]).toHaveLength(1); + const collection = writeOf(creates[0] ?? [], COLLECTION); + expect(collection.value.items).toEqual([ + { itemIdentifier: { uri: featureUri, cid: FEATURE_CID }, itemWeight: "0.50" }, + { itemIdentifier: { uri: activityUri, cid: ACTIVITY_CID }, itemWeight: "2" }, + { itemIdentifier: { uri: collectionUri, cid: COLLECTION_CID } }, + { itemIdentifier: { uri: activityUri, cid: ACTIVITY_CID }, itemWeight: "007" }, + ]); + expect(result).toEqual({ + ...refTo(collection), + tags: [], + items: [ + { uri: featureUri, cid: FEATURE_CID, type: "feature", created: false }, + { uri: activityUri, cid: ACTIVITY_CID, type: "activity", created: false }, + { uri: collectionUri, cid: COLLECTION_CID, type: "collection", created: false }, + { uri: activityUri, cid: ACTIVITY_CID, type: "activity", created: false }, + ], + }); + }); + + it("creates a bare Collection and keeps an empty items array", async () => { + const { context, creates } = fakeRepo(); + + const bare = await createCollection(context, { title: "Empty" }); + await createCollection(context, { title: "Empty", items: [], tags: [] }); + + const first = writeOf(creates[0] ?? [], COLLECTION); + expect(first.value).toMatchObject({ $type: COLLECTION, title: "Empty" }); + expect(first.value.items).toBeUndefined(); + expect(writeOf(creates[1] ?? [], COLLECTION).value).toMatchObject({ items: [], tags: [] }); + expect(bare).toEqual({ ...refTo(first), tags: [], items: [] }); + }); +}); + +describe("createCollection with new records", () => { + const existingTag = uriOf(VOCAB, "theme.water"); + const existingCollection = uriOf(COLLECTION, "3jzfcijpj2z2c"); + + it("creates every new record in one write, with references to the exact records created", async () => { + const { context, requests, creates } = fakeRepo({ + existing: { [existingTag]: CID, [existingCollection]: COLLECTION_CID }, + }); + + const result = await createCollection(context, { + title: "Portfolio", + type: "portfolio", + location: location("root"), + tags: [tag("theme", "forest"), { uri: existingTag }], + items: [ + { activity: { ...activity, locations: [location("site")] }, weight: "3" }, + { feature: { title: "Zone", tags: [{ uri: existingTag }] } }, + { + collection: { + title: "Child", + items: [{ activity }, { uri: existingCollection, weight: "1.50" }], + }, + }, + ], + }); + + // The existing records and the new tag's key are read first; nothing is uploaded. + const sent = methods(requests); + expect(sent.at(-1)).toBe("com.atproto.repo.applyWrites"); + expect(sent.filter((method) => method !== "com.atproto.repo.getRecord")).toHaveLength(1); + // One request holds everything: records before the ones that point at them, the Collection last. + expect(creates).toHaveLength(1); + const writes = creates[0] ?? []; + expect(writes.map((write) => write.collection)).toEqual([ + LOCATION, + VOCAB, + LOCATION, + ACTIVITY, + FEATURE, + ACTIVITY, + COLLECTION, + COLLECTION, + ]); + const at = (index: number): ReceivedCreate => { + const write = writes[index]; + if (write === undefined) throw new Error(`no write at ${index}`); + return write; + }; + const [rootLocation, forest, siteLocation, tree, zone, childTree, child, root] = [ + 0, 1, 2, 3, 4, 5, 6, 7, + ].map(at) as [ + ReceivedCreate, + ReceivedCreate, + ReceivedCreate, + ReceivedCreate, + ReceivedCreate, + ReceivedCreate, + ReceivedCreate, + ReceivedCreate, + ]; + expect(forest.rkey).toBe("theme.forest"); + + expect(root.value).toMatchObject({ + $type: COLLECTION, + type: "portfolio", + location: refTo(rootLocation), + tags: [refTo(forest), { uri: existingTag, cid: CID }], + items: [ + { itemIdentifier: refTo(tree), itemWeight: "3" }, + { itemIdentifier: refTo(zone) }, + { itemIdentifier: refTo(child) }, + ], + }); + expect(tree.value.locations).toEqual([refTo(siteLocation)]); + expect(zone.value.tags).toEqual([{ uri: existingTag, cid: CID }]); + expect(child.value.items).toEqual([ + { itemIdentifier: refTo(childTree) }, + { itemIdentifier: { uri: existingCollection, cid: COLLECTION_CID }, itemWeight: "1.50" }, + ]); + + expect(result).toEqual({ + ...refTo(root), + location: { ...refTo(rootLocation), created: true }, + tags: [ + { ...refTo(forest), created: true }, + { uri: existingTag, cid: CID, created: false }, + ], + items: [ + { + ...refTo(tree), + type: "activity", + created: true, + locations: [{ ...refTo(siteLocation), created: true }], + contributors: [], + }, + { + ...refTo(zone), + type: "feature", + created: true, + tags: [{ uri: existingTag, cid: CID, created: false }], + locations: [], + }, + { + ...refTo(child), + type: "collection", + created: true, + tags: [], + items: [ + { + ...refTo(childTree), + type: "activity", + created: true, + locations: [], + contributors: [], + }, + { uri: existingCollection, cid: COLLECTION_CID, type: "collection", created: false }, + ], + }, + ], + }); + }); + + it("creates the same data twice as two records and does not search for matches", async () => { + const { context, requests, creates } = fakeRepo(); + + const result = await createCollection(context, { + title: "Twice", + items: [{ feature: { title: "Zone" } }, { feature: { title: "Zone" } }], + }); + + expect(methods(requests)).toEqual(["com.atproto.repo.applyWrites"]); + const [first, second] = result.items; + expect(first?.uri).not.toBe(second?.uri); + expect(creates[0]?.map((write) => write.collection)).toEqual([FEATURE, FEATURE, COLLECTION]); + }); +}); + +describe("createCollection description and images", () => { + const document: LeafletDocument = { $type: LEAFLET, blocks: [] }; + const documentUri = uriOf("pub.leaflet.document"); + + it.each<[string, CollectionInput["description"], object]>([ + [ + "a string inline", + "Plain text.", + { $type: "org.hypercerts.defs#descriptionString", value: "Plain text." }, + ], + ["a Leaflet document inline, as it is", document, document], + [ + "{ uri } of a record of any type as a typed strong reference", + { uri: documentUri }, + { $type: "com.atproto.repo.strongRef", uri: documentUri, cid: CID }, + ], + ])("stores %s", async (_, description, stored) => { + const { context, creates } = fakeRepo({ existing: { [documentUri]: CID } }); + + await createCollection(context, { title: "Portfolio", description }); + + expect(writeOf(creates[0] ?? [], COLLECTION).value.description).toEqual(stored); + }); + + it("keeps a URL avatar as it is and uploads a banner file before the one write", async () => { + const { context, requests, creates } = fakeRepo(); + + await createCollection(context, { + title: "Portfolio", + avatar: "https://example.org/avatar.png", + banner: png(), + }); + + expect(methods(requests)).toEqual([ + "com.atproto.repo.uploadBlob", + "com.atproto.repo.applyWrites", + ]); + expect(writeOf(creates[0] ?? [], COLLECTION).value).toMatchObject({ + avatar: { $type: "org.hypercerts.defs#uri", uri: "https://example.org/avatar.png" }, + banner: { $type: "org.hypercerts.defs#largeImage" }, + }); + }); + + it("uploads an avatar file as a small image", async () => { + const { context, creates } = fakeRepo(); + + await createCollection(context, { title: "Portfolio", avatar: png() }); + + expect(writeOf(creates[0] ?? [], COLLECTION).value.avatar).toMatchObject({ + $type: "org.hypercerts.defs#smallImage", + }); + }); +}); + +describe("createCollection when the input is wrong", () => { + const item = (value: unknown) => ({ title: "Portfolio", items: [value as CollectionItemInput] }); + + it.each<[string, CollectionInput, RegExp]>([ + [ + "an item with no choice", + item({ weight: "1" }), + /input\.items\[0\] must have exactly one of activity, feature, collection or uri, but has none/, + ], + [ + "an item with two choices", + item({ uri: uriOf(ACTIVITY), feature: { title: "Zone" } }), + /input\.items\[0\] must have exactly one of .* but has feature and uri/, + ], + [ + "a bare Activity as an item", + item(activity), + /input\.items\[0\].*Allowed fields: activity, feature, collection, uri, weight/, + ], + [ + "an item URI that is not an Activity, Feature or Collection", + item({ uri: uriOf(LOCATION) }), + /input\.items\[0\]\.uri points at a record of type app\.certified\.location, but this field holds org\.hypercerts\.claim\.activity, org\.hypercerts\.entity\.feature, org\.hypercerts\.collection/, + ], + [ + "an item URI with a handle as the authority", + item({ uri: `at://alice.example.com/${ACTIVITY}/3k2a4b5c6d7e8` }), + /input\.items\[0\]\.uri must use a DID as the authority, not a handle/, + ], + [ + "an item URI without a record key", + item({ uri: `at://did:plc:abc123/${ACTIVITY}` }), + /input\.items\[0\]\.uri is not a record URI/, + ], + ["an invalid weight", item({ uri: uriOf(ACTIVITY), weight: "0" }), /positive decimal number/], + [ + "a work-scope tag where a vocabulary tag is needed", + { title: "Portfolio", tags: [{ uri: uriOf(WORKSCOPE) }] }, + /input\.tags\[0\] points at a record of type org\.hypercerts\.workscope\.tag, but this field holds org\.hypercerts\.vocab\.tag/, + ], + [ + "a misspelled field in a child Collection", + item({ collection: { title: "Child", itms: [] } }), + /input\.items\[0\]\.collection.*Allowed fields: type, title,/, + ], + [ + "a description that is none of the three forms", + { title: "Portfolio", description: 7 as never }, + /input\.description: must be a string, \{ uri \}/, + ], + [ + "the same new tag in two places", + { + title: "Portfolio", + tags: [tag("theme", "forest")], + items: [{ feature: { title: "Zone", tags: [tag("theme", "forest")] } }], + }, + /input\.tags\[0\] and input\.items\[0\]\.feature\.tags\[0\] are both new/, + ], + ])("sends nothing for %s", async (_, input, message) => { + const { context, requests } = fakeRepo(); + + await expect(createCollection(context, input)).rejects.toThrow(message); + + expect(requests).toHaveLength(0); + }); + + it("counts the records nested in every item and sends nothing for more than 200", async () => { + const { context, requests } = fakeRepo(); + const items = Array.from({ length: 100 }, () => ({ + activity: { ...activity, locations: [location("site")] }, + })); + + // 100 Activities and 100 locations, plus the Collection. + await expect(createCollection(context, { title: "Big", items })).rejects.toThrow( + /would write 201 records, but one write holds at most 200/, + ); + + expect(requests).toHaveLength(0); + }); + + it("writes 200 records in one request, the Collection included", async () => { + const { context, creates } = fakeRepo(); + const items = Array.from({ length: 199 }, (_, i) => ({ feature: { title: `Zone ${i}` } })); + + await createCollection(context, { title: "Big", items }); + + expect(creates).toHaveLength(1); + expect(creates[0]).toHaveLength(200); + }); +}); + +describe("createCollection reads everything before it uploads anything", () => { + it("does not upload for an item that fails validation after another that has a file", async () => { + const { context, requests } = fakeRepo(); + + await expect( + createCollection(context, { + title: "Portfolio", + avatar: png(), + items: [ + { activity: { ...activity, image: png() } }, + { activity: { title: "No summary" } as never }, + ], + }), + ).rejects.toThrow(/invalid input\.items\[1\]\.activity/); + + expect(requests).toHaveLength(0); + }); + + it("does not upload for an existing record of a later item that cannot be read", async () => { + const { context, requests } = fakeRepo(); + + await expect( + createCollection(context, { + title: "Portfolio", + banner: png(), + items: [ + { activity: { ...activity, image: png() } }, + { activity: { ...activity, rights: { uri: uriOf(RIGHTS) } } }, + ], + }), + ).rejects.toThrow(/cannot resolve .*the record was not found/); + + expect(methods(requests)).toEqual(["com.atproto.repo.getRecord"]); + }); + + it("does not upload when a new tag's key is taken", async () => { + const taken = uriOf(VOCAB, "theme.forest"); + const { context, requests } = fakeRepo({ existing: { [taken]: CID } }); + + const error = await createCollection(context, { + title: "Portfolio", + avatar: png(), + tags: [tag("theme", "forest")], + }).catch((e) => e); + + expect(error).toBeInstanceOf(RecordAlreadyExistsError); + expect(error).toMatchObject({ uri: taken, attempted: [] }); + expect(methods(requests)).toEqual(["com.atproto.repo.getRecord"]); + }); +}); + +describe("createCollection when the write fails", () => { + const input: CollectionInput = { + title: "Portfolio", + items: [{ activity }, { feature: { title: "Zone" } }], + }; + const attempted = (body: Uint8Array | undefined) => + ( + JSON.parse(new TextDecoder().decode(body)).writes as { collection: string; rkey: string }[] + ).map((write) => `at://${DID}/${write.collection}/${write.rkey}`); + + it("reports a definite rejection of the single request, and does not retry", async () => { + const { context, requests } = fakeRepo({ + applyWrites: () => json(400, { error: "InvalidRequest", message: "Bad record" }), + }); + + const error = await createCollection(context, input).catch((e) => e); + + expect(error).toBeInstanceOf(WriteRejectedError); + expect(error).toMatchObject({ status: 400, code: "InvalidRequest" }); + expect(error.message).toMatch(/the 3 records were not created/); + expect(methods(requests)).toEqual(["com.atproto.repo.applyWrites"]); + }); + + it("reports an unknown outcome with every new record, the Collection last, and does not retry", async () => { + const { context, requests } = fakeRepo({ applyWrites: () => json(500, {}) }); + + const error = await createCollection(context, input).catch((e) => e); + + expect(error).toBeInstanceOf(WriteOutcomeUnknownError); + expect(methods(requests)).toEqual(["com.atproto.repo.applyWrites"]); + const uris = attempted(requests[0]?.body); + expect(uris).toHaveLength(3); + expect(error.attempted.map((ref: { uri: string }) => ref.uri)).toEqual(uris); + expect(uris.map((uri) => uri.split("/")[3])).toEqual([ACTIVITY, FEATURE, COLLECTION]); + }); +}); diff --git a/packages/sdk/src/collection/create-collection.ts b/packages/sdk/src/collection/create-collection.ts new file mode 100644 index 0000000..464f5ec --- /dev/null +++ b/packages/sdk/src/collection/create-collection.ts @@ -0,0 +1,156 @@ +import type { SDKContext } from "../context.js"; +import { createBuildContext } from "../internal/build-context.js"; +import { prepareSlots } from "../internal/slots.js"; +import { submitWrites } from "../internal/submit-writes.js"; +import { + type CollectionInput, + type CreateCollectionResult, + planCollection, +} from "./build-collection.js"; + +const OPERATION = "createCollection"; + +/** + * Creates a Collection (`org.hypercerts.collection`) with its items and supporting records, in one + * atomic write: the Collection, and every new Activity, Feature, child Collection, location and + * tag in it. + * + * @remarks + * Each item is exactly one of `{ activity }`, `{ feature }` and `{ collection }` (the data of a + * new record, created with the same rules as {@link createActivity}, {@link createFeature} and + * this function) or `{ uri }` (an existing Activity, Feature or Collection, reused as it is). The + * choice is explicit; the SDK never guesses it from the fields. Each item can have a `weight`. + * The same holds for `location` and `tags`: the data of a new record, or `{ uri }` of an existing + * one. The SDK builds every strong reference itself, pointing at the exact CID of each new record. + * + * Existing records are read once to learn their current CID, never rewritten, and may belong to + * another account if they are public. Nothing is searched for or deduplicated: the same data twice + * creates two records, and the same `{ uri }` twice gives two items. Order is kept, and weights are + * stored as the strings you pass; they are not normalised. + * + * `description` is a string or a Leaflet document (with `$type: "pub.leaflet.pages.linearDocument"`), + * both stored in the Collection, or `{ uri }` of an existing record of any type, stored as a + * reference. `avatar` and `banner` are a URL string, stored as it is, or an image `Blob` or `File` + * that the SDK uploads. `createdAt` defaults to the time of the call, for every new record. + * + * Everything is created in one `com.atproto.repo.applyWrites` request, which the PDS commits as a + * whole: all the records exist afterwards, or none do. One write holds at most 200 records, the + * Collection and every record nested in it included; the SDK never splits a call, so a larger one + * throws before anything is read or uploaded. Create some records first and pass them as `{ uri }`. + * The session needs `create` on every collection it writes to. + * + * A new tag's record key is `.`, so the SDK checks first that no such tag exists and + * throws {@link RecordAlreadyExistsError} if one does; use the existing tag's `uri` instead. Two + * new tags with the same key in one call are a `TypeError`. + * + * Order: validate input, count records, read existing records and tag keys, upload blobs, build, + * write. Input and read errors stop before any upload; a rejected file or lexicon rule is found + * at build time, after other files may have uploaded. Uploaded blobs are not rolled back; see + * {@link uploadBlob}. + * + * Not supported: the lexicon's `signatures`, a blob that was uploaded already, adding items to a + * Collection that exists (see {@link createActivityInCollection} for an Activity), and changing + * an existing record. + * + * @param context - The shared context from {@link createSDKContext}. Its client must be + * authenticated; the records are created in that account's repo. + * @param input - The Collection, its items and the records it needs. See {@link CollectionInput}. + * @returns The new Collection's URI and CID, and what was created or reused for its location, tags + * and items, in input order. See {@link CreateCollectionResult}. + * @throws {@link TypeError} If `input` has an unknown field, an item that is not exactly one of + * the four choices, a `{ uri }` that is not an Activity, Feature or Collection (or a location or + * tag of the wrong type), a `description` that is none of the three forms, an invalid weight, a + * nested record that is invalid, more new records than one write holds, or any value the lexicons + * do not allow. The message names the field. Nothing was written. + * @throws {@link Error} If the client is not authenticated, an existing record cannot be read, the + * key of a new tag cannot be checked, or an upload fails. Nothing was written. + * @throws {@link RecordAlreadyExistsError} If a new tag's key is taken. Nothing was written. + * @throws {@link WriteRejectedError} If the PDS rejected the write (HTTP 4xx). + * @throws {@link WriteOutcomeUnknownError} If the PDS gave no usable answer. Not safe to retry: + * a successful first attempt would be duplicated. `attempted` lists the new records, Collection last. + * @throws {@link WriteResultMismatchError} If the PDS reported different URIs or CIDs than the SDK + * computed. + * + * @example A Collection with a new Activity, a new Feature and an existing Activity + * ```ts + * import { createCollection } from "@hypercerts-org/sdk/collection"; + * + * // `context` comes from createSDKContext. + * const collection = await createCollection(context, { + * type: "portfolio", + * title: "Phobjikha valley restoration", + * description: "Everything we do in the valley.", + * avatar: "https://example.org/valley.png", // a URL is stored as it is + * items: [ + * { + * activity: { + * title: "Reforestation in the Phobjikha valley", + * shortDescription: "Planting 5,000 native trees.", + * }, + * weight: "2", // created + * }, + * { feature: { type: "zone", title: "Valley floor zone" } }, // created + * { uri: "at://did:plc:abc123/org.hypercerts.claim.activity/3k2a4b5c6d7e8", weight: "0.5" }, // reused + * ], + * }); + * console.log(collection.uri, collection.items[0]); // { uri, cid, type: "activity", created: true, ... } + * ``` + * + * @example A child Collection, a new tag and an existing location + * ```ts + * import { createCollection } from "@hypercerts-org/sdk/collection"; + * + * await createCollection(context, { + * title: "Bhutan programmes", + * location: { uri: "at://did:plc:abc123/app.certified.location/3k2a4b5c6d7e8" }, // reused + * tags: [{ category: "theme", key: "forest", name: "Forest", status: "accepted" }], // created + * items: [ + * { + * collection: { + * title: "Phobjikha", + * items: [{ uri: "at://did:plc:abc123/org.hypercerts.collection/3k2a4b5c6d7e8" }], + * }, + * }, + * ], + * }); + * ``` + * + * @example An item is not a bare Activity + * ```ts + * import { createCollection } from "@hypercerts-org/sdk/collection"; + * + * await createCollection(context, { + * title: "Portfolio", + * // TypeError: an item is { activity }, { feature }, { collection } or { uri } + * items: [{ title: "Reforestation", shortDescription: "Planting trees." } as never], + * }); + * ``` + * + * @example Handling an unknown outcome + * ```ts + * import { createCollection, WriteOutcomeUnknownError } from "@hypercerts-org/sdk/collection"; + * + * try { + * await createCollection(context, input); + * } catch (error) { + * if (error instanceof WriteOutcomeUnknownError) { + * // The Collection is last. Check it with com.atproto.repo.getRecord: the write is + * // all-or-nothing, so if it exists, every record in `attempted` does. + * console.log(error.attempted.map((ref) => ref.uri)); + * } + * throw error; + * } + * ``` + */ +export async function createCollection( + context: SDKContext, + input: CollectionInput, +): Promise { + const build = createBuildContext(context, OPERATION); + const planned = planCollection(build, "input", input); + // Reads first, then uploads: a URI that cannot be resolved, or a taken tag key, fails before + // anything is uploaded. + const { created, refOf } = await prepareSlots(build, [planned.slot], { checkKeys: true }); + await submitWrites(context, build.did, [...created.values()], OPERATION); + return planned.result(created, refOf); +} diff --git a/packages/sdk/src/collection/index.ts b/packages/sdk/src/collection/index.ts index e9512f1..1e06e57 100644 --- a/packages/sdk/src/collection/index.ts +++ b/packages/sdk/src/collection/index.ts @@ -1,14 +1,31 @@ // Entry point for @hypercerts-org/sdk/collection. export type { ActivityInput, CreateActivityResult } from "../activity/build-activity.js"; +export type { + CreateFeatureResult, + FeatureInput, + LeafletDocument, +} from "../feature/build-feature.js"; export type { StrongRef } from "../internal/resolve-record-ref.js"; +export type { ExistingRecord, SupportingRecordRef } from "../internal/slots.js"; export { + RecordAlreadyExistsError, WriteError, WriteOutcomeUnknownError, WriteRejectedError, WriteResultMismatchError, } from "../internal/submit-writes.js"; +export type { LocationInput } from "../location/build-location.js"; +export type { VocabularyTagInput, VocabularyTagResult } from "../tag/build-tag.js"; +export type { + CollectionInput, + CollectionItemInput, + CollectionItemType, + CreateCollectionItemResult, + CreateCollectionResult, +} from "./build-collection.js"; export type { CreateActivityInCollectionInput, CreateActivityInCollectionResult, } from "./create-activity-in-collection.js"; export { createActivityInCollection } from "./create-activity-in-collection.js"; +export { createCollection } from "./create-collection.js"; diff --git a/packages/sdk/src/feature/build-feature.ts b/packages/sdk/src/feature/build-feature.ts index 98a3747..439c9c9 100644 --- a/packages/sdk/src/feature/build-feature.ts +++ b/packages/sdk/src/feature/build-feature.ts @@ -76,7 +76,7 @@ export interface CreateFeatureResult { locations: readonly SupportingRecordRef[]; } -const featureKind: RecordKind = recordKind( +export const featureKind: RecordKind = recordKind( feature.main, { type: optionalText, @@ -95,7 +95,7 @@ const featureKind: RecordKind = recordKind( }, ); -type FeatureTree = SupportingRecordRef & { +export type FeatureTree = SupportingRecordRef & { tags?: readonly VocabularyTagResult[]; locations?: readonly SupportingRecordRef[]; }; @@ -200,11 +200,12 @@ export async function createFeature( context: SDKContext, input: FeatureInput, ): Promise { - const { uri, cid, tags, locations } = await createRecordTree( - context, - "createFeature", - featureKind, - input, + return featureResult( + await createRecordTree(context, "createFeature", featureKind, input), ); +} + +/** The public result of a Feature, from what `createRecordTree` reports for its record. */ +export function featureResult({ uri, cid, tags, locations }: FeatureTree): CreateFeatureResult { return { uri, cid, tags: tags ?? [], locations: locations ?? [] }; } diff --git a/packages/sdk/src/internal/resolve-record-ref.test.ts b/packages/sdk/src/internal/resolve-record-ref.test.ts index 1f67c99..4213a82 100644 --- a/packages/sdk/src/internal/resolve-record-ref.test.ts +++ b/packages/sdk/src/internal/resolve-record-ref.test.ts @@ -58,6 +58,39 @@ describe("resolveRecordRef", () => { expect(read.searchParams.get("repo")).toBe(owner); }); + it("resolves an owner's PDS once for concurrent reads that share a client map", async () => { + const owner = "did:plc:ewvi7nxzyoun6zhxrhs64oiz"; + const uris = ["3k2a4b5c6d7e8", "3k2a4b5c6d7e9"].map( + (rkey) => `at://${owner}/org.hypercerts.claim.activity/${rkey}`, + ); + let didFetches = 0; + vi.stubGlobal("fetch", async (input: Request | string | URL) => { + const url = new URL(input instanceof Request ? input.url : String(input)); + if (url.origin === "https://plc.directory") { + didFetches += 1; + return json(200, { + id: owner, + service: [ + { + id: "#atproto_pds", + type: "AtprotoPersonalDataServer", + serviceEndpoint: "https://owner-pds.test", + }, + ], + }); + } + const rkey = url.searchParams.get("rkey"); + return json(200, { uri: uris.find((u) => u.endsWith(`/${rkey}`)), cid: CID, value: {} }); + }); + const { context } = fakePds(() => json(500, {})); + const pdsClients = new Map(); + + const refs = await Promise.all(uris.map((uri) => resolveRecordRef(context, uri, pdsClients))); + + expect(refs).toEqual(uris.map((uri) => ({ uri, cid: CID }))); + expect(didFetches).toBe(1); + }); + it("fails without asking the client's PDS when the owner's PDS cannot be resolved", async () => { const uri = "at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/org.hypercerts.claim.activity/3k2a4b5c6d7e8"; vi.stubGlobal("fetch", async () => json(404, {})); diff --git a/packages/sdk/src/internal/resolve-record-ref.ts b/packages/sdk/src/internal/resolve-record-ref.ts index b701749..c11a0d7 100644 --- a/packages/sdk/src/internal/resolve-record-ref.ts +++ b/packages/sdk/src/internal/resolve-record-ref.ts @@ -34,8 +34,12 @@ export interface StoredRecord { * * Throws what `readRecord` throws. */ -export async function resolveRecordRef(context: SDKContext, uri: string): Promise { - return (await readRecord(context, uri, "resolveRecordRef")).ref; +export async function resolveRecordRef( + context: SDKContext, + uri: string, + pdsClients?: PdsClients, +): Promise { + return (await readRecord(context, uri, "resolveRecordRef", pdsClients)).ref; } /** @@ -65,11 +69,12 @@ export async function readRecord( context: SDKContext, uri: string, operation: string, + pdsClients?: PdsClients, ): Promise { const { atUri: requested, did, collection, rkey } = parseRecordUri(uri, operation); let response: Awaited>; try { - const client = await clientForRepo(context, did); + const client = await clientForRepo(context, did, pdsClients); response = await client.getRecord(collection, rkey, { repo: did }); } catch (error) { const reason = isRecordNotFound(error) @@ -119,9 +124,25 @@ export async function recordExists( } } -async function clientForRepo(context: SDKContext, did: DidString): Promise { +/** + * Clients for other accounts' PDSes, keyed by DID, so that reads of several records from one + * account share a single DID resolution, including reads still in flight. Scope one map to one + * operation: a DID's PDS can change. + */ +export type PdsClients = Map>; + +async function clientForRepo( + context: SDKContext, + did: DidString, + pdsClients?: PdsClients, +): Promise { if (context.client.did === did) return context.client; - return new Client(await resolvePds(did)); + let pending = pdsClients?.get(did); + if (pending === undefined) { + pending = resolvePds(did).then((pds) => new Client(pds)); + pdsClients?.set(did, pending); + } + return pending; } /** diff --git a/packages/sdk/src/internal/slots.ts b/packages/sdk/src/internal/slots.ts index dc9899d..6f1f8a1 100644 --- a/packages/sdk/src/internal/slots.ts +++ b/packages/sdk/src/internal/slots.ts @@ -1,11 +1,17 @@ -import { l, type NsidString, type RecordKeyString } from "@atproto/lex"; +import { isDidString, l, type NsidString, type RecordKeyString } from "@atproto/lex"; import * as strongRef from "@hypercerts-org/lexicon/com/atproto/repo/strongRef"; import { type BuildContext, parseInput } from "./build-context.js"; import { describeError, describeValue } from "./describe.js"; import type { PreparedRecord } from "./prepare-record.js"; import { buildRecord, type RecordKind, type ReferenceField, recordKeyOf } from "./record-kind.js"; -import { recordExists, resolveRecordRef, type StrongRef } from "./resolve-record-ref.js"; -import { RecordAlreadyExistsError } from "./submit-writes.js"; +import { + type PdsClients, + parseRecordUri, + recordExists, + resolveRecordRef, + type StrongRef, +} from "./resolve-record-ref.js"; +import { assertWithinWriteLimit, RecordAlreadyExistsError } from "./submit-writes.js"; /** * Points at a record that already exists, to reuse it as it is. @@ -135,11 +141,26 @@ export function toSlot( } /** - * Reads a field that accepts only an existing record, of any type: `{ uri }`. + * Reads a field that accepts only an existing record: `{ uri }`. Any record type is accepted, + * unless `collections` lists the ones that are, in which case the slot's `collection` is the one + * the URI points at. * - * Throws TypeError, naming `field`, when `value` is not `{ uri }` of a record. + * Throws TypeError, naming `field`, when `value` is not `{ uri }` of a record, or of a record of + * another type than `collections`. */ -function toExistingSlot(build: BuildContext, field: string, value: unknown): Slot { +export function toExistingSlot(build: BuildContext, field: string, value: unknown): Slot; +export function toExistingSlot( + build: BuildContext, + field: string, + value: unknown, + collections: readonly TCollection[], +): Slot & { readonly collection: TCollection }; +export function toExistingSlot( + build: BuildContext, + field: string, + value: unknown, + collections?: readonly NsidString[], +): Slot { const expected = "{ uri } of an existing record"; if (!isExistingRecord(value)) { throw new TypeError( @@ -147,7 +168,9 @@ function toExistingSlot(build: BuildContext, field: string, value: unknown): Slo ); } assertNotMixed(build, field, value, "uri", expected); - return { collection: undefined, field, choice: existing(build, field, value, undefined) }; + const choice = existing(build, field, value, collections); + const collection = collections?.find((candidate) => candidate === collectionOf(choice.uri)); + return { collection, field, choice }; } /** @@ -249,22 +272,39 @@ function assertNotMixed( const inlineValue = l.object({ inline: l.string() }); const existingRecord = l.object({ uri: l.string({ format: "at-uri" }) }); +// The collection part of a record URI, which `at-uri` alone does not require. +const collectionOf = (uri: string): string | undefined => uri.split("/")[3]; + function existing( build: BuildContext, field: string, value: object, - collection: NsidString | undefined, -): SlotChoice { + collections: NsidString | readonly NsidString[] | undefined, +): Extract { const { uri } = parseInput(build, field, existingRecord, value); - // `at-uri` alone also accepts other record types, handle authorities and URIs without a key. The - // type is checked here so a wrong one fails before any request; `resolveRecordRef` rejects the - // rest. - const actual = uri.split("/")[3]; - if (actual === undefined || (collection !== undefined && actual !== collection)) { + const allowed: readonly string[] | undefined = + typeof collections === "string" ? [collections] : collections; + // `at-uri` alone also accepts other record types, handle authorities and URIs without a key. All + // are checked here so a wrong one fails before any request, naming the field. + const authority = uri.split("/")[2]; + if (authority === undefined || !isDidString(authority)) { + throw new TypeError( + `${build.operation}: ${field} must use a DID as the authority, not a handle: ${uri}. Expected at:////.`, + ); + } + const actual = collectionOf(uri); + if (actual === undefined || (allowed !== undefined && !allowed.includes(actual))) { const target = actual === undefined ? "something that is not a record" : `a record of type ${actual}`; throw new TypeError( - `${build.operation}: ${field} points at ${target}, but this field holds ${collection ?? "a record"}: ${uri}. Expected at:///${collection ?? ""}/.`, + `${build.operation}: ${field} points at ${target}, but this field holds ${allowed?.join(", ") ?? "a record"}: ${uri}. Expected at:///${allowed?.join("|") ?? ""}/.`, + ); + } + try { + parseRecordUri(uri, build.operation); + } catch { + throw new TypeError( + `${build.operation}: ${field} is not a record URI: ${uri}. Expected at:///${actual}/.`, ); } return { kind: "existing", uri }; @@ -310,7 +350,7 @@ export function assertDistinctKeys(build: BuildContext, slots: readonly NewSlot[ /** * Reads, before anything is uploaded or written, what the call depends on: every `existing` slot - * from the PDS that hosts it, in parallel, and (with `checkKeys`) that no record exists yet at the + * from the PDS that hosts it (each URI once), in parallel, and (with `checkKeys`) that no record exists yet at the * key a new record chose. A failure here has no side effects. * * Throws what `resolveRecordRef` throws, and RecordAlreadyExistsError when a chosen key is taken. @@ -320,10 +360,20 @@ export async function readBeforeWriting( slots: readonly Slot[], { checkKeys }: { checkKeys: boolean }, ): Promise> { + // A URI repeated across slots is read once, so every slot gets the same CID. + const reads = new Map>(); + // Records from one remote account share one resolution of its PDS. + const pdsClients: PdsClients = new Map(); + const read = (uri: string): Promise => { + let pending = reads.get(uri); + if (pending === undefined) { + pending = resolveRecordRef(build.context, uri, pdsClients); + reads.set(uri, pending); + } + return pending; + }; const [resolved] = await Promise.all([ - settle(slots, ({ choice }) => - choice.kind === "existing" ? resolveRecordRef(build.context, choice.uri) : undefined, - ), + settle(slots, ({ choice }) => (choice.kind === "existing" ? read(choice.uri) : undefined)), checkKeys ? assertKeysFree(build, newSlots(slots)) : undefined, ]); return resolved; @@ -418,6 +468,35 @@ export function referenceLookup( }; } +/** What `prepareSlots` returns: the reads, the records to write, and every slot's reference. */ +export interface PreparedSlots { + existing: Map; + created: Map; + refOf: ReferenceOf; +} + +/** + * Plans, reads and prepares every record `roots` writes or points at, in the order every caller + * needs: input checks first, then the reads, then the uploads, so a bad input or a failed read has + * no side effects. `otherWrites` counts the writes the caller adds to the same request. + * + * Throws TypeError for two new records with the same key or more records than one write holds, and + * what `readBeforeWriting` and `prepareNew` throw. + */ +export async function prepareSlots( + build: BuildContext, + roots: readonly Slot[], + { checkKeys, otherWrites = 0 }: { checkKeys: boolean; otherWrites?: number }, +): Promise { + const slots = flattenSlots(roots); + const fresh = newSlots(slots); + assertDistinctKeys(build, fresh); + assertWithinWriteLimit(build.operation, fresh.length + otherWrites); + const existing = await readBeforeWriting(build, slots, { checkKeys }); + const created = await prepareNew(slots, existing); + return { existing, created, refOf: referenceLookup(existing, created) }; +} + /** The reference of the record in `slot`, with whether this call created it. */ export function recordRef( slot: Slot, diff --git a/packages/sdk/src/organization/create-organization.ts b/packages/sdk/src/organization/create-organization.ts index 608a54d..7c34962 100644 --- a/packages/sdk/src/organization/create-organization.ts +++ b/packages/sdk/src/organization/create-organization.ts @@ -9,22 +9,14 @@ import { strictObject } from "../internal/input.js"; import { readRepoHead } from "../internal/read-repo-head.js"; import { readRecord, type StoredRecord, type StrongRef } from "../internal/resolve-record-ref.js"; import { - flattenSlots, type NewSlot, - newSlots, planNew, - prepareNew, - readBeforeWriting, + prepareSlots, recordRef, referencedResults, - referenceLookup, type SupportingRecordRef, } from "../internal/slots.js"; -import { - assertWithinWriteLimit, - RecordAlreadyExistsError, - submitWrites, -} from "../internal/submit-writes.js"; +import { RecordAlreadyExistsError, submitWrites } from "../internal/submit-writes.js"; import { type OrganizationInput, organizationKind } from "./build-organization.js"; const OPERATION = "createOrganization"; @@ -274,12 +266,13 @@ export async function createOrganization( const profilePlan = planProfile(build, start.profile, fallback); // A reused profile is not in the write, so its images are never uploaded. - const slots = flattenSlots( + // The organization and profile keys are `self`, and the organization was found absent above, so + // the keys are not checked again here. + const { created, refOf } = await prepareSlots( + build, profilePlan.kind === "created" ? [profilePlan.slot, organizationSlot] : [organizationSlot], + { checkKeys: false }, ); - assertWithinWriteLimit(OPERATION, newSlots(slots).length); - const existing = await readBeforeWriting(build, slots, { checkKeys: false }); - const created = await prepareNew(slots, existing); // The organization has a fixed key and was found absent above, so calling again after an unknown // outcome creates everything or finds the organization, and never makes a copy. @@ -289,7 +282,6 @@ export async function createOrganization( retrySafeReason: RETRY_SAFE_REASON, }); - const refOf = referenceLookup(existing, created); return { organization: refOf(organizationSlot), profile: