feat(references): add support for remote datasets

.claude/harness/architecture.md

@@ -0,0 +1,83 @@
+# JavaScript Content API Library — Architecture
+
+Reference description of the project, its layout, the core modules, the request flow, and the CaaS storage model it reads from. Referenced from the project `CLAUDE.md`.
+
+## Project overview
+
+**fsxa-api** (the JavaScript Content API Library, a.k.a. Content API) is a published npm library — not an application. It reads content from the FirstSpirit **CaaS** (Content-as-a-Service) and from the **Navigation Service**, and maps the raw CaaS JSON into a stable, consumer-facing shape for PWAs and other frontends. It ships as CommonJS + ES5 bundles plus type declarations, and is consumed by Crownpeak PWA templates and by customer projects.
+
+Because it is a library, its exported surface is the product. See "Public API surface" below.
+
+## Layout
+
+| Path | Description |
+|------|-------------|
+| `src/modules/` | The core: both API implementations, the mapper, the query builder, the event stream, logging |
+| `src/integrations/` | Adapters for hosting the proxy backend — the Express router and the framework-agnostic wrapper |
+| `src/types.ts` | Both sides of every mapping: `CaaSApi_*` (raw CaaS shapes) and the mapped consumer shapes |
+| `src/enums.ts` | Error message enums, content mode, proxy routes, HTTP status |
+| `src/routes.ts` | Proxy route paths and request body interfaces — the contract between proxy and remote side |
+| `src/testutils/` | Fixture factories used by unit tests (`createPageRef`, `createDataset`, …) |
+| `src/helpers/`, `src/utils.ts` | Small pure helpers (locale discovery, navigation-map pruning, regex validation, rich-text link merging) |
+| `integrationtests/` | Tests against a real CaaS tenant — see the testing guidelines |
+| `proxy/` | npm workspace publishing the proxy-only entry point |
+| `dev/` | Local scratch harness (`npm run dev`) for hitting a real CaaS by hand |
+| `docs/superpowers/plans/` | Implementation plans written by the `writing-plans` skill |
+
+## The two API implementations
+
+Both implement the same `FSXAApi` interface (`src/types.ts`) and expose `fetchElement`, `fetchByFilter`, `fetchNavigation`, `fetchProjectProperties`:
+
+- **`FSXARemoteApi`** (`src/modules/FSXARemoteApi.ts`) — talks to CaaS and the Navigation Service directly. It holds the API key, so it only ever runs server-side. It owns URL construction (`buildCaaSUrl`, `buildNavigationServiceUrl`), the remote project lookup (`getRemoteConfigById`, `verifyRemoteProjectExists`), and the config: `apikey`, `caasURL`, `navigationServiceURL`, `tenantID`, `projectID`, `contentMode`, `remotes`, `maxReferenceDepth`, `customMapper`, `navigationItemFilter`, `caasItemFilter`.
+- **`FSXAProxyApi`** (`src/modules/FSXAProxyApi.ts`) — same interface, but forwards each call as an HTTP POST to a backend that hosts an `FSXARemoteApi`. It carries no secrets and is the client-side implementation.
+
+`FSXAApiSingleton` holds one process-wide instance. The proxy backend is built with `src/integrations/express.ts` (`getExpressRouter`) or, for other frameworks, `useEndpointIntegrationWrapper` in `src/integrations/endpointIntegrationWrapper.ts`. The routes and body shapes both sides agree on live in `src/routes.ts`.
+
+**Consequence for any change to a fetch method:** a new parameter has to be threaded through `FSXAApi` (the interface), `FSXARemoteApi` (the implementation), `FSXAProxyApi` (serialize into the request body), `src/routes.ts` (the body interface), and the integration adapters (deserialize and validate). Changing only the remote side silently leaves proxy-mode consumers behind.
+
+## CaaSMapper and reference resolution
+
+`src/modules/CaaSMapper.ts` is the heart of the library and the file most work touches. It converts raw CaaS documents (`CaaSApi_PageRef`, `CaaSApi_Dataset`, `CaaSApi_Media`, …) into mapped items, walking the FirstSpirit document model: `PageRef` → `Page` → `Body` → `Section`, each carrying `formData` / `metaFormData` of typed input components (`CMS_INPUT_*`, `FS_REFERENCE`, `FS_DATASET`, `FS_INDEX`, `FS_CATALOG`, `CMS_INPUT_IMAGEMAP`, `Content2Section`).
+
+Reference resolution is **two-phase**, and understanding this is a prerequisite for editing the mapper:
+
+1. **Register.** While mapping, a reference is not fetched. `registerReference` / `registerReferencedItem` record the referenced id together with the path in the output object where the resolved item must later be placed, and mapping returns a placeholder string. One id can be registered at many paths.
+2. **Resolve.** `resolveAllReferences` walks the registered groups and calls `resolveReferencesForGroup` per group, which chunks ids (`REFERENCED_ITEMS_CHUNK_SIZE = 30`) and fetches them via `fetchByFilter`.
+3. **Denormalize.** `MappingUtils.denormalizeResolvedReferences` writes each fetched item into every path registered for it — or, in normalized mode, the caller receives `items` plus a flat `referenceMap` instead.
+
+Grouping is keyed by **(projectId, locale)** — `buildGroupKey` — because one CaaS filter query carries exactly one locale and one collection. Each disjoint pair therefore costs at least one request. `unifyId` namespaces ids as `projectId#uuid.locale` so items from different projects cannot collide in the cache or the reference map.
+
+`resolveReferenceTarget` decides where a reference is resolved. The **project** comes from the reference's own CaaS document URL (`src/modules/ReferenceUrlParser.ts`), falling back to the `remoteProject` field of a media reference and then to the surrounding document. The **locale** comes from the `remotes` entry configured for that project, or — with `useSourceLocale` — from `CaaSMapper.sourceLocale`, the locale the request was made in. The configuration is consulted before the own project id, so an entry naming the own project reads its references in another locale. A project that is neither the own one nor in `remotes` is not fetched: `registerUnresolvableReference` logs a warning and leaves the placeholder in the payload. See the README section "Resolving references across projects".
+
+Two guards bound the recursion: `maxReferenceDepth` (default `DEFAULT_MAX_REFERENCE_DEPTH = 2`) and `_processedItems`, which prevents re-fetching an id already handled.
+
+**Sharp edge:** `setLocaleFromCaasItem` mutates `this.locale` per mapped item, so it is only safe for `buildPreviewId`. Everything that keys or groups references uses the immutable `sourceLocale` instead. Do not reintroduce `this.locale` into `unifyId` or `resolveReferencesForGroup`.
+
+## Request flow (remote mode)
+
+1. The consumer calls `fetchElement` / `fetchByFilter` / `fetchNavigation` / `fetchProjectProperties`.
+2. `FSXARemoteApi` builds the URL (`buildCaaSUrl`) and, for filters, translates the query via `QueryBuilder` into the CaaS filter syntax; the requested locale becomes `locale.language` + `locale.country` filters.
+3. The response is handed to `CaaSMapper`, which maps documents and registers references (phase 1).
+4. References are resolved per group and denormalized into the result (phases 2 and 3).
+5. Optional hooks run: `customMapper` per data entry, `caasItemFilter` on mapped items, `navigationItemFilter` on navigation items.
+
+`CaaSEventStream` (lazily loaded via `CaaSEventStreamLazy` so `better-sse` stays out of browser bundles) is a separate path: it streams CaaS change events instead of fetching documents.
+
+## CaaS storage model
+
+A CaaS document URL is `baseURL / tenantID / collectionID / documentID`, for example:
+
+```
+https://enterprise-caas-api.e-spirit.cloud/enterprise-prod/3bb083df-446f-4cdd-af7e-514886c4dc20.preview.content/21f3109e-63b2-47e8-9728-5680c2decb02.en_US
+```
+
+- `collectionID` = `<project uuid>.<content mode>.content`
+- `documentID` = `<document uuid>.<locale>`
+
+Content lives in `preview.content` / `release.content`; binaries in `preview.files` / `release.files`. The **content mode never comes from content data** — it comes only from this library's configuration. Preserve that when touching URL construction or reference parsing.
+
+## Public API surface
+
+Everything re-exported from `src/index.ts` is published: the modules listed there, all of `src/enums.ts`, all of `src/types.ts`, the helpers, the exceptions, `ROUTES`, and the integration wrappers. A rename or a narrowed type in `src/types.ts` is a breaking change for consumers even if nothing inside `src/` notices.
+
+`src/modules/index.ts` re-exports in a **deliberate order** to break a circular-dependency cycle. Do not reorder those export lines.

.claude/harness/coding-guidelines.md

@@ -0,0 +1,101 @@
+# JavaScript Content API Library — Coding Guidelines
+
+Project-specific coding rules that complement the project overview in `CLAUDE.md`. Apply at write time, not only at review time.
+
+## Code style & quality
+
+- **Language**: TypeScript, `strict: true`, compiled to ES5 / CommonJS (`tsconfig.json`). No decorators in new code despite `experimentalDecorators` being on.
+- **Formatting**: Prettier with `semi: false`, `singleQuote: true` (config lives in `package.json`). Committed code uses the Prettier 2 trailing-comma default — see the formatting section in `CLAUDE.md` before running Prettier at all.
+- **Node**: `.nvmrc` pins 24.12.0; CI runs 24.x. `engines` still claims `>=14`, so do not use syntax or APIs newer than the declared floor in shipped code without raising `engines` deliberately.
+- **No linter**: there is no ESLint and no `lint` script. Type errors are the only automated style signal, so run `npx tsc --noEmit` yourself — `npm test` will not surface a type error in a file no test imports.
+- **Dependencies**: this is a library consumed in browsers. Prefer none. A new runtime dependency lands in every consumer's bundle and needs an explicit justification; heavy or server-only packages belong in `optionalDependencies` and must be lazily imported like `better-sse` is in `CaaSEventStreamLazy.ts`.
+
+## Never break the published surface
+
+**Rule:** Treat every symbol re-exported from `src/index.ts` — including every type in `src/types.ts` — as a public contract. Additive changes only: new optional fields, new optional parameters, widened return types. When behavior must change or a name must go, deprecate instead of removing.
+
+This library is published to npm and consumed by PWA templates and customer projects that upgrade on their own schedule. A removed field or a narrowed type breaks builds in repositories you cannot see or fix. Removing something is also a release decision, not just a code decision: `release-it` reads Conventional Commits, so a `BREAKING CHANGE:` footer ships a major version.
+
+**Smell:** renaming a field in a `CaaSApi_*` or mapped type "because nothing in `src/` uses the old name" — `src/` is not the consumer.
+
+### How to deprecate
+
+1. Keep the old symbol working, and keep it exported.
+2. Add a `@deprecated` JSDoc tag naming the replacement and the README section that explains the migration.
+3. Document the change in `README.md` under a stable heading (update notices link to those anchors).
+4. Carry the deprecated path for at least one release before proposing removal — and propose it, do not do it unasked.
+
+If a behavior change is unavoidable even while the old API keeps compiling (a value now derived from a different source, a config field now ignored), that is still a behavior change for existing applications: document it explicitly as such, not just in the changelog.
+
+## Content data must never decide where a request goes
+
+**Rule:** No value read from a CaaS document may determine the host, tenant, content mode, or locale of an outbound request that carries the API key. A URL found in content may supply one thing: the id of the project a reference points at — and only after that id has been found in the `remotes` configuration.
+
+`FSXARemoteApi` holds the CaaS API key and runs server-side. A URL inside a document is editor-controlled data. If it selected the target host, a crafted document would make the server send its API key to an attacker's endpoint — a textbook SSRF with credential leak. The `remotes` allowlist is what prevents that: every request url is assembled from `caasURL`, `tenantID`, the configured project id and the configured content mode. The locale is excluded for a different reason — an editor cannot control which locale a reference url carries, so following it produces requests for documents that do not exist.
+
+**Smell:** `fetch(reference.url, { headers: { Authorization: apikey } })`, passing a parsed `baseUrl` into `buildCaaSUrl`, or reading `ParsedReferenceUrl.locale` to decide what to fetch.
+
+### Checklist when adding anything URL-derived
+
+1. Parse with `ReferenceUrlParser` — do not hand-roll string splitting on CaaS URLs, and do not duplicate the layout knowledge it owns.
+2. Take the project id from it and nothing else. Locale comes from `remotes`, content mode and host from the api configuration.
+3. Look the project id up with `getRemoteConfigById` before anything is fetched.
+4. On a project that is not configured, or an unparsable URL, log a warning and leave the reference unresolved — its placeholder stays in the payload. Do not fall back to "try it anyway".
+
+## Mirror the change across parallel siblings
+
+**Rule:** When you touch a method, branch, type, or route — whether fixing a bug, refactoring, or cleaning up — immediately scan the parallel siblings with the same pattern, test files included. If the same change on a sibling is structurally identical, include it in the same pass without asking, and flag it in the commit/PR message.
+
+This codebase is built from mirrored pairs and families. Fixing one member ships the same bug under a different name in the others.
+
+The sibling groups to check:
+
+1. **`FSXARemoteApi` ↔ `FSXAProxyApi`** — both implement `FSXAApi`. A changed parameter or behavior almost always needs the proxy counterpart, plus the body interface in `src/routes.ts`, plus the Express router in `src/integrations/express.ts` and `endpointIntegrationWrapper.ts`, plus `parameterValidation.ts`.
+2. **The `mapDataEntry` component branches** — `FS_REFERENCE`, `FS_DATASET`, `FS_INDEX`, `FS_CATALOG`, `CMS_INPUT_IMAGEMAP` all register references; a fix to one registration site usually applies to the others.
+3. **The `map*` family** — `mapPageRef`, `mapDataset`, `mapGCAPage`, `mapMedia*`, `mapProjectProperties`.
+4. **Raw ↔ mapped types** — a new field on a `CaaSApi_*` type usually needs its counterpart on the mapped type, and vice versa.
+5. **preview ↔ release** content modes.
+
+**Scope:** the sweep covers every file changed on the current branch, committed and uncommitted alike. Do not leave a half-applied edit in the working tree.
+
+A structurally identical mirror change is bounded — apply it inline. A mirror that needs real per-sibling design judgment is separate scope — name it and leave it, rather than silently widening the diff.
+
+## Register-then-resolve: never fetch during mapping
+
+**Rule:** Mapping code (`mapDataEntry`, `map*`) must not issue a fetch for a referenced item. Register the reference and return the placeholder; let `resolveAllReferences` do the I/O.
+
+The two-phase design is what makes reference loading batched: one request per (projectId, locale) group of up to 30 ids instead of one request per reference. A single `await api.fetchElement(...)` inside a mapping branch turns a page with 50 images into 50 sequential round-trips, and it bypasses both `_processedItems` dedup and the `maxReferenceDepth` guard.
+
+**Smell:** an `await this.api.fetch…` inside a `case 'FS_…'` branch, or building a result object that already contains a resolved item rather than the id returned by `registerReferencedItem`.
+
+### Checklist when adding a new referencing component type
+
+1. Add the branch to `mapDataEntry` and register via `registerReference` with the reference's own CaaS URL, so the project is derived, not guessed.
+2. Return the placeholder `registerReference` gives you. It always returns one, including for a project that is not configured — that reference is deliberately left unresolved rather than dropped.
+3. Confirm the new id ends up in the right group: assert on the group key and on the resulting `fetchByFilter` calls, not just on the mapped output.
+4. Add the raw shape to `src/types.ts` and a factory to `src/testutils/`.
+
+## Comments must add information not derivable from the code
+
+**Rule:** Default to writing no comments. A comment is only justified when it adds information not derivable from the code under it — a hidden constraint, a workaround for a specific bug, a non-obvious invariant, behavior that would surprise a reader. If unsure whether a comment adds information: delete it.
+
+This applies at write time and at all times after — write no useless comment in the first place, and remove any useless comment you come across, even one you just wrote. Comment hygiene is never deferred to a cleanup pass. It applies to test code exactly as to production code.
+
+Never write comments that:
+
+1. Restate what the code does — a well-named identifier, signature, or return type already says it.
+2. Reference the current task, ticket, or caller ("added for CAAS-123", "used by the proxy") — that belongs in the commit message and the PR description.
+3. Mark removed code (`// removed foo`).
+4. Explain a parameter whose meaning is obvious from its type and name.
+
+**Judge against the code, not a snippet:** evaluate a comment by reading it together with the full declaration and body it sits on, never from a `git diff` hunk or a `grep` match in isolation.
+
+Conversely, keep comments that encode a decision the code cannot show. This repo has several that earn their place: why `src/modules/index.ts` fixes its export order (circular dependency), why `CaaSEventStream` is loaded lazily (browser bundle size), why a reference URL is only trusted after an origin check. Documentation of the public API is different in kind — JSDoc on exported symbols is consumer-facing and welcome.
+
+## Errors: reuse the enums, stay actionable
+
+**Rule:** User-facing failure messages belong in `FSXAApiErrors` / `CaaSMapperErrors` / `QueryBuilderErrors`, not inline in a `throw`. Before adding a case, check whether one already covers it.
+
+Consumers assert on these strings and match on them in their own error handling, which makes the enum values part of the public surface — editing an existing message is a breaking change for someone. Add a new member instead.
+
+Choose deliberately between throwing and warning. A misconfiguration the consumer must fix (missing API key, invalid locale) throws at construction or call time. Bad *content* — an unparsable reference URL, a broken reference, a component the mapper does not know — must not take down a page render: log through `this._logger` / `this.logger` and degrade, as the mapper already does for dropped references. Never use bare `console.log`; the `Logger` respects the configured `logLevel`.