From 1a4bedea1d14acc4df0a9cde93f0bae5afbeb9fe Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Mon, 22 Jun 2026 11:41:24 +0100 Subject: [PATCH 01/15] RFC 090: Axiell-FOLIO Sync Architecture - Proposes Step Functions + Lambda + S3 architecture - Details orchestration, storage, invocation design choices - Compares alternatives with trade-off analysis - Includes implementation details from working prototype - Outlines future pipelining strategy for scale - Cost estimates for current and future volumes --- rfcs/090-axiell-folio-sync/README.md | 672 +++++++++++++++++++++++++++ 1 file changed, 672 insertions(+) create mode 100644 rfcs/090-axiell-folio-sync/README.md diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md new file mode 100644 index 00000000..ccb00f75 --- /dev/null +++ b/rfcs/090-axiell-folio-sync/README.md @@ -0,0 +1,672 @@ +# RFC 090: Axiell-FOLIO Sync Architecture + +## Purpose + +Design a system to synchronize library location and item holdings from Axiell (legacy adapter) into FOLIO (new system of record) on a 15-minute cadence, with strict requirements for idempotency, auditability, and graceful error isolation. + +**Last modified:** 2026-06-22 + +--- + +## Background + +### Current State +- **Axiell Adapter Platform**: Runs on a 15-minute cadence, emitting changesets with new/modified/deleted records +- **FOLIO**: The new system of record for library data (items, holdings, locations) +- **Integration Gap**: Changesets are written to Apache Iceberg tables on S3, but FOLIO is not automatically updated +- **Volume**: ~10–500 records per changeset (15-minute window); typical day: ~1,000 records across 80 syncs + +### Why This Matters +- Library staff rely on FOLIO for real-time item availability and location data +- Axiell-to-FOLIO sync is a core data pipeline for the new catalogue system +- Sync must be reliable, auditable, and manually recoverable + +--- + +## Problem Statement + +How do we build a system that: + +1. **Reliably syncs incremental changesets** (10–500 records per 15-min window) from Axiell to FOLIO? +2. **Handles failures gracefully**: One bad record (mapping error, API failure) must not halt processing of others? +3. **Maintains a complete audit trail**: Every record must be traceable (created, updated, suppressed, failed)? +4. **Enables safe replay**: Failed changesets must be replayable without causing duplicates or data corruption? +5. **Keeps costs predictable**: For ~1K records/day, infrastructure cost should be minimal (~$5/month)? + +--- + +## Executive Summary: Key Design Decisions + +We propose a **Step Functions + Lambda + S3 architecture** with synchronous invocation, per-record error isolation, and 90-day audit retention. This approach balances **operational visibility**, **fault resilience**, and **simplicity**. + +| Aspect | Choice | Primary Benefit | +|--------|--------|-----------------| +| **Orchestration** | AWS Step Functions | Configurable retries + rich execution history for audit | +| **Compute** | Lambda (ECR container) | Stateless, scales to 0, integrates with Step Functions | +| **Data Storage** | S3 NDJSON manifests (90-day TTL) | Cost-efficient batch writes + queryable via S3 Select | +| **Trigger** | EventBridge on adapter completion | Event-driven (not polling); decoupled from adapter | +| **Invocation** | Synchronous Lambda (via Step Function) | Natural backpressure; clear visibility on success/failure | +| **Error Handling** | Per-record isolation | Batch completes even if individual records fail | + +**Expected outcomes:** +- Safe replay without data corruption (idempotent upserts via FOLIO HRIDs) +- Complete audit trail: every decision (create/update/suppress/skip) logged in S3 + CloudWatch +- Low operational overhead: ~$3–5/month for typical volume +- Clear mental model: no eventual consistency puzzles, ordered execution + +--- + +## Architecture: Current Implementation + +### System Diagram + +``` +┌──────────────────────────────────────────────────────────────────────────┐ +│ Axiell Adapter Platform (15-minute cadence) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ • Writes changesets to Iceberg table (namespace, id, content, deleted) │ +│ • Emits EventBridge event: axiell.adapter.completed │ +│ { changeset_ids: [...], job_id: "xyz", transformer_type: "axiell" } │ +└────────────────────┬─────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────────┐ +│ EventBridge Rule: axiell-folio-sync-axiell-adapter-completed │ +│ • Pattern: source="axiell.adapter" + transformer_type="axiell" │ +│ • Target: Step Function (synchronous invocation) │ +└────────────────────┬─────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────────┐ +│ Step Function: axiell-folio-sync-sfn │ +│ • Passes event detail to Lambda │ +│ • Retry policy: max 3 attempts, exponential backoff (2s, 4s, 8s) │ +│ • Error handling: Catch-all → SyncFailed (terminal state) │ +│ • Execution history: CloudWatch Logs (30-day retention) │ +└────────────────────┬─────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────────┐ +│ Lambda: axiell-folio-sync (ECR image) │ +│ • Timeout: 300s (5 min) — handles ~200 records with 3 API calls each │ +│ • Memory: 1024 MB │ +│ │ +│ Execution Flow: │ +│ 1. Auth: Fetch OKAPI credentials from SSM, POST /authn/login │ +│ 2. Scan: Query Iceberg for changesets (WHERE changeset_id IN [...]) │ +│ 3. Map & Validate: Apply YAML rules, build Instance/Holdings/Item │ +│ 4. Upsert: For each record, POST/PUT to FOLIO (per-record errors) │ +│ 5. Write Manifests: Flush to S3 NDJSON (success, errors, metadata) │ +│ │ +│ Environment Variables (injected by Terraform): │ +│ • OKAPI_URL, OKAPI_TENANT, OKAPI_SECRET_PARAM │ +│ • MANIFEST_S3_BUCKET, S3_TABLE_BUCKET_ARN, ICEBERG_TABLE_NAME │ +│ • AWS_REGION, DRY_RUN (override via event) │ +└────────────────────┬─────────────────────────────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────────────────────────────────┐ +│ S3: Manifest Storage (axiell-folio-sync-manifests-{account}-{region}) │ +│ │ +│ Files (per job_id): │ +│ • {job_id}.ids.ndjson — success records (5K/batch) │ +│ • {job_id}.ids.failures.ndjson — error records (if any) │ +│ • {job_id}.manifest.json — metadata summary │ +│ │ +│ Lifecycle: 90-day expiration (auto-delete old manifests) │ +│ Versioning: Enabled (audit trail for object changes) │ +│ Encryption: SSE-S3 (default) │ +└──────────────────────────────────────────────────────────────────────────┘ +``` + +### 1. EventBridge Trigger + +**Rule**: `axiell-folio-sync-axiell-adapter-completed` + +**Event Pattern**: +```json +{ + "source": ["axiell.adapter"], + "detail-type": ["axiell.adapter.completed"], + "detail": { + "transformer_type": ["axiell"] + } +} +``` + +**Event Payload** (emitted by Axiell adapter): +```json +{ + "changeset_ids": ["axiell-cs-20260622-001", "axiell-cs-20260622-002"], + "job_id": "adapter-job-xyz-12345", + "transformer_type": "axiell", + "dry_run": false, + "sample_limit": null +} +``` + +**IAM Permissions**: `states:StartExecution` on Step Function ARN + +--- + +### 2. Step Function State Machine + +**Name**: `axiell-folio-sync-sfn` +**Type**: STANDARD + +**Flow**: +``` +Input (from EventBridge) + ↓ +[Task] Invoke Lambda (pass event detail) + ├─ Input path: $.detail + ├─ Output: result JSON with counts, manifest URIs, errors + │ + ├─ [Retry] + │ • MaxAttempts: 3 (configurable) + │ • BackoffRate: 2.0 + │ • IntervalSeconds: 2 + │ + └─ [Catch] + • States.TaskFailed → SyncFailed (terminal) + • Log to CloudWatch + ↓ +Output (Success) → SyncComplete +``` + +**Execution History**: Stored in CloudWatch Logs (`/aws/states/axiell-folio-sync-sfn`, 30-day retention) + +**IAM Role Permissions**: +- `lambda:InvokeFunction` on sync Lambda ARN +- `logs:CreateLogGroup`, `logs:CreateLogStream`, `logs:PutLogEvents` for CloudWatch + +--- + +### 3. Lambda Function: axiell-folio-sync + +**Docker Image**: ECR (`uk.ac.wellcome/axiell-folio-sync:TAG`) + +**Execution Steps**: + +#### Step 1: Authenticate with FOLIO +``` +SSM Parameter Store (SecureString) + Path: /axiell-folio-sync/okapi-creds + Value: {"username": "service_account", "password": "..."} + +POST /authn/login + x-okapi-tenant: wellcome + Body: credentials + Response: x-okapi-token (valid 24h) + → Reused for all subsequent API calls in this invocation +``` + +#### Step 2: Scan Iceberg Changesets +``` +Query S3 Tables Iceberg catalog + Bucket: {S3_TABLE_BUCKET_ARN} + Table: {ICEBERG_TABLE_NAME} (e.g., default.axiell_changesets) + +SELECT [namespace, id, content, changeset, last_modified, deleted] +WHERE changeset IN ({changeset_ids from event}) + +Schema: + namespace string — e.g., "location", "item" + id string — external identifier from Axiell + content string — JSON-serialized record + changeset string — changeset ID from adapter + last_modified timestamp — when record changed + deleted boolean — true if marked deleted +``` + +#### Step 3: Map & Validate +```python +# Load YAML mapping rules (from mapping.yaml) +mapper = YamlMapper("mapping.yaml") + +for row in iceberg_records: + try: + record = json.loads(row['content']) + + # Build Instance, Holdings, Item payloads + instance_payload = mapper.build_instance_payload(record) + holdings_payload = mapper.build_holdings_payload(record) + item_payload = mapper.build_item_payload(record) + + # Validate required fields per FOLIO schema + payloads.append({...}) + except MappingError as e: + # Capture error, continue to next record + errors.append({ + "source_id": row['id'], + "type": "mapping", + "detail": str(e) + }) + continue +``` + +#### Step 4: Upsert to FOLIO (Per Record) +``` +For each record: + Execute: Instance → Holdings → Item (dependencies) + + Instance: + If deleted=true: + GET /inventory/instances/{instance_id} + PUT with discoverySuppress=true + Action: "suppress" + Else: + GET /inventory/instances?query=(hrid=={hrid}) + If exists: PUT (update) + Else: POST (create) + Action: "create" or "update" + + Holdings (if Instance succeeded): + POST /holdings-storage/holdings (create linked to Instance) + Action: "create" + + Item (if Holdings succeeded): + POST /item-storage/items (create linked to Holdings) + Action: "create" + +Per-record error handling: + • HTTP 4xx/5xx → captured, batch continues + • All errors logged to CloudWatch AND accumulated +``` + +#### Step 5: Batch & Write Manifests +```python +# Success NDJSON (one record per line, 5K records per batch) +for i in range(0, len(success_results), 5000): + batch = success_results[i:i+5000] + ndjson_lines = [json.dumps(r) for r in batch] + s3.put_object( + Bucket=MANIFEST_S3_BUCKET, + Key=f"manifests/{job_id}.ids.ndjson", + Body="\n".join(ndjson_lines) + ) + +# Error NDJSON (one error per line) +if errors: + error_lines = [json.dumps(e) for e in errors] + s3.put_object( + Bucket=MANIFEST_S3_BUCKET, + Key=f"manifests/{job_id}.ids.failures.ndjson", + Body="\n".join(error_lines) + ) + +# Metadata summary +metadata = { + "job_id": job_id, + "start_time": "2026-06-22T10:15:03Z", + "end_time": "2026-06-22T10:15:45Z", + "status": "SUCCESS|PARTIAL|FAILED", + "counts": { + "total": 247, + "created": 10, + "updated": 200, + "suppressed": 25, + "skipped": 12, + "failed": 0 + }, + "success_manifest_uri": f"s3://{bucket}/manifests/{job_id}.ids.ndjson", + "error_manifest_uri": f"s3://{bucket}/manifests/{job_id}.ids.failures.ndjson" +} +s3.put_object( + Bucket=MANIFEST_S3_BUCKET, + Key=f"manifests/{job_id}.manifest.json", + Body=json.dumps(metadata, indent=2) +) + +# CloudWatch Metrics +cloudwatch.put_metric_data( + Namespace="AxiellFolioSync", + MetricData=[ + {"MetricName": "RecordsCreated", "Value": metadata['counts']['created']}, + {"MetricName": "RecordsUpdated", "Value": metadata['counts']['updated']}, + {"MetricName": "RecordsSuppressed", "Value": metadata['counts']['suppressed']}, + {"MetricName": "RecordsFailed", "Value": metadata['counts']['failed']}, + ] +) + +return metadata +``` + +--- + +### 4. S3 Manifest Storage + +**Bucket**: `axiell-folio-sync-manifests-{account-id}-{region}` + +**File Structure**: +``` +manifests/ + ├─ adapter-job-xyz-20260622-10-15.ids.ndjson + ├─ adapter-job-xyz-20260622-10-15.ids.failures.ndjson + ├─ adapter-job-xyz-20260622-10-15.manifest.json + ├─ adapter-job-xyz-20260622-10-30.ids.ndjson + └─ ... +``` + +**Sample Success Record** (NDJSON): +```json +{ + "source_id": "location_item_123", + "instance": { + "action": "update", + "id": "inst-abc-def", + "hrid": "HRID-LOC-123" + }, + "holdings": { + "action": "create", + "id": "hold-xyz-uvw" + }, + "item": { + "action": "create", + "id": "item-001-002" + }, + "errors": [] +} +``` + +**Sample Error Record** (NDJSON): +```json +{ + "source_id": "location_item_456", + "error": "Mapping error: required field 'barcode' missing", + "detail": "YAML rule for barcode returned null", + "type": "mapping" +} +``` + +**Lifecycle Policy**: 90-day expiration (auto-delete old manifests) + +--- + +## Design Rationale: Why These Choices? + +### Orchestration: Step Functions vs. Alternatives + +| Approach | Pros | Cons | Cost/mo | +|----------|------|------|---------| +| **A: Direct Lambda** (EventBridge → Lambda, fire-and-forget) | Fewer services, lower overhead, fastest time-to-invocation | No built-in retry logic, no execution history, hard to extend, silent failures possible | ~$2 | +| **B: Step Functions + Lambda** ⭐ **CHOSEN** | Explicit retry policy + audit trail, easy to parallelize later, clear visibility into success/failure | Extra service, marginal latency added | ~$3–5 | +| **C: Async Queue** (EventBridge → SQS → Lambda workers) | Decouples producer/consumer, handles bursts, resilient to adapter restarts | Complex ordering semantics, harder to reason about, no clear success/failure signal, monitoring overhead | ~$10–15 | + +**Why we chose B (Step Functions + Lambda):** + +1. **Extension point for scale**: When volume grows beyond 500 records/event, we swap the Lambda invocation for a Step Function Map state (fan-out per changeset). Same manifest format, no breaking changes. + +2. **Execution history = audit trail**: Every state transition is logged to CloudWatch. If a sync fails partway through, we know exactly where and why. No need to rebuild state from Lambda logs. + +3. **Explicit retry policy**: Max attempts and backoff are declarative. Operations can tweak `max_retries` without code changes (configure in Step Function definition). + +4. **Synchronous invocation provides backpressure**: Adapter waits for sync to complete → prevents queue buildup if FOLIO is slow. If sync times out, adapter knows to retry. + +5. **Cost remains minimal**: At 1K records/day (~80 invocations), we're talking $0.50/month for Step Function state transitions. Not a material driver. + +**Why not A (Direct Lambda)?** +- Silent failures: EventBridge doesn't tell us if the invocation succeeded or failed. +- No retry: If Lambda times out, there's no automatic retry. We'd have to implement our own (add complexity). +- No execution history: Debugging a failed sync means parsing Lambda logs. Step Function history is cleaner. + +**Why not C (Async Queue)?** +- Ordering: SQS doesn't guarantee processing order within a batch. We'd need to handle out-of-order upserts or add logic to sort before processing. +- Complexity: Workers need to coordinate (who's processing which changeset?). Adds operational burden. +- Cost: For current volume, async queue is cheaper per invocation, but the monitoring overhead (error tracking, retry logic, dead-letter queue management) makes it not worth it. + +--- + +### Storage: S3 NDJSON vs. Alternatives + +| Approach | Pros | Cons | Cost/mo (90-day) | +|----------|------|------|------------------| +| **A: DynamoDB** (per-record writes, TTL) | Query-friendly, real-time dashboards possible, strong consistency | O(n) write cost, schema evolution painful, expensive for batches, not auditable | ~$20–50 | +| **B: S3 NDJSON** ⭐ **CHOSEN** | Batched writes (low cost), queryable (S3 Select), matches org patterns, audit via versioning, easy to compress/archive | Requires S3 Select for queries (not real-time), not ideal for high-cardinality point lookups | ~$1–2 | +| **C: Streaming** (Kinesis/Firehose → Parquet) | High throughput, good for ML pipelines, efficient compression | Overkill for current volume, adds infrastructure complexity, higher cost | ~$5–15 | + +**Why we chose B (S3 NDJSON):** + +1. **Cost scales with volume**: Per-record writes scale O(n). At 5K records/day, you'd pay for 5K DynamoDB writes. With S3 NDJSON, we batch-write every 5K records → 1 PUT per 5K records. Cost dominance flips at ~100 records/day (already exceeded). + +2. **Matches existing patterns**: Axiell adapter platform already uses S3 for changesets. This keeps the operational model consistent (known backup, recovery, and query patterns). + +3. **Queryable post-hoc**: S3 Select allows SQL-like queries without ETL: + ```sql + SELECT * FROM s3://bucket/manifest.ndjson WHERE errors > 0 + ``` + Takes ~500ms on 100 KB file. Good enough for operational investigation. + +4. **Audit trail via versioning**: S3 versioning enabled → every write creates a new object version. If someone accidentally deletes a manifest, we can recover it. Metadata JSON tracks all historical jobs. + +5. **Easy to compress/archive**: After 90 days, we can archive NDJSON to Glacier for long-term audit retention (pennies/month). DynamoDB has no cheap archive option. + +**Why not A (DynamoDB)?** +- Per-record cost: 5K records/day = 5K write units (on-demand) = ~$2.50/day = ~$75/month. S3 is ~$0.05/month for the same data. +- Schema evolution: If we add a new field to the record (e.g., `retry_count`), DynamoDB requires a scan-and-rewrite. S3 NDJSON: just add the field to new records. +- Not suitable for audit: While DynamoDB TTL works, there's no clean way to query "what was deleted from this table last week?" + +**Why not C (Streaming)?** +- Overkill: Typical changeset is 10–500 records. Kinesis + Firehose is designed for high-throughput (1000s records/sec). We'd be paying for infrastructure we don't use. +- Operational complexity: Firehose auto-flushes based on buffer size or timeout. Extra operational knowledge needed. + +--- + +### Invocation Pattern: Synchronous vs. Asynchronous + +| Approach | Pros | Cons | +|----------|------|------| +| **A: Async** (EventBridge → Lambda, fire-and-forget) | Decoupled, adapter doesn't wait, low latency | No backpressure; if adapter emits 10 events in quick succession, Lambda might queue up (can overwhelm FOLIO); no failure signal | +| **B: Synchronous** ⭐ **CHOSEN** | Backpressure (adapter waits), clear success/failure, enables replay on failure | Adapter must wait ~45 sec for sync to complete before next event; if FOLIO is slow, adapter is blocked | + +**Why we chose B (Synchronous):** + +1. **Natural backpressure**: If FOLIO is slow or unavailable, Step Function times out → adapter pauses before emitting next event. No need for queue management. + +2. **Failure visibility**: If sync fails, EventBridge knows → can retry the same changeset. No silent failures. + +3. **Replay support**: If sync fails partway (e.g., 100 of 200 records succeeded before timeout), we can: + - Query S3 manifest to see which records failed + - Fix the issue (e.g., restart FOLIO, fix mapping rule) + - Re-trigger sync with same changeset_ids + - Idempotent upserts (via FOLIO HRID lookup) prevent duplication + +4. **Adapter is designed for it**: Axiell adapter runs on 15-min cycles. Waiting 45 sec for sync fits within the cadence. + +**Why not A (Async)?** +- Queue buildup: If adapter emits 10 changesets in quick succession, and each sync takes 45 sec, we'd have a 7-minute backlog. Async doesn't naturally handle this (we'd need manual queue monitoring). +- Failure not visible: If a sync fails, the adapter doesn't know. No built-in retry or alerting. + +--- + +### Error Handling: Per-Record Isolation + +**Policy**: One bad record must not halt the entire batch. + +**Error Levels**: + +1. **Per-record errors** (non-blocking): + - Mapping error (YAML validation fails): `{"source_id": "...", "type": "mapping", "detail": "..."}` + - API error (FOLIO returns 4xx/5xx): `{"source_id": "...", "type": "api", "detail": "..."}` + - Action: Log to S3 error manifest, continue to next record + +2. **Batch-level errors** (blocking): + - Auth failure (OKAPI login fails): Exception raised → Step Function retries up to `max_retries` + - Iceberg connection failure: Exception raised → Step Function retries + - S3 write failure: Exception raised → Step Function retries + - Action: If retries exhausted → SyncFailed (terminal state), alert operations + +**Observable via**: +- **CloudWatch Logs**: `/aws/lambda/axiell-folio-sync` (detailed per-record execution) +- **S3 Manifests**: `.ids.failures.ndjson` (queryable error list via S3 Select) +- **CloudWatch Metrics**: `RecordsFailed`, `RecordsCreated`, etc. (can alert on high failure rate) +- **Step Function History**: `/aws/states/axiell-folio-sync-sfn` (state transitions, retry events) + +--- + +## Testing & Operations + +### Test Levels + +**Unit Tests** (in `tests/`): +- YAML mapper applies rules correctly to test records +- Upsert logic handles create/update/suppress/error scenarios +- Manifest NDJSON serialization and metadata structure + +**Contract Tests** (against mock FOLIO): +- `dry_run=true`: Lambda computes changes but makes no writes +- Validates payload shape, required fields, API call sequence (no actual HTTP calls) + +**Integration Tests** (against dev FOLIO): +- Create test changesets in Iceberg +- Trigger sync via EventBridge +- Verify records appear in FOLIO with expected fields +- Cleanup: suppress test records + +**Smoke Tests** (pre-deployment): +- CloudFormation drift: no manual infrastructure changes +- Secrets: OKAPI credentials set in SSM +- EventBridge rule: active and routable to Step Function +- Lambda image: exists in ECR and is accessible +- Iceberg: S3 bucket and table readable + +### Operational Runbooks + +**Replay Failed Changesets**: +``` +1. Identify failed changeset_id from CloudWatch or S3 manifest +2. Emit EventBridge event with same changeset_ids +3. Monitor S3 manifest and CloudWatch to confirm success +``` + +**Manual Inspection**: +``` +# Query S3 manifest for records with errors +aws s3api select-object-content \ + --bucket axiell-folio-sync-manifests-123456-eu-west-1 \ + --key manifests/{job_id}.ids.failures.ndjson \ + --expression "SELECT * FROM s3object WHERE type = 'api'" \ + --expression-type SQL \ + --input-serialization '{"JSON": {}}' \ + --output-serialization '{"JSON": {}}' \ + output.json +``` + +**Emergency Shutdown**: +``` +# Disable EventBridge rule (stops new syncs) +aws events disable-rule --name axiell-folio-sync-axiell-adapter-completed + +# Monitor in-flight executions via Step Function console +# When all complete, investigate root cause +# Re-enable when ready +aws events enable-rule --name axiell-folio-sync-axiell-adapter-completed +``` + +--- + +## Future Scalability: Pipelining (When >500 Records/Event) + +### Current Limits +- **Lambda timeout**: 300 seconds (5 min) +- **Typical performance**: ~200 records (with 3 API calls per record) in 45 sec +- **Token lifetime**: 24 hours (no refresh needed within 5-min window) + +### Growth Trigger +When we observe >500 records per event (i.e., >80 records per changeset on average), parallelization becomes valuable: + +**Baseline**: 1 Lambda processes 1 event → 200 records/45s → 1,600 records/hour +**Limit**: 5 min timeout → ~280 records/event max (not good) + +**With parallelization**: 4 Lambda workers → 800 records/45s → 6,400 records/hour + +### Proposed Pipelining Architecture + +``` +Step Function: Map state (fan-out per changeset) + ├─ [Task] Fetch/cache OKAPI token (ElastiCache or in-memory) + │ + ├─ [Map] Parallel workers (one per changeset) + │ ├─ Worker 1: Process changeset 1 (reuse cached token) + │ ├─ Worker 2: Process changeset 2 + │ ├─ Worker 3: Process changeset 3 + │ └─ [... N workers in parallel] + │ + └─ [Join] Aggregate manifests + • Collect all success/error manifests from workers + • Write single metadata JSON (summarizing all workers) + • Emit CloudWatch metrics (aggregated counts) +``` + +**Key Optimization: Token Caching** +- Current: Each Lambda auth calls OKAPI → O(N changesets) auth calls +- Future: Fetch token once, cache in ElastiCache or Lambda shared memory → O(1) auth calls + +**Same Manifest Format** +- No breaking changes: Workers still write NDJSON to S3 +- Single metadata JSON consolidates all results +- Queries remain unchanged (S3 Select can union multiple manifest files) + +**Configuration Change**: +```terraform +# In Step Function definition, swap single Lambda invocation for Map state +# Lambda code changes: add token_cache parameter +# No changes to manifest format, API schema, or operational procedures +``` + +**Timeline**: Evaluate when we consistently see >500 records/event in production (track via CloudWatch metric `RecordsPerEvent`) + +--- + +## Cost Analysis + +### Current Volume: ~1,000 records/day + +| Service | Operation | Qty/mo | Rate | Cost/mo | +|---------|-----------|--------|------|---------| +| Lambda | 80 invocations × 300s × 1 GB | 80 invocations | $0.0000167/GB-s | ~$1.20 | +| Step Functions | 80 state transitions | 80 transitions | $0.000025/transition | ~$0.02 | +| EventBridge | 80 events | 80 events | $1/M events | ~$0.00 | +| S3 (manifests) | ~80 objects written, 90-day retention | 80 objects + storage | $0.023/K objects + $0.023/GB/mo | ~$1.50 | +| CloudWatch Logs | ~80 × 5 KB = 400 KB/month | 400 KB ingested | $0.50/GB ingested | ~$0.20 | +| **Total** | | | | **~$3–5** | + +### Future Volume: ~10,000 records/day (10x growth) + +| Service | Cost/mo | +|---------|---------| +| Lambda | ~$12 (10x invocations, 4x longer processing time due to size) | +| Step Functions | ~$0.20 (10x transitions) | +| S3 | ~$15 (more objects, higher storage) | +| CloudWatch Logs | ~$2 (10x log volume) | +| **Total** | **~$30–40** | + +**Note**: If >50 changesets/event, pipelining becomes cost-effective. With parallelization, Lambda compute cost would drop (faster wall-clock time → fewer seconds billed), but Step Function costs would increase (more state transitions per Map worker). + +--- + +## Assumptions & Constraints + +- **FOLIO HRID requirement**: Records must have a stable, unique HRID per type (Instance, Holdings, Item) for idempotent upserts. Axiell must provide this in mapping.yaml. +- **No real-time requirements**: 15-minute cadence is acceptable. If sub-minute sync becomes required, architecture must change (streaming Kinesis, real-time database). +- **FOLIO API stability**: Assumes FOLIO API is available and stable. If FOLIO is down for hours, sync manifests will accumulate (90-day retention allows manual replay after recovery). +- **Axiell stability**: Assumes Axiell adapter completes consistently. If adapter fails, no changeset is emitted → no sync triggered (not our problem, but should monitor). + +--- + +## Next Steps + +1. **Approval**: Review this RFC and confirm architectural choices align with Wellcome Collection platform strategy. +2. **Implementation**: Create Terraform modules, Lambda container image, Step Function definition, unit/integration tests. +3. **Deployment**: Stage in dev FOLIO → test end-to-end → deploy to prod with runbooks. +4. **Monitoring**: Set up CloudWatch dashboards, alarms on failure rate, manifest size growth. +5. **Feedback loop**: Monitor production for 1–2 months, adjust retry policy and timeout if needed. + +--- + +## References + +- **Axiell Adapter Platform**: Emits changesets to Iceberg; documentation in location-movement-control-docs +- **FOLIO API**: https://api-wellcome.folio.ebsco.com (OKAPI auth required) +- **AWS S3 Tables**: Iceberg catalog on S3; managed via Terraform +- **YAML Mapping Rules**: mapping.yaml (stored in Lambda container or S3) From a0859e431f430ed155c26915f5e7a232959800f0 Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Tue, 23 Jun 2026 07:26:17 +0100 Subject: [PATCH 02/15] CMS to LMS Sync RFC --- rfcs/090-axiell-folio-sync/README.md | 595 ++++++++++++------ .../calm-to-sierra-harvester-flow.md | 82 +++ .../folio-axc-fields-mapping.md | 70 +++ rfcs/090-axiell-folio-sync/mapping.yaml | 86 +++ 4 files changed, 634 insertions(+), 199 deletions(-) create mode 100644 rfcs/090-axiell-folio-sync/calm-to-sierra-harvester-flow.md create mode 100644 rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md create mode 100644 rfcs/090-axiell-folio-sync/mapping.yaml diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index ccb00f75..c76072c0 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -1,41 +1,252 @@ -# RFC 090: Axiell-FOLIO Sync Architecture +# RFC 090: CMS to LMS Sync ## Purpose -Design a system to synchronize library location and item holdings from Axiell (legacy adapter) into FOLIO (new system of record) on a 15-minute cadence, with strict requirements for idempotency, auditability, and graceful error isolation. +This is proposal to synchronize library location and item holdings from Axiell Collections(Content Management System) into FOLIO (Library Management system) with strict requirements for idempotency, auditability, and graceful error isolation. **Last modified:** 2026-06-22 +## Table of Contents + +- [Purpose](#purpose) +- [Background](#background) + - [Current CMS and LMS systems](#current-cms-and-lms-systems) + - [Migration Context](#migration-context) + - [Why This Matters](#why-this-matters) +- [System Architecture](#system-architecture) + - [Current Data Feeds for AxC and Folio Data](#current-data-feeds-for-axc-and-folio-data) + - [Key Characteristics](#key-characteristics) +- [Proposed Change: FOLIO Item Upsert on Every Adapter Run](#proposed-change-folio-item-upsert-on-every-adapter-run) + - [Fan-out Mechanism](#fan-out-mechanism) + - [Target Upsert/Mapping Architecture](#target-upsertemapping-architecture) +- [Transformation Design](#transformation-design) + - [Approach](#approach) + - [Transformation Pipeline](#transformation-pipeline) + - [YAML Mapper](#yaml-mapper-mappingyaml--yamlmapper) + - [Reference Data Cache](#reference-data-cache-ref_cachepy) +- [Proposed AWS Architecture](#proposed-aws-architecture) + - [Key Design Considerations](#key-design-considerations) + - [System Diagram](#system-diagram) + - [1. EventBridge Trigger](#1-eventbridge-trigger) + - [2. Step Function State Machine](#2-step-function-state-machine) + - [RefCache Resolution](#refcache-resolution) + - [Sample Field Mapping](#sample-field-mapping) + - [FOLIO Field Mapping Reference](#folio-field-mapping-reference) + - [Change Detection Mechanism](#change-detection-mechanism) + - [3. Lambda Function: axiell-folio-sync](#3-lambda-function-axiell-folio-sync) + - [Upsert Key Strategy (Idempotency)](#upsert-key-strategy-idempotency) + - [4. S3 Manifest Storage](#4-s3-manifest-storage) +- [Design Rationale: Why These Choices?](#design-rationale-why-these-choices) + - [Orchestration: Step Functions vs. Alternatives](#orchestration-step-functions-vs-alternatives) + - [Storage: S3 NDJSON vs. Alternatives](#storage-s3-ndjson-vs-alternatives) + - [Invocation Pattern: Synchronous vs. Asynchronous](#invocation-pattern-synchronous-vs-asynchronous) + - [Error Handling: Per-Record Isolation](#error-handling-per-record-isolation) +- [FOLIO API Client](#folio-api-client) +- [Cost Analysis](#cost-analysis) +- [Assumptions & Constraints](#assumptions--constraints) +- [Open Questions](#open-questions) +- [Next Steps](#next-steps) +- [References](#references) + + --- ## Background -### Current State -- **Axiell Adapter Platform**: Runs on a 15-minute cadence, emitting changesets with new/modified/deleted records -- **FOLIO**: The new system of record for library data (items, holdings, locations) -- **Integration Gap**: Changesets are written to Apache Iceberg tables on S3, but FOLIO is not automatically updated -- **Volume**: ~10–500 records per changeset (15-minute window); typical day: ~1,000 records across 80 syncs +## Current CMS and LMS systems +The Wellcome Collection library systems currently operate as: +- **CALM** (Collections Management): Content Management System (CMS) — bibliographic and collection metadata source of truth +- **Sierra**: Library Management System (LMS) — patron management, circulation, holds, and requesting + +The CALM-to-Sierra harvester is the integration mechanism that synchronizes bibliographic metadata from CALM into Sierra for discovery and requesting workflows. + +## Migration Context + The library systems migration is replacing these systems: +- **CALM → Axiell Collections**: Data from Calm is being migrated to Axiell Collections +- **Sierra → FOLIO**: Patron management, circulation, and requesting migrating to FOLIO + +The current CALM-to-Sierra harvester will be superseded by an Axiell Collections-to-FOLIO integration pipeline. This document captures the existing harvester behavior for reference during the migration period. The data needs to be synched between the CMS and LMS so that the circulation of items and patron managment can be carried out. ### Why This Matters - Library staff rely on FOLIO for real-time item availability and location data - Axiell-to-FOLIO sync is a core data pipeline for the new catalogue system - Sync must be reliable, auditable, and manually recoverable +## System Architecture + +### Current Data Feeds for AxC and Folio Data +- **Axiell Adapter Platform**: Runs on a 15-minute cadence, emitting changesets with new/modified/deleted records +- **FOLIO**: The new system of record for library management (items, holdings and patron data) +- **Integration Gap**: Changesets are written to Apache Iceberg tables on S3, but FOLIO is not automatically updated +- **Volume**: ~10–500 records per changeset (15-minute window); typical day: ~1,000 records across 80 syncs + + +``` +EventBridge Scheduler + │ + ▼ +┌──────────┐ ┌──────────┐ ┌────────────────────┐ ┌─────────────────┐ +│ Trigger │───▶│ Loader │───▶│ Transformer (ES) │───▶│ Elasticsearch │ +└──────────┘ └──────────┘ └────────────────────┘ └─────────────────┘ + │ + ▼ + Iceberg Adapter Table +``` + + +| Stage | File | Role | +|-------|------|------| +| Trigger | `src/adapters/steps/oai_pmh/trigger.py` | Compute next harvest window from WindowStore history | +| Loader | `src/adapters/steps/oai_pmh/loader.py` | Harvest OAI-PMH records in window, write raw MARCXML to Iceberg, emit `changeset_ids` | +| Transformer | `src/adapters/steps/transformer.py` + `src/adapters/transformers/axiell_transformer.py` | Read changesets from Iceberg, parse MARCXML with pymarc, produce SourceWork, index to ES | +| Reconciler | `src/adapters/transformers/axiell_reconciler.py` | Track GUID→ID mapping changes, emit DeletedSourceWork for superseded identifiers | + +#### Key Characteristics + +- **Metadata prefix**: `oai_marcxml` +- **OAI set**: `collect` +- **Auth**: Custom `Token` header (not standard `Authorization`) +- **Identity**: `axiell-guid` from MARC 001 +- **Visibility**: `InvisibleSourceWork` (MimsyWorksAreNotVisible) +- **Window cadence**: 15 min windows, 7 day lookback, 360 min max lag +- **FOLIO OAI-PMH feed**: FOLIO exposes an OAI-PMH feed that is available every 15 minutes as well +- **Storage**: Single Iceberg table schema (`namespace`, `id`, `content`, `changeset`, `last_modified`, `deleted`) + +--- + +## Proposed Change: FOLIO Item Upsert on Every Adapter Run + +### Fan-out Mechanism + +``` +EventBridge Scheduler + │ + ▼ +┌──────────┐ ┌──────────┐ ┌────────────────────┐ ┌─────────────────┐ +│ Trigger │───▶│ Loader │────┬───▶│ Transformer (ES) │───▶│ Elasticsearch │ +└──────────┘ └──────────┘ │ └────────────────────┘ └─────────────────┘ + │ │ + ▼ │ ┌──────────────────────────────────────────────┐ + Iceberg Table └───▶│ FOLIO Upserter (new step) │ + │ 1. Authenticate to FOLIO │ + │ 2. Load reference data cache │ + │ 3. Read changeset rows from Iceberg │ + │ 4. YAML mapper → instance/holdings/item │ + │ 5. Upsert: create · update · suppress │ + └──────────────────────────────────────────────┘ + │ + ▼ + ┌──────────────┐ + │ FOLIO │ + │ Inventory │ + └──────────────┘ +``` + +After the loader emits `changeset_ids`, the event is routed to **two** downstream targets: +1. Existing Axiell ES transformer (unchanged). +2. New **FOLIO upserter step**. + +Both consume the same `changeset_ids` independently. Either path can fail and retry without affecting the other. + +--- +### Target Upsert/Mapping Architecture + +```mermaid +flowchart LR + scheduler[EventBridge Scheduler
15-minute cadence] --> trigger[Trigger step] + trigger --> loader[Loader step
OAI-PMH harvest] + loader --> iceberg[(Iceberg adapter table)] + loader --> event[changeset_ids event] + + event --> es_transformer[Transformer step
AxiellTransformer] + es_transformer --> es[(Elasticsearch)] + + event --> folio_sync[Lambda: axiell-folio-sync
prototypes/axiell-folio-sync] + folio_sync --> ssm[SSM SecureStrings
FOLIO credentials] + folio_sync --> refcache[ref_cache.py
locations/material/loan types] + folio_sync --> read_changes[Read Iceberg rows
by changeset_ids] + read_changes --> mapper[yaml_mapper.py + mapping.yaml
MARCXML -> instance/holdings/item] + mapper --> upsert[upsert.py
create/update/suppress] + upsert --> okapi[(FOLIO OKAPI Inventory APIs)] +``` + + +## Transformation Design + +### Approach + +The upserter uses a **YAML-driven mapper** (`mapping.yaml` + `YamlMapper`) to convert raw MARCXML from AxC apaptor into FOLIO payloads. This separates all field mapping configuration from Python application code — the mapping file can be reviewed and adjusted independently. + +### Transformation Pipeline + +Raw MARCXML (from Iceberg) + │ + ▼ +┌────────────────────────────────────────┐ +│ YAML Mapper │ +│ Input: raw MARCXML record │ +│ Output: FOLIO item-level JSON │ +└────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────┐ +│ Schema Validation │ +│ Validate required fields present │ +│ Check identifiers well-formed │ +└────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────┐ +│ FOLIO API Payload Builder │ +│ Map validated output to FOLIO │ +│ Inventory API request shape │ +└────────────────────────────────────────┘ + +### YAML Mapper (`mapping.yaml` + `YamlMapper`) + +The YAML mapper is the core transformation engine. It uses to declaratively map MARCXML fields to FOLIO instance/holdings/item JSON payloads. + +**Design benefits:** +- **Separation of concerns**: Mapping logic lives in `mapping.yaml`, not Python code +- **Non-programmer-friendly**: Library staff and metadata experts can review/adjust mappings without code review +- **Version control**: Changes to mapping are tracked in git history +- **Testability**: Mapping rules can be tested independently of the Python runtime + +**Example workflow:** +1. Input: Raw MARCXML record from Iceberg (e.g., `...`) +2. Mapper applies rules: Extract subjects from `datafield[@tag='650']` → produces structured data +3. Output: JSON structure matching FOLIO Inventory API schema (e.g., `{"instanceTypeId": "...", "subjects": [...]}`) + +**Key files:** +- `mapping.yaml`: Declarative mapping rules for instance, holdings, item transformation +- `YamlMapper` class: Loads YAML, applies transformation rules, executes transformation on each record + --- -## Problem Statement +### Reference Data Cache (`ref_cache.py`) + +The reference cache maintains in-memory lookups for static FOLIO configuration data that rarely changes: +- **Locations**: location UUIDs and codes +- **Material types**: material type IDs (e.g., "Book", "Microfilm") +- **Loan types**: loan type IDs (e.g., "Can Circulate", "Reference") + +**Why needed:** +- FOLIO APIs require UUIDs/IDs, not human-readable names +- Axiell provides names; ref_cache translates to FOLIO IDs +- Caching avoids repeated API calls during a sync run (performance + resilience) -How do we build a system that: +**Workflow:** +1. Lambda startup → `ref_cache.load()` fetches from FOLIO reference endpoints +2. Mapper uses cache: `location_uuid = ref_cache.location["Wellcome Science"]` +3. If location name not found → error logged, record marked for manual review -1. **Reliably syncs incremental changesets** (10–500 records per 15-min window) from Axiell to FOLIO? -2. **Handles failures gracefully**: One bad record (mapping error, API failure) must not halt processing of others? -3. **Maintains a complete audit trail**: Every record must be traceable (created, updated, suppressed, failed)? -4. **Enables safe replay**: Failed changesets must be replayable without causing duplicates or data corruption? -5. **Keeps costs predictable**: For ~1K records/day, infrastructure cost should be minimal (~$5/month)? +**Cache persistence:** Reloaded fresh on every Lambda invocation; no cross-invocation caching. --- -## Executive Summary: Key Design Decisions +## Proposed AWS Architecture + +### Key Design Considerations We propose a **Step Functions + Lambda + S3 architecture** with synchronous invocation, per-record error isolation, and 90-day audit retention. This approach balances **operational visibility**, **fault resilience**, and **simplicity**. @@ -56,67 +267,43 @@ We propose a **Step Functions + Lambda + S3 architecture** with synchronous invo --- -## Architecture: Current Implementation - ### System Diagram -``` -┌──────────────────────────────────────────────────────────────────────────┐ -│ Axiell Adapter Platform (15-minute cadence) │ -├──────────────────────────────────────────────────────────────────────────┤ -│ • Writes changesets to Iceberg table (namespace, id, content, deleted) │ -│ • Emits EventBridge event: axiell.adapter.completed │ -│ { changeset_ids: [...], job_id: "xyz", transformer_type: "axiell" } │ -└────────────────────┬─────────────────────────────────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────────────────────────────────────────┐ -│ EventBridge Rule: axiell-folio-sync-axiell-adapter-completed │ -│ • Pattern: source="axiell.adapter" + transformer_type="axiell" │ -│ • Target: Step Function (synchronous invocation) │ -└────────────────────┬─────────────────────────────────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────────────────────────────────────────┐ -│ Step Function: axiell-folio-sync-sfn │ -│ • Passes event detail to Lambda │ -│ • Retry policy: max 3 attempts, exponential backoff (2s, 4s, 8s) │ -│ • Error handling: Catch-all → SyncFailed (terminal state) │ -│ • Execution history: CloudWatch Logs (30-day retention) │ -└────────────────────┬─────────────────────────────────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────────────────────────────────────────┐ -│ Lambda: axiell-folio-sync (ECR image) │ -│ • Timeout: 300s (5 min) — handles ~200 records with 3 API calls each │ -│ • Memory: 1024 MB │ -│ │ -│ Execution Flow: │ -│ 1. Auth: Fetch OKAPI credentials from SSM, POST /authn/login │ -│ 2. Scan: Query Iceberg for changesets (WHERE changeset_id IN [...]) │ -│ 3. Map & Validate: Apply YAML rules, build Instance/Holdings/Item │ -│ 4. Upsert: For each record, POST/PUT to FOLIO (per-record errors) │ -│ 5. Write Manifests: Flush to S3 NDJSON (success, errors, metadata) │ -│ │ -│ Environment Variables (injected by Terraform): │ -│ • OKAPI_URL, OKAPI_TENANT, OKAPI_SECRET_PARAM │ -│ • MANIFEST_S3_BUCKET, S3_TABLE_BUCKET_ARN, ICEBERG_TABLE_NAME │ -│ • AWS_REGION, DRY_RUN (override via event) │ -└────────────────────┬─────────────────────────────────────────────────────┘ - │ - ▼ -┌──────────────────────────────────────────────────────────────────────────┐ -│ S3: Manifest Storage (axiell-folio-sync-manifests-{account}-{region}) │ -│ │ -│ Files (per job_id): │ -│ • {job_id}.ids.ndjson — success records (5K/batch) │ -│ • {job_id}.ids.failures.ndjson — error records (if any) │ -│ • {job_id}.manifest.json — metadata summary │ -│ │ -│ Lifecycle: 90-day expiration (auto-delete old manifests) │ -│ Versioning: Enabled (audit trail for object changes) │ -│ Encryption: SSE-S3 (default) │ -└──────────────────────────────────────────────────────────────────────────┘ +```mermaid +flowchart TD + subgraph upstream["Axiell Adapter Platform - upstream, not in this stack"] + adapter["Adapter - 15-min cadence
writes changesets to Iceberg
emits axiell.adapter.completed"] + bus(["EventBridge bus
catalogue-pipeline-adapter-event-bus"]) + adapter -->|"PutEvents - detail:
changeset_ids, job_id, transformer_type"| bus + end + + subgraph sync["axiell-folio-sync - this Terraform stack"] + rule["EventBridge Rule
axiell-folio-sync-axiell-adapter-completed
pattern: source=axiell.adapter +
detail-type=axiell.adapter.completed +
detail.transformer_type=axiell"] + sfn["Step Function - STANDARD
axiell-folio-sync-sfn
InvokeSyncLambda then Retry 3x (backoff 2.0)
Catch States.ALL then SyncFailed"] + lambda["Lambda - ECR image
axiell-folio-sync
timeout 300s, mem 512MB
auth, scan, map, upsert, manifests"] + rule -->|"target: StartExecution
role: eventbridge-exec"| sfn + sfn -->|"lambda InvokeFunction
payload: detail.* plus dry_run"| lambda + end + + bus --> rule + + subgraph data["Data stores and observability"] + ssm[("SSM SecureString
okapi-credentials")] + iceberg[("S3 Tables / Iceberg
wellcomecollection-platform-axiell-adapter")] + ecr[("ECR
uk.ac.wellcome/axiell-folio-sync")] + s3man[("S3 manifests
axiell-folio-sync-manifests-acct-region
90-day lifecycle, versioned, SSE-S3")] + cw["CloudWatch
Logs: /aws/lambda + /aws/states (30d)
Metrics: AxiellFolioSync"] + end + + folio(["FOLIO OKAPI - external
api-wellcome.folio.ebsco.com"]) + + ecr -.->|"pull image"| lambda + ssm -.->|"GetParameter + KMS decrypt"| lambda + iceberg -.->|"scan changeset rows"| lambda + lambda -->|"authn/login + upsert
instance, holdings, item"| folio + lambda -->|"NDJSON: ids / failures / manifest"| s3man + lambda -->|"metrics when not dry_run"| cw + sfn -.->|"execution history"| cw ``` ### 1. EventBridge Trigger @@ -182,6 +369,102 @@ Output (Success) → SyncComplete --- +### RefCache Resolution + +All reference fields (`permanentLocationId`, `instanceTypeId`, `materialTypeId`, `permanentLoanTypeId`) are resolved at sync time via **RefCache**, which caches FOLIO reference data (locations, material types, loan types, instance types). + +**Resolution Process**: +1. On Lambda startup, RefCache initializes by querying FOLIO reference APIs (one-time call per invocation) +2. For each record, mapper looks up codes against cached values (in-memory, O(1) lookup) +3. If code not found: upsert fails with `MappingError` and record is skipped +4. Resolved UUIDs are embedded in payload + +**Current Normalisations**: +- **Material type**: AxC `Object_category` → FOLIO material type (e.g., `"Archives - Non-digital"` → `"unspecified"`; audio → `"sound recording"`) +- **Location**: AxC hierarchical location code (e.g., `"215;B11;MR;84;3;7"`) → FOLIO location UUID +- **Loan type**: AxC `OrderingCodes` (e.g., `"Archives - Requestable"`) → FOLIO loan type UUID +- **Instance type**: Typically mapped to `"text"` or domain-specific type + +**Implementation**: Can be checked in the prototype + +---` + +Benefits: +- **A/B audit**: Compare records created with different mapper versions +- **Rollback**: If a mapping change introduces errors, identify affected records by version +- **Metadata tracking**: Historical record of mapping logic used for each record + +### Sample Field Mapping + +| MARC Source (Axiell) | mapping.yaml key | FOLIO Target | Example | +|---------------------|------------------|--------------|----------| +| 001 (GUID) | `external_id` | Instance/Item `hrid` | `axiell:PP/CJS/B.3/9` | +| 245$a | `title` | Instance title | Daniel Morley, an English Philosopher... | +| 852$b | `location_code` | Holdings permanent location | `215;B11;MR;84;3;7` | +| 852$c | `shelving_location` | Holdings shelving location | `Arch`, `Ref` | +| 852$h | `call_number` | Holdings call number | — | +| 949$a | `barcode` | Item barcode | — | +| 949$c | `item_type` | Item material type | `Archives - Non-digital` | +| 949$l | `loan_type` | Item loan type | `Archives - Requestable` | +| 876$p | `copy_number` | Item copy number | `copy 1`, `copy 2` | +| 876$t | `volume` | Item volume designation | `v.1`, `disc 1 of 2` | +| 856$u | `electronic_access_uri` | Electronic access URL | — | + +--- + +## FOLIO Field Mapping Reference + +For a comprehensive and detailed mapping of all Axiell Collections fields to FOLIO Inventory API fields, see **[folio-axc-fields-mapping.md](folio-axc-fields-mapping.md)**. + +This document provides: +- Complete field-by-field mapping for Instance, Holdings, and Item entities +- Transformation rules and MARC source information +- Required vs. optional fields for FOLIO +- Reference data requirements (locations, material types, loan types) +- Field validation rules and edge cases + +--- + +## Change Detection Mechanism + +The FOLIO upsert step leverages the existing Axiell adapter's OAI-PMH change detection to determine which records to sync. + +### How It Works + +1. **OAI-PMH Datestamp Windows**: Axiell adapter harvests records modified within a time window. Only records with `last_modified` within `[window_start, window_end)` are returned from OAI-PMH. + +2. **Changeset IDs**: Each harvest window produces a unique `changeset_id`. Records written to Iceberg in that window are tagged with it. + +3. **Iceberg Columns**: Records stored with: + - `namespace` — record type (e.g., "location", "item") + - `id` — external identifier + - `content` — JSON-serialized record + - `changeset` — changeset ID from adapter + - `last_modified` — timestamp from OAI datestamp header + - `deleted` — `true` if OAI returned a tombstone + +4. **FOLIO Upsert Input**: Lambda receives `changeset_ids` and reads only those rows: + ```python + changed_records = adapter_store.read_changed(changeset_ids) + ``` + +### Change Detection Signals + +| Signal | Source | Meaning | +|--------|--------|----------| +| Record in changeset | OAI-PMH datestamp window | Record was created or modified in source | +| `deleted=true` | OAI tombstone | Record was removed from source | +| Payload hash mismatch | XSL output comparison (optional) | FOLIO-relevant fields actually changed | +| Reconciler GUID remap | Axiell reconciler step | Old identity superseded, emit delete for old | + +### Every Record in Changeset Is Either: + +- **New** (first time this `id` appears in Iceberg) → create in FOLIO +- **Updated** (existing `id`, newer `last_modified`) → update in FOLIO +- **Deleted** (`deleted=true`) → suppress/remove in FOLIO + +--- + ### 3. Lambda Function: axiell-folio-sync **Docker Image**: ECR (`uk.ac.wellcome/axiell-folio-sync:TAG`) @@ -246,6 +529,7 @@ for row in iceberg_records: ``` #### Step 4: Upsert to FOLIO (Per Record) + ``` For each record: Execute: Instance → Holdings → Item (dependencies) @@ -274,6 +558,24 @@ Per-record error handling: • All errors logged to CloudWatch AND accumulated ``` +### Upsert Key Strategy (Idempotency) + +**Priority order for matching existing records**: + +1. **External source identifier**: Axiell GUID from MARC 001 (most stable) +2. **Barcode**: From transformed record (949$a) +3. **Composite fallback**: `axiell:{record_id}` + +**Behavior**: +- **Found**: Update mutable fields, preserve FOLIO-internal metadata +- **Not found**: Create new item, link to holding/instance via location +- **Missing required fields**: Skip with structured error logged + +**Replay Safety**: +- Upserts are idempotent by external identifier +- Same changeset processed twice produces same outcome +- Manifest deduplication: check if changeset already processed successfully before running + #### Step 5: Batch & Write Manifests ```python # Success NDJSON (one record per line, 5K records per batch) @@ -504,119 +806,23 @@ manifests/ --- -## Testing & Operations +## FOLIO API Client -### Test Levels +**Location**: `prototypes/axiell-folio-sync/axiell_folio_sync/upsert.py` -**Unit Tests** (in `tests/`): -- YAML mapper applies rules correctly to test records -- Upsert logic handles create/update/suppress/error scenarios -- Manifest NDJSON serialization and metadata structure +**Responsibilities**: +- Authenticate to FOLIO via OKAPI (`/authn/login`), token refreshed automatically on 401 +- Resolve existing records by CQL query (GET before PUT) +- Create (POST), update (PUT), or suppress deleted records per entity +- Handle rate limiting and retries -**Contract Tests** (against mock FOLIO): -- `dry_run=true`: Lambda computes changes but makes no writes -- Validates payload shape, required fields, API call sequence (no actual HTTP calls) - -**Integration Tests** (against dev FOLIO): -- Create test changesets in Iceberg -- Trigger sync via EventBridge -- Verify records appear in FOLIO with expected fields -- Cleanup: suppress test records - -**Smoke Tests** (pre-deployment): -- CloudFormation drift: no manual infrastructure changes -- Secrets: OKAPI credentials set in SSM -- EventBridge rule: active and routable to Step Function -- Lambda image: exists in ECR and is accessible -- Iceberg: S3 bucket and table readable - -### Operational Runbooks - -**Replay Failed Changesets**: -``` -1. Identify failed changeset_id from CloudWatch or S3 manifest -2. Emit EventBridge event with same changeset_ids -3. Monitor S3 manifest and CloudWatch to confirm success -``` - -**Manual Inspection**: -``` -# Query S3 manifest for records with errors -aws s3api select-object-content \ - --bucket axiell-folio-sync-manifests-123456-eu-west-1 \ - --key manifests/{job_id}.ids.failures.ndjson \ - --expression "SELECT * FROM s3object WHERE type = 'api'" \ - --expression-type SQL \ - --input-serialization '{"JSON": {}}' \ - --output-serialization '{"JSON": {}}' \ - output.json -``` - -**Emergency Shutdown**: -``` -# Disable EventBridge rule (stops new syncs) -aws events disable-rule --name axiell-folio-sync-axiell-adapter-completed - -# Monitor in-flight executions via Step Function console -# When all complete, investigate root cause -# Re-enable when ready -aws events enable-rule --name axiell-folio-sync-axiell-adapter-completed -``` +**Token Management**: +- Initial auth: `POST /authn/login` with credentials from SSM +- Token lifetime: 24 hours (sufficient for 5-min Lambda execution) +- Reused across all API calls in single invocation (no refresh overhead) --- -## Future Scalability: Pipelining (When >500 Records/Event) - -### Current Limits -- **Lambda timeout**: 300 seconds (5 min) -- **Typical performance**: ~200 records (with 3 API calls per record) in 45 sec -- **Token lifetime**: 24 hours (no refresh needed within 5-min window) - -### Growth Trigger -When we observe >500 records per event (i.e., >80 records per changeset on average), parallelization becomes valuable: - -**Baseline**: 1 Lambda processes 1 event → 200 records/45s → 1,600 records/hour -**Limit**: 5 min timeout → ~280 records/event max (not good) - -**With parallelization**: 4 Lambda workers → 800 records/45s → 6,400 records/hour - -### Proposed Pipelining Architecture - -``` -Step Function: Map state (fan-out per changeset) - ├─ [Task] Fetch/cache OKAPI token (ElastiCache or in-memory) - │ - ├─ [Map] Parallel workers (one per changeset) - │ ├─ Worker 1: Process changeset 1 (reuse cached token) - │ ├─ Worker 2: Process changeset 2 - │ ├─ Worker 3: Process changeset 3 - │ └─ [... N workers in parallel] - │ - └─ [Join] Aggregate manifests - • Collect all success/error manifests from workers - • Write single metadata JSON (summarizing all workers) - • Emit CloudWatch metrics (aggregated counts) -``` - -**Key Optimization: Token Caching** -- Current: Each Lambda auth calls OKAPI → O(N changesets) auth calls -- Future: Fetch token once, cache in ElastiCache or Lambda shared memory → O(1) auth calls - -**Same Manifest Format** -- No breaking changes: Workers still write NDJSON to S3 -- Single metadata JSON consolidates all results -- Queries remain unchanged (S3 Select can union multiple manifest files) - -**Configuration Change**: -```terraform -# In Step Function definition, swap single Lambda invocation for Map state -# Lambda code changes: add token_cache parameter -# No changes to manifest format, API schema, or operational procedures -``` - -**Timeline**: Evaluate when we consistently see >500 records/event in production (track via CloudWatch metric `RecordsPerEvent`) - ---- ## Cost Analysis @@ -631,19 +837,6 @@ Step Function: Map state (fan-out per changeset) | CloudWatch Logs | ~80 × 5 KB = 400 KB/month | 400 KB ingested | $0.50/GB ingested | ~$0.20 | | **Total** | | | | **~$3–5** | -### Future Volume: ~10,000 records/day (10x growth) - -| Service | Cost/mo | -|---------|---------| -| Lambda | ~$12 (10x invocations, 4x longer processing time due to size) | -| Step Functions | ~$0.20 (10x transitions) | -| S3 | ~$15 (more objects, higher storage) | -| CloudWatch Logs | ~$2 (10x log volume) | -| **Total** | **~$30–40** | - -**Note**: If >50 changesets/event, pipelining becomes cost-effective. With parallelization, Lambda compute cost would drop (faster wall-clock time → fewer seconds billed), but Step Function costs would increase (more state transitions per Map worker). - ---- ## Assumptions & Constraints @@ -653,14 +846,18 @@ Step Function: Map state (fan-out per changeset) - **Axiell stability**: Assumes Axiell adapter completes consistently. If adapter fails, no changeset is emitted → no sync triggered (not our problem, but should monitor). --- +## Open Questions + +### Field Mapping from Axc to Folio Instance, Holdings and Items needs to be defined + +- Sample file **[folio-axc-fields-mapping.md](folio-axc-fields-mapping.md)**, File shared with Collection Information for feedback +- Minimum Stub Record needs to be defined. +- Source of data Marc/Folio for inventory types. Test with both the options + ## Next Steps -1. **Approval**: Review this RFC and confirm architectural choices align with Wellcome Collection platform strategy. -2. **Implementation**: Create Terraform modules, Lambda container image, Step Function definition, unit/integration tests. -3. **Deployment**: Stage in dev FOLIO → test end-to-end → deploy to prod with runbooks. -4. **Monitoring**: Set up CloudWatch dashboards, alarms on failure rate, manifest size growth. -5. **Feedback loop**: Monitor production for 1–2 months, adjust retry policy and timeout if needed. +- Prototype adding another lambda in the step function for moving the read of data from iceberg and write of data to Folio in separare lambdas and check on the approach --- diff --git a/rfcs/090-axiell-folio-sync/calm-to-sierra-harvester-flow.md b/rfcs/090-axiell-folio-sync/calm-to-sierra-harvester-flow.md new file mode 100644 index 00000000..c052b7bc --- /dev/null +++ b/rfcs/090-axiell-folio-sync/calm-to-sierra-harvester-flow.md @@ -0,0 +1,82 @@ +# CALM to Sierra Harvester Flow + +## Current System Architecture +The Wellcome Collection library systems currently operate as: +- **CALM** (Collections Management): Content Management System (CMS) — bibliographic and collection metadata source of truth +- **Sierra**: Library Management System (LMS) — patron management, circulation, holds, and requesting + +The CALM-to-Sierra harvester is the integration mechanism that synchronizes bibliographic metadata from CALM into Sierra for discovery and requesting workflows. + +## Migration Context +This harvester flow is **transitional**. The library systems migration is replacing these systems: +- **CALM → Axiell Collections**: Bibliographic and collection data migrating to Axiell Collections +- **Sierra → FOLIO**: Patron management, circulation, and requesting migrating to FOLIO + +The current CALM-to-Sierra harvester will be superseded by an Axiell Collections-to-FOLIO integration pipeline. This document captures the existing harvester behavior for reference during the migration period. + +## Purpose +This document describes how records currently move from CALM into Sierra for requesting and closed-stack workflows. + +## Scope +The flow covers: +- CALM record eligibility (Harvest flag) +- OAI-PMH exposure and transformation +- Sierra harvester scheduling and execution +- Sierra load table behavior for bib and item create/overlay logic + +## End-to-End Flow Summary +1. A cataloguer sets the CALM local field **Harvest** to `Yes` on records that should be exported. +2. The CALM OAI repository publishes only eligible records through the configured set. +3. The configuration settings for the OAI repository (on wt-CALM at Start>Programs>Axiell>OAI Suite>OAI Server Configuration) ensure that only records with the status of ‘Harvested’=Yes should be made available to external harvesters via the OAI repository: +4. The OAI layer applies [docs/CalmInnopac.xsl](docs/CalmInnopac.xsl) to transform CALM metadata into MARCXML. +5. Sierra Harvester Task pulls records from the CALM OAI endpoint (set + format + match parameters). +6. A scheduled Sierra Harvester Job runs nightly and loads transformed records. +7. Sierra Load Table rules create or overlay bib/item records based on `035` and `949$c` matching. + +## Mermaid Diagram +```mermaid +flowchart TD + A[CALM Record Created/Updated] --> B{Harvest field = Yes?} + B -- No --> Z[Record Not Harvested] + B -- Yes --> C[CALM OAI Repository Includes Record] + + C --> D[OAI Set: Catalog1 / All] + D --> E[Transform via docs/CalmInnopac.xsl] + E --> F[MARCXML Output] + + F --> G[Sierra Harvester Task] + G --> H[Sierra Harvester Job\nDaily ~12:05am to ~05:30am] + H --> I[Sierra Load Table Rules] + + I --> J{Match on 035?} + J -- No --> K[Create New Bib + New Item from 949] + J -- Yes --> L{Match on 949$c?} + L -- Yes --> M[Overlay Bib + Overlay Item] + L -- No --> N[Overlay Bib + Insert New Item] + + M --> O[Updated Sierra Records Available] + N --> O + K --> O +``` + +## Key Configuration Points +- CALM client label: **Harvest** (`Yes`/`No`), default is `No` for new records. +- Underlying CALM field reference noted in source: **Transmission** (label shown as Harvested in parts of the OAI setup). +- The criteria for harvest is further refined byOAI set definition source: `CalmDataSource.xml` with set id named as `Catalog1`. +- Calm Data source also specifies which stylesheet should be defined to transform data.[text](../../catalogue-pipeline/catalogue_graph/docs/axiell-folio-upsert.md) +- Transform stylesheet: [docs/CalmInnopac.xsl](docs/CalmInnopac.xsl). +- Metadata format used for harvest: MARC21/MARCXML. + +## Sierra Harvester Components +- **Task**: Data source endpoint, set, metadata format, overlay match point (CALM record identifier usage). +- **Job**: Active/inactive schedule and execution window. + +## Sierra Load Table Decision Logic +- No `035` match: create bib and create item from `949`. +- `035` match and `949$c` match: overlay bib and overlay item. +- `035` match and no `949$c` match: overlay bib and add a new item. + +## Operational Notes +- Harvest runs daily and may take several hours. +- A summary report is produced after completion. +- Any change to set rules or transform stylesheet can alter downstream bib/item behavior and should be tested before production rollout. diff --git a/rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md b/rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md new file mode 100644 index 00000000..1910f2a4 --- /dev/null +++ b/rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md @@ -0,0 +1,70 @@ +# FOLIO Minimum Fields Mapping (AxC → FOLIO) + +**Date:** 2026-06-17 +**Status:** Reference guide for AxC to FOLIO data integration +**Related:** [axc-folio-field-mapping.md](axc-folio-field-mapping.md) · [axiell-to-folio-upsert.md](axiell-to-folio-upsert.md) +**Code:** `prototypes/axiell-folio-sync/axiell_folio_sync/mapper.py` + +--- + +## Instance + +Minimum recommended fields to create a discoverable, functional instance record in FOLIO Inventory. + +| FOLIO Field | Type | Status | OIM Field (AxC) | MARC Tag | Notes | +|-------------|------|--------|-----------------|----------|-------| +| `hrid` | string | Required | `object_number` | `001` | Hierarchical record identifier. Format: `axiell:{source_id}`. Example: `axiell:PP/CJS/B.3/9` | +| `title` | string | Required | `Titles/title` | `245$a` | Primary title from object. Essential for discoverability; fails API if absent. Example: `Daniel Morley, an English Philosopher of the XIIth Century. Isis, 1920, iii, 263-9` | +| `source` | string | Required | — | — | Hardcoded to `"Axiell"` to mark origin system. | +| `instanceTypeId` | UUID | Required | — | — | Resolved from RefCache. Typically `"text"` or domain type (e.g., `"unspecified"` for mixed media). | +| `formerIds` | array | Recommended | `guid` + `Alternative_number` (Calm RefNo) | `035$a` + `244$a` | Preserve alternate identifiers for reference and migration. Includes AxC GUID and legacy Calm RefNo. Example: `["axiell:PP/CJS/B.3/9", "PPCJS/B/3/9"]` | +| `discoverySuppress` | boolean | Recommended | `accession_status` | — | Set `true` for RESTRICTED items or draft records. Default `false` for OPEN items. | + +--- + +## Holdings + +Minimum recommended fields to link instance to location and call numbers. + +| FOLIO Field | Type | Status | OIM Field (AxC) | MARC Tag | Notes | +|-------------|------|--------|-----------------|----------|-------| +| `hrid` | string | Required | `object_number` + `location.default.name` | `001` + `852$b` | Composite identifier linking record to location. Format: `axiell:{source_id}-holding-{location_slug}`. Example: `axiell:PP/CJS/B.3/9-holding-215-b11-mr-84-3-7` where `location_slug` is lowercased, hyphenated. | +| `instanceId` | UUID | Required | — | — | FOLIO instance UUID. Resolved after instance POST succeeds. | +| `permanentLocationId` | UUID | Required | `location.default.name` | `852$b` | Hierarchical shelf location from AxC. Resolved from RefCache using location code string. Example: `215;B11;MR;84;3;7` → UUID lookup. | +| `callNumber` | string | Recommended | — | `852$h` | Call number suffix. Optional but aids patron retrieval. | +| `callNumberPrefix` | string | Optional | — | `852$c` | Call number prefix or classification. Example: `"Arch"`, `"Ref"`. | +| `shelvingOrder` | string | Optional | — | `852$j` | Sort/shelving sequence for physical ordering on shelf. | +| `formerIds` | array | Recommended | `guid` + `Alternative_number` | `035$a` + `244$a` | Preserve AxC identifiers. Example: `["axiell:PP/CJS/B.3/9"]` | +| `discoverySuppress` | boolean | Recommended | `accession_status` | — | Set `true` if holdings should not appear in public discovery. Inherits from instance or may override. | + +--- + +## Item + +Minimum recommended fields to represent the physical/digital object in circulation. + +| FOLIO Field | Type | Status | OIM Field (AxC) | MARC Tag | Notes | +|-------------|------|--------|-----------------|----------|-------| +| `hrid` | string | Required | `object_number` | `001` | Item record identifier. Format: `axiell:{source_id}`. Example: `axiell:PP/CJS/B.3/9` | +| `holdingsRecordId` | UUID | Required | — | — | FOLIO holdings UUID. Resolved after holdings POST succeeds. | +| `status.name` | string | Required | — | — | Hardcoded to `"Available"` for initial sync. Valid values: `"Available"`, `"Awaiting pickup"`, `"Checked out"`, etc. | +| `materialTypeId` | UUID | Required | `Object_category/object_category` | `949$c` | Material type (format) of the item. Resolved from RefCache. Example: `Archives - Non-digital` → `unspecified` UUID. See material type normalisation table. | +| `permanentLoanTypeId` | UUID | Required | `Free_texts` (type=`OrderingCodes`) | `949$l` | Loan policy. Resolved from RefCache. Example: `"Archives - Requestable"` → loan type UUID. | +| `barcode` | string | Optional | — | `949$a` | Machine-readable barcode. Used for physical item tracking. May be absent for archival items. | +| `copyNumber` | string | Optional | — | `876$p` | Distinguishes multiple copies of the same item. Example: `"copy 1"`, `"copy 2"`. | +| `volume` | string | Optional | — | `876$t` | Volume designation for multi-part works. Example: `"v.1"`, `"disc 1 of 2"`. | +| `formerIds` | array | Recommended | `guid` | `035$a` | Preserve AxC GUID for reference. Example: `["axiell:4d8f1208-9812-4bb5-84ef-da436b22d9e2"]` | +| `discoverySuppress` | boolean | Recommended | `accession_status` | — | Set `true` for RESTRICTED items. Hides item from public search results. | + +--- + +## RefCache Resolution + +All reference fields (`permanentLocationId`, `instanceTypeId`, `materialTypeId`, `permanentLoanTypeId`) are resolved at sync time via **RefCache**, which caches FOLIO reference data (locations, material types, loan types, instance types). If a code cannot be resolved, the upsert fails with `MappingError`. + +**Current normalisations:** +- **Material type:** AxC `Object_category` → FOLIO material type (e.g., `"Archives - Non-digital"` → `"unspecified"`; audio → `"sound recording"`). +- **Location:** AxC hierarchical location code (e.g., `"215;B11;MR;84;3;7"`) → FOLIO location UUID. +- **Loan type:** AxC `OrderingCodes` (e.g., `"Archives - Requestable"`) → FOLIO loan type UUID. + +See `prototypes/axiell-folio-sync/axiell_folio_sync/ref_cache.py` for lookup implementation. diff --git a/rfcs/090-axiell-folio-sync/mapping.yaml b/rfcs/090-axiell-folio-sync/mapping.yaml new file mode 100644 index 00000000..bcdf089b --- /dev/null +++ b/rfcs/090-axiell-folio-sync/mapping.yaml @@ -0,0 +1,86 @@ +# AxC → FOLIO field mapping +# Version is stamped into every produced payload for traceability. +# Field rules: +# from_marc: "TAG$subfield" — extract first matching subfield value +# from_marc: "TAG" — extract control field value (no $) +# default: — use this literal if from_marc is absent or empty +# lookup: — resolve value via RefCache (location|materialType|loanType|instanceType) +# map: — apply value mapping from mapping_tables section +# template: "" — interpolate {source_id}, {instance_hrid}, {holdings_hrid} +# as_list: true — wrap result in a JSON array +# as_electronic_access: true — wrap URI as [{uri: }] +# transforms: [trim|upper|lower] +# required: true — MappingError raised if value is null after resolution +# Fields with no value after all rules are omitted from the payload (not null). + +version: "1.0.0" + +mapping_tables: + # Material type normalization: Axiell source codes → FOLIO standard names + material_type: + "sound only": "sound recording" + "audio-visual material - visual": "video recording" + "audio-visual material - e-sound only": "sound recording" + "audio-visual material - e-visual only": "video recording" + "published material": "Books" + # Archives variants default to "unspecified" (handled via pattern matching below) + "archives": "unspecified" + +identity: + # MARC controlfield used as the source record identifier + marc_controlfield: "001" + # Prefix prepended to form the instance hrid: AxC:{source_id} + hrid_prefix: "AxC" + +instance: + fields: + hrid: + template: "AxC:{source_id}" + title: + from_marc: "245$a" + transforms: [trim] + required: true + source: + default: "AxC" + instanceType.id: + lookup: instanceType + required: true + +holdings: + fields: + hrid: + template: "AxC:{source_id}-holding-{location_slug}" + # FOLIO requires holdings sourceId; resolve it by name via RefCache. + sourceId: + default: "MARC" + lookup: holdingsSource + required: true + permanentLocationId: + from_marc: "852$b" + default: "History of Medicine" + lookup: location + required: true + +item: + fields: + hrid: + template: "AxC:{source_id}" + materialType.id: + from_marc: "949$c" + map: material_type + default: "Books" + lookup: materialType + required: true + permanentLoanType.id: + from_marc: "949$l" + default: "Can Circulate" + lookup: loanType + required: true + permanentLocation.id: + from_marc: "852$b" + default: "History of Medicine" + lookup: location + required: true + status: + default: + name: "Available" From 157a7b45260d8d0c18a6dbccc5a1bffee404a118 Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Tue, 23 Jun 2026 09:13:23 +0100 Subject: [PATCH 03/15] Fix the last modified date --- rfcs/090-axiell-folio-sync/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index c76072c0..1ef0a715 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -4,7 +4,7 @@ This is proposal to synchronize library location and item holdings from Axiell Collections(Content Management System) into FOLIO (Library Management system) with strict requirements for idempotency, auditability, and graceful error isolation. -**Last modified:** 2026-06-22 +**Last modified:** 2026-06-23T00:00:00+00:00 ## Table of Contents From 43ef72ca391bc6e841e3fb8ccc3b32f054d1c82d Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Tue, 23 Jun 2026 09:33:15 +0100 Subject: [PATCH 04/15] Update RFC table --- rfcs/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/rfcs/README.md b/rfcs/README.md index 32fbb20e..e97c811d 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -74,6 +74,7 @@ _This is generated from the RFCs in this directory using `.scripts/create_table_ | RFC ID | Summary | Next Line | Last Modified | |--------|---------|-----------|---------------| +| [090-axiell-folio-sync](090-axiell-folio-sync/README.md) | RFC 090: CMS to LMS Sync | This is proposal to synchronize library location and item holdings from Axiell Collections(Content Management System) into FOLIO (Library Management system) with strict requirements for idempotency, auditability, and graceful error isolation. | 23 Jun 2026 | | [088-folio-identity-requesting-migration](088-folio-identity-requesting-migration/README.md) | RFC 088: Migrating identity, requesting and items APIs from Sierra to FOLIO | This RFC describes how we move the identity, requesting and item-availability APIs that power `wellcomecollection.org` from our current Library Management System (LMS), **Sierra**, to its replacement, **FOLIO**. It sets out the proposed architecture (a parallel, FOLIO-backed **v2** identity API fronted by Auth0), the embedded API contract, the migration plan (a per-request website toggle plus lazy patron migration, culminating in a single coordinated cutover), and the questions still open before cutover. | 22 Jun 2026 | | [089-identifiers-api](089-identifiers-api/README.md) | RFC 089: Identifiers API | This RFC proposes a small, read-only **Identifiers API** that resolves a **canonical** catalogue identifier to its **source** identifier(s) and back, served from the catalogue ID Registry (the same store the ID Minter writes to, per [RFC 083](../083-stable_identifiers/README.md)). It provides that translation in one place, between the canonical ids the public surface uses and the source ids (Sierra numbers, FOLIO UUIDs, CALM/Axiell refs) that the underlying systems require across the Sierra/CALM → FOLIO/Axiell migration. It sets out the contract, the AWS architecture, the authentication and cost model, the caching strategy, and what a working prototype has already established. | 18 Jun 2026 | | [087-kiosk-mode](087-kiosk-mode/README.md) | RFC 087: wellcomecollection.org in kiosk mode | This RFC serves to outline how we propose to offer in-venue experiences using our current website, while optimising it for a different experience than usual. | 13 May 2026 | From f9a16a848cfb8a03d90c31ad56ed56796cf696ea Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Tue, 23 Jun 2026 09:46:56 +0100 Subject: [PATCH 05/15] Improve the diagram --- rfcs/090-axiell-folio-sync/README.md | 28 ++++++---------------------- 1 file changed, 6 insertions(+), 22 deletions(-) diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index 1ef0a715..a728bf2e 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -179,28 +179,12 @@ The upserter uses a **YAML-driven mapper** (`mapping.yaml` + `YamlMapper`) to co ### Transformation Pipeline -Raw MARCXML (from Iceberg) - │ - ▼ -┌────────────────────────────────────────┐ -│ YAML Mapper │ -│ Input: raw MARCXML record │ -│ Output: FOLIO item-level JSON │ -└────────────────────────────────────────┘ - │ - ▼ -┌────────────────────────────────────────┐ -│ Schema Validation │ -│ Validate required fields present │ -│ Check identifiers well-formed │ -└────────────────────────────────────────┘ - │ - ▼ -┌────────────────────────────────────────┐ -│ FOLIO API Payload Builder │ -│ Map validated output to FOLIO │ -│ Inventory API request shape │ -└────────────────────────────────────────┘ +```mermaid +graph TD + A["Raw MARCXML
(from Iceberg)"] --> B["YAML Mapper
Input: raw MARCXML record
Output: FOLIO item-level JSON"] + B --> C["Schema Validation
Validate required fields present
Check identifiers well-formed"] + C --> D["FOLIO API Payload Builder
Map validated output to FOLIO
Inventory API request shape"] +``` ### YAML Mapper (`mapping.yaml` + `YamlMapper`) From 9547ec438acb57c851fa3ee6b3b2b8074f73f1ed Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Tue, 30 Jun 2026 12:18:40 +0100 Subject: [PATCH 06/15] Updated RFC with the requested changes --- rfcs/090-axiell-folio-sync/README.md | 791 +++++++++--------- .../folio-axc-fields-mapping.md | 104 ++- rfcs/090-axiell-folio-sync/mapping.py | 302 +++++++ rfcs/090-axiell-folio-sync/mapping.yaml | 86 -- 4 files changed, 751 insertions(+), 532 deletions(-) create mode 100644 rfcs/090-axiell-folio-sync/mapping.py delete mode 100644 rfcs/090-axiell-folio-sync/mapping.yaml diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index a728bf2e..3d20d843 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -2,49 +2,60 @@ ## Purpose -This is proposal to synchronize library location and item holdings from Axiell Collections(Content Management System) into FOLIO (Library Management system) with strict requirements for idempotency, auditability, and graceful error isolation. +This RFC proposes an automated pipeline to synchronize data from **Axiell Collections (AxC)**, the new Content Management System (CMS), into **FOLIO**, the new Library Management System (LMS). The records from Axiell Collections need to exist in FOLIO so they can be requested and circulated in the LMS. It is designed for **idempotency** (safe to replay without duplication), **auditability** (every create / update / suppress is recorded), and **graceful error isolation** (one bad record never halts the batch). -**Last modified:** 2026-06-23T00:00:00+00:00 +**Last modified:** 2026-06-30T00:00:00+00:00 ## Table of Contents - [Purpose](#purpose) - [Background](#background) - - [Current CMS and LMS systems](#current-cms-and-lms-systems) - - [Migration Context](#migration-context) - - [Why This Matters](#why-this-matters) +- [Scope](#scope) - [System Architecture](#system-architecture) - [Current Data Feeds for AxC and Folio Data](#current-data-feeds-for-axc-and-folio-data) - - [Key Characteristics](#key-characteristics) - [Proposed Change: FOLIO Item Upsert on Every Adapter Run](#proposed-change-folio-item-upsert-on-every-adapter-run) - [Fan-out Mechanism](#fan-out-mechanism) - - [Target Upsert/Mapping Architecture](#target-upsertemapping-architecture) + - [Target Upsert/Mapping Architecture](#target-upsertmapping-architecture) - [Transformation Design](#transformation-design) - [Approach](#approach) - [Transformation Pipeline](#transformation-pipeline) - - [YAML Mapper](#yaml-mapper-mappingyaml--yamlmapper) + - [Pydantic Mapping](#pydantic-mapping-mappingpy) - [Reference Data Cache](#reference-data-cache-ref_cachepy) + - [RefCache Resolution](#refcache-resolution) + - [Sample Field Mapping](#sample-field-mapping) + - [FOLIO Field Mapping Reference](#folio-field-mapping-reference) +- [Change Detection Mechanism](#change-detection-mechanism) + - [How It Works](#how-it-works) + - [Record selection: only harvest-flagged records](#record-selection-only-harvest-flagged-records) + - [Change Detection Signals](#change-detection-signals) + - [Every Record in a Changeset Is Either](#every-record-in-a-changeset-is-either) + - [Delete detection: the reconciler, not OAI tombstones](#delete-detection-the-reconciler-not-oai-tombstones) - [Proposed AWS Architecture](#proposed-aws-architecture) - [Key Design Considerations](#key-design-considerations) - [System Diagram](#system-diagram) - [1. EventBridge Trigger](#1-eventbridge-trigger) - [2. Step Function State Machine](#2-step-function-state-machine) - - [RefCache Resolution](#refcache-resolution) - - [Sample Field Mapping](#sample-field-mapping) - - [FOLIO Field Mapping Reference](#folio-field-mapping-reference) - - [Change Detection Mechanism](#change-detection-mechanism) - [3. Lambda Function: axiell-folio-sync](#3-lambda-function-axiell-folio-sync) - [Upsert Key Strategy (Idempotency)](#upsert-key-strategy-idempotency) - [4. S3 Manifest Storage](#4-s3-manifest-storage) - [Design Rationale: Why These Choices?](#design-rationale-why-these-choices) + - [Mapping: Pydantic Models vs. YAML Mapper](#mapping-pydantic-models-vs-yaml-mapper) - [Orchestration: Step Functions vs. Alternatives](#orchestration-step-functions-vs-alternatives) - [Storage: S3 NDJSON vs. Alternatives](#storage-s3-ndjson-vs-alternatives) - - [Invocation Pattern: Synchronous vs. Asynchronous](#invocation-pattern-synchronous-vs-asynchronous) + - [Invocation Pattern: EventBridge Trigger & Ordering](#invocation-pattern-eventbridge-trigger--ordering) - [Error Handling: Per-Record Isolation](#error-handling-per-record-isolation) +- [SRS-backed Instances and the Update Path](#srs-backed-instances-and-the-update-path) - [FOLIO API Client](#folio-api-client) - [Cost Analysis](#cost-analysis) -- [Assumptions & Constraints](#assumptions--constraints) + - [Current Volume: ~1,000 records/day](#current-volume-1000-recordsday) - [Open Questions](#open-questions) + - [Field Mapping from Axc to Folio Instance, Holdings and Items needs to be defined](#field-mapping-from-axc-to-folio-instance-holdings-and-items-needs-to-be-defined) + - [Delete semantics: what should reconciler-detected deletes do in FOLIO?](#delete-semantics-what-should-reconciler-detected-deletes-do-in-folio) + - [Watermark storage: where does the source timestamp live?](#watermark-storage-where-does-the-source-timestamp-live) + - [Reference-data caching: when and how to cache across Lambda runs](#reference-data-caching-when-and-how-to-cache-across-lambda-runs) + - [Manifest query mechanism: S3 Select is not an established org pattern](#manifest-query-mechanism-s3-select-is-not-an-established-org-pattern) + - [Item creation should likely be gated on record `Level`](#item-creation-should-likely-be-gated-on-record-level) + - [Item notes not visible on the FOLIO OAI-PMH feed](#item-notes-not-visible-on-the-folio-oai-pmh-feed) - [Next Steps](#next-steps) - [References](#references) @@ -53,64 +64,56 @@ This is proposal to synchronize library location and item holdings from Axiell C ## Background -## Current CMS and LMS systems -The Wellcome Collection library systems currently operate as: -- **CALM** (Collections Management): Content Management System (CMS) — bibliographic and collection metadata source of truth -- **Sierra**: Library Management System (LMS) — patron management, circulation, holds, and requesting +The Wellcome Collection library systems are undergoing migration. They currently operate as distinct layers: **CALM** (Collections Management), the Content Management System (CMS), serves as the +collection metadata source of truth, while **Sierra**, the Library Management System (LMS), manages patron management, circulation, holds, and requesting. The CALM-to-Sierra harvester is the integration mechanism that synchronizes bibliographic metadata from CALM into Sierra for discovery and requesting workflows. -The CALM-to-Sierra harvester is the integration mechanism that synchronizes bibliographic metadata from CALM into Sierra for discovery and requesting workflows. +The library systems migration is replacing both layers. **CALM** is being migrated to **Axiell Collections**, and **Sierra** is being migrated to **FOLIO**. The current CALM-to-Sierra harvester will be superseded by an Axiell Collections-to-FOLIO integration pipeline. The data must be synchronized between the CMS and LMS so that the circulation of items and patron management can be carried out. -## Migration Context - The library systems migration is replacing these systems: -- **CALM → Axiell Collections**: Data from Calm is being migrated to Axiell Collections -- **Sierra → FOLIO**: Patron management, circulation, and requesting migrating to FOLIO +CALM-sourced data in Sierra is not being migrated to FOLIO. Therefore, AxC data must be synced to FOLIO to enable requesting and circulation workflows for items that were not included in the initial FOLIO migration. Library staff rely on FOLIO for real-time item availability and location data. Axiell-to-FOLIO sync is a core data pipeline for the new catalogue system and must be reliable, auditable, and manually recoverable. -The current CALM-to-Sierra harvester will be superseded by an Axiell Collections-to-FOLIO integration pipeline. This document captures the existing harvester behavior for reference during the migration period. The data needs to be synched between the CMS and LMS so that the circulation of items and patron managment can be carried out. +## Scope -### Why This Matters -- Library staff rely on FOLIO for real-time item availability and location data -- Axiell-to-FOLIO sync is a core data pipeline for the new catalogue system -- Sync must be reliable, auditable, and manually recoverable +**In scope:** the idempotent upsert of AxC bibliographic records into FOLIO Inventory as instance → holdings → item, for records flagged for harvest in MARC `980 $a`. Records are keyed by GUID-based hrids (`AxC-{instance,holding,item}-{guid}`), created Inventory-native (`source = "FOLIO"`), updated in place, and suppressed on delete. -## System Architecture +**Out of scope:** patron data and circulation transactions (loans, holds, requests); real-time sync (the 15-minute adapter cadence is sufficient, sub-minute latency is not required); multiple items per AxC record (a 1:1 instance-to-item mapping is assumed for now, see [Item creation should likely be gated on record `Level`](#item-creation-should-likely-be-gated-on-record-level)); updating SRS/MARC-backed instances (their bibliographic fields are owned by quickMARC/SRS, not mod-inventory, see [SRS-backed Instances and the Update Path](#srs-backed-instances-and-the-update-path)); and discovery indexing (that stays with the existing ES transformer, this pipeline writes only to the LMS). + +## System Architecture ### Current Data Feeds for AxC and Folio Data -- **Axiell Adapter Platform**: Runs on a 15-minute cadence, emitting changesets with new/modified/deleted records -- **FOLIO**: The new system of record for library management (items, holdings and patron data) -- **Integration Gap**: Changesets are written to Apache Iceberg tables on S3, but FOLIO is not automatically updated -- **Volume**: ~10–500 records per changeset (15-minute window); typical day: ~1,000 records across 80 syncs +**Axiell Collections (AxC)** is the source: the Axiell Adapter Platform harvests AxC over +OAI-PMH every 15 minutes and writes each record as raw MARCXML into a single Apache +Iceberg table on S3, emitting on every run a *changeset*, the records created, +modified, or deleted in that window, identified by `changeset_id`. **FOLIO** is the +new system of record for library management (instances, holdings, items, patrons) +and also exposes its own OAI-PMH feed every 15 minutes. -``` -EventBridge Scheduler - │ - ▼ -┌──────────┐ ┌──────────┐ ┌────────────────────┐ ┌─────────────────┐ -│ Trigger │───▶│ Loader │───▶│ Transformer (ES) │───▶│ Elasticsearch │ -└──────────┘ └──────────┘ └────────────────────┘ └─────────────────┘ - │ - ▼ - Iceberg Adapter Table -``` +**Volume:** 10–500 records per changeset, ~1,000 records/day across ~80 syncs. +#### Adapter pipeline + +```mermaid +flowchart TD + scheduler[EventBridge Scheduler] --> trigger[Trigger] + trigger --> loader[Loader] + loader --> transformer[Transformer (ES)] + loader --> reconciler[Reconciler] + transformer --> es[Elasticsearch] + loader --> iceberg[Iceberg Adapter Table] + iceberg --> transformer + iceberg --> reconciler +``` -| Stage | File | Role | -|-------|------|------| -| Trigger | `src/adapters/steps/oai_pmh/trigger.py` | Compute next harvest window from WindowStore history | -| Loader | `src/adapters/steps/oai_pmh/loader.py` | Harvest OAI-PMH records in window, write raw MARCXML to Iceberg, emit `changeset_ids` | -| Transformer | `src/adapters/steps/transformer.py` + `src/adapters/transformers/axiell_transformer.py` | Read changesets from Iceberg, parse MARCXML with pymarc, produce SourceWork, index to ES | -| Reconciler | `src/adapters/transformers/axiell_reconciler.py` | Track GUID→ID mapping changes, emit DeletedSourceWork for superseded identifiers | +| Stage | Role | +|-------|------| +| Trigger | Compute the next harvest window from WindowStore history | +| Loader | Harvest OAI-PMH records in the window, write raw MARCXML to Iceberg, emit `changeset_ids` | +| Transformer | Read changesets from Iceberg, parse MARCXML (pymarc) into a SourceWork, index to Elasticsearch | +| Reconciler | Track GUID→ID mapping changes; emit DeletedSourceWork for superseded identifiers | -#### Key Characteristics +#### Key characteristics -- **Metadata prefix**: `oai_marcxml` -- **OAI set**: `collect` -- **Auth**: Custom `Token` header (not standard `Authorization`) -- **Identity**: `axiell-guid` from MARC 001 -- **Visibility**: `InvisibleSourceWork` (MimsyWorksAreNotVisible) -- **Window cadence**: 15 min windows, 7 day lookback, 360 min max lag -- **FOLIO OAI-PMH feed**: FOLIO exposes an OAI-PMH feed that is available every 15 minutes as well -- **Storage**: Single Iceberg table schema (`namespace`, `id`, `content`, `changeset`, `last_modified`, `deleted`) +The Axiell adapter harvests over OAI-PMH using the `oai_marcxml` metadata prefix and the `collect` OAI set, authenticating with a custom `Token` header rather than the standard `Authorization` header. Record identity comes from the `axiell-guid` extracted from MARC `001`, while visibility is controlled by the `InvisibleSourceWork` flag (MimsyWorksAreNotVisible). Harvesting runs in 15-minute windows with a 7-day lookback and a 360-minute maximum lag. All records land in a single Iceberg table per adapter, whose schema is `namespace`, `id`, `content`, `changeset`, `last_modified`, and `deleted`. --- @@ -118,35 +121,26 @@ EventBridge Scheduler ### Fan-out Mechanism +```mermaid +flowchart TD + scheduler[EventBridge Scheduler] --> trigger[Trigger] + trigger --> loader[Loader] + loader --> transformer[Transformer (ES)] + transformer --> es[Elasticsearch] + loader --> iceberg[Iceberg Table] + loader --> upserter[FOLIO Upserter (new step)] + iceberg --> upserter + upserter --> folio[FOLIO Inventory] + + upserterSteps["1. Authenticate to FOLIO
2. Load reference data cache
3. Read changeset rows from Iceberg
4. Pydantic mapping (mapping.py) -> instance/holdings/item
5. Upsert: create/update/suppress"] + upserter -. process .-> upserterSteps ``` -EventBridge Scheduler - │ - ▼ -┌──────────┐ ┌──────────┐ ┌────────────────────┐ ┌─────────────────┐ -│ Trigger │───▶│ Loader │────┬───▶│ Transformer (ES) │───▶│ Elasticsearch │ -└──────────┘ └──────────┘ │ └────────────────────┘ └─────────────────┘ - │ │ - ▼ │ ┌──────────────────────────────────────────────┐ - Iceberg Table └───▶│ FOLIO Upserter (new step) │ - │ 1. Authenticate to FOLIO │ - │ 2. Load reference data cache │ - │ 3. Read changeset rows from Iceberg │ - │ 4. YAML mapper → instance/holdings/item │ - │ 5. Upsert: create · update · suppress │ - └──────────────────────────────────────────────┘ - │ - ▼ - ┌──────────────┐ - │ FOLIO │ - │ Inventory │ - └──────────────┘ -``` -After the loader emits `changeset_ids`, the event is routed to **two** downstream targets: -1. Existing Axiell ES transformer (unchanged). -2. New **FOLIO upserter step**. +The FOLIO upserter is best understood as a **new type of transformer** in the existing catalogue pipeline architecture. The architecture already has the concept of a transformer: a step that consumes a changeset from Iceberg and writes the records into a downstream sink. The current `AxiellTransformer` transforms MARCXML into a `SourceWork` and indexes it into **Elasticsearch** (the discovery sink). + +The FOLIO upserter follows the same pattern but targets a different sink, the **LMS (FOLIO)**, instead of Elasticsearch, and its write operation is an **idempotent upsert of item data** (instance → holdings → item) over the FOLIO OKAPI Inventory APIs rather than an index write. In other words: same input (`changeset_ids` over the same Iceberg table), new transformer, new destination. -Both consume the same `changeset_ids` independently. Either path can fail and retry without affecting the other. +So after the loader emits `changeset_ids`, the event fans out to **two** independent transformers: the existing ES transformer (`AxiellTransformer`, unchanged) writing to Elasticsearch for discovery, and the new FOLIO upserter writing to FOLIO Inventory for circulation and requesting in the LMS. Both consume the same `changeset_ids` independently, and either path can fail and retry without affecting the other, so adding the LMS sink does not put the existing discovery feed at risk. --- ### Target Upsert/Mapping Architecture @@ -165,7 +159,7 @@ flowchart LR folio_sync --> ssm[SSM SecureStrings
FOLIO credentials] folio_sync --> refcache[ref_cache.py
locations/material/loan types] folio_sync --> read_changes[Read Iceberg rows
by changeset_ids] - read_changes --> mapper[yaml_mapper.py + mapping.yaml
MARCXML -> instance/holdings/item] + read_changes --> mapper[mapping.py (Pydantic models)
MARCXML -> instance/holdings/item] mapper --> upsert[upsert.py
create/update/suppress] upsert --> okapi[(FOLIO OKAPI Inventory APIs)] ``` @@ -175,56 +169,150 @@ flowchart LR ### Approach -The upserter uses a **YAML-driven mapper** (`mapping.yaml` + `YamlMapper`) to convert raw MARCXML from AxC apaptor into FOLIO payloads. This separates all field mapping configuration from Python application code — the mapping file can be reviewed and adjusted independently. +The upserter converts raw MARCXML from the AxC adapter into FOLIO payloads using **typed Python with Pydantic models** (`mapping.py`). The three payloads (Instance, Holdings, and Item) are Pydantic models with `extra="forbid"`, so the field-mapping rules are ordinary, reviewable Python and the payloads are typed contracts: a malformed payload (missing/ill-typed required field, typo'd key) **fails at build time, before any OKAPI call**, instead of surfacing as a FOLIO 422 mid-batch. + +This replaces an earlier YAML-driven mapper (`mapping.yaml` + `YamlMapper`); see [Mapping: Pydantic Models vs. YAML Mapper](#mapping-pydantic-models-vs-yaml-mapper) for the trade-off. ### Transformation Pipeline ```mermaid graph TD - A["Raw MARCXML
(from Iceberg)"] --> B["YAML Mapper
Input: raw MARCXML record
Output: FOLIO item-level JSON"] - B --> C["Schema Validation
Validate required fields present
Check identifiers well-formed"] - C --> D["FOLIO API Payload Builder
Map validated output to FOLIO
Inventory API request shape"] + A["Raw MARCXML
(from Iceberg)"] --> B["parse_marcxml (mapper.py)
MARC_SOURCE table → CanonicalRecord"] + B --> C["build_instance/holdings/item (mapping.py)
map + RefCache resolve → typed Pydantic models"] + C --> D["Pydantic validation
required fields, types, no unknown keys (extra=forbid)"] + D --> E["model_dump(exclude_none=True)
JSON-ready instance / holdings / item dicts"] ``` -### YAML Mapper (`mapping.yaml` + `YamlMapper`) +### Pydantic Mapping (`mapping.py`) -The YAML mapper is the core transformation engine. It uses to declaratively map MARCXML fields to FOLIO instance/holdings/item JSON payloads. +`mapping.py` is the single home for everything that decides *what an Axiell record becomes in FOLIO*: the MARC source table, normalization tables, defaults, the hrid scheme, the typed payload contracts, and the field-by-field builders. -**Design benefits:** -- **Separation of concerns**: Mapping logic lives in `mapping.yaml`, not Python code -- **Non-programmer-friendly**: Library staff and metadata experts can review/adjust mappings without code review -- **Version control**: Changes to mapping are tracked in git history -- **Testability**: Mapping rules can be tested independently of the Python runtime +The flow runs in three steps. `parse_marcxml()` (in `mapper.py`) first extracts MARC fields via the `MARC_SOURCE` table (e.g. `title: 245$a`, `location_code: 852$b`, `barcode: 949$a`) into a typed `CanonicalRecord`, where MARC `001` is the Axiell GUID and the basis for every hrid. `build_instance/holdings/item()` then map that record into the `Instance`, `Holdings`, and `Item` Pydantic models, resolving reference data (location, material type, loan type, holdings source, instance type) through **RefCache** and applying defaults. Finally, Pydantic validates on construction (`extra="forbid"` rejects unknown keys) and `build_payloads()` returns `{"instance", "holdings", "item", "meta"}` as JSON-ready dicts, with `meta` carrying `source_id`, the three hrids, and `mapping_version`. -**Example workflow:** -1. Input: Raw MARCXML record from Iceberg (e.g., `...`) -2. Mapper applies rules: Extract subjects from `datafield[@tag='650']` → produces structured data -3. Output: JSON structure matching FOLIO Inventory API schema (e.g., `{"instanceTypeId": "...", "subjects": [...]}`) +This buys several things. Invalid payloads raise `MappingError`/`ValidationError` at build time, before any FOLIO write, so there are no round-trip 422s; the payload shape is an enforced typed contract (`extra="forbid"`), so a typo'd key is a build error rather than a silently dropped field; the rules, defaults, and contracts live in one reviewable Python module with full language expressiveness for normalization and conditionals; and every payload is stamped with `mapping_version` for traceability. -**Key files:** -- `mapping.yaml`: Declarative mapping rules for instance, holdings, item transformation -- `YamlMapper` class: Loads YAML, applies transformation rules, executes transformation on each record +**hrid scheme** (the idempotency key for every upsert): `AxC-instance-{guid}`, `AxC-holding-{guid}`, `AxC-item-{guid}`. `Instance.source = "FOLIO"`. The sync creates no linked SRS MARC record, so the instance is FOLIO-native and its bibliographic fields stay editable via mod-inventory. --- ### Reference Data Cache (`ref_cache.py`) -The reference cache maintains in-memory lookups for static FOLIO configuration data that rarely changes: -- **Locations**: location UUIDs and codes -- **Material types**: material type IDs (e.g., "Book", "Microfilm") -- **Loan types**: loan type IDs (e.g., "Can Circulate", "Reference") +The reference cache maintains in-memory lookups for static FOLIO configuration data that rarely changes. `RefCache.load()` fetches and indexes **six** reference sets: +- **Locations**: indexed by both code *and* name → UUID +- **Material types**: by name → UUID (e.g., "Books", "video recording") +- **Loan types**: by name → UUID (e.g., "Can Circulate", "Reference") +- **Holdings sources**: by name → UUID (e.g., "MARC") +- **Item note types**: by name → UUID (resolves the "Axiell location" note's `itemNoteTypeId`) +- **Instance types**: the resolved default type id (e.g., "text"), applied to every instance + +(It also retains the full record list per set, but those are used only by the `summary()` / introspection helpers, not by the mapping.) + +FOLIO APIs require UUIDs/IDs rather than human-readable names, but Axiell supplies names, so `ref_cache` translates names to FOLIO IDs; caching them avoids repeated API calls during a sync run, which helps both performance and resilience. + +**Workflow:** on Lambda startup, `ref_cache.load()` fetches from the FOLIO reference endpoints; the mapper then resolves names against the cache (e.g. `location_uuid = ref_cache.location["Wellcome Science"]`); and if a name isn't found, the error is logged and the record is marked for manual review. + +**Cache persistence & freshness:** `RefCache.load()` runs once at the start of each +invocation and is **not** carried across invocations. Reloading every run +mostly re-fetches identical data; the point is simply that doing so is trivially +cheap (about six reference-endpoint GETs, a few hundred KB, ~1–3 s on a job that runs +every ~15 minutes, ~80 syncs/day), negligible next to the per-record OKAPI upserts, +and needs no extra infrastructure: no DynamoDB/ElastiCache/S3 layer, no TTL, no +cache-invalidation logic. Reading fresh each run also means that on the rare occasion +a new code does appear it is picked up on the very next sync instead of failing as a +`MappingError`, but that is a bonus rather than the reason. -**Why needed:** -- FOLIO APIs require UUIDs/IDs, not human-readable names -- Axiell provides names; ref_cache translates to FOLIO IDs -- Caching avoids repeated API calls during a sync run (performance + resilience) +The trigger to revisit is reference-load latency genuinely dominating runtime (much +higher frequency or far larger reference sets); even then the order is +**warm-singleton + reload-on-miss** first (the pattern the FOLIO token already uses), +and an external cross-run cache only beyond that. -**Workflow:** -1. Lambda startup → `ref_cache.load()` fetches from FOLIO reference endpoints -2. Mapper uses cache: `location_uuid = ref_cache.location["Wellcome Science"]` -3. If location name not found → error logged, record marked for manual review +--- + +### RefCache Resolution + +All reference lookups (location, material type, loan type, holdings source, item note type, and the default instance type) are resolved at sync time via **RefCache**, which caches that FOLIO reference data. + +**Resolution process:** on Lambda startup RefCache initializes by querying the FOLIO reference APIs once per invocation; for each record the mapper looks codes up against the cached values (an in-memory, O(1) lookup); if a code isn't found the upsert fails with `MappingError` and the record is skipped; and the resolved UUIDs are embedded directly in the payload. + +**Current normalisations:** material types map from AxC `Object_category` to a FOLIO material-type name (e.g. `"archives"` → `"unspecified"`, audio → `"sound recording"`); locations map from an AxC hierarchical location code (e.g. `"215;B11;MR;84;3;7"`) to a FOLIO location UUID; loan types map from AxC `OrderingCodes` (e.g. `"Archives - Requestable"`) to a FOLIO loan type UUID; and the instance type is typically `"text"` or a domain-specific type. + + +--- + +Stamping each payload with `mapping_version` supports audit (comparing records created with different mapper versions), rollback (identifying the records affected by a mapping change by version), and metadata tracking (a historical record of the mapping logic used for each record). + +### Sample Field Mapping + +`CanonicalRecord` field = the attribute parsed from MARC via the `MARC_SOURCE` +table in `mapping.py`; the hrids are derived from the GUID (MARC `001`). -**Cache persistence:** Reloaded fresh on every Lambda invocation; no cross-invocation caching. +| MARC Source (Axiell) | `CanonicalRecord` field | FOLIO Target | Example | +|---------------------|------------------|--------------|----------| +| 001 (GUID) | `source_id` → `instance_hrid` | Instance `hrid` | `AxC-instance-4d8f1208-9812-4bb5-84ef-da436b22d9e2` | +| 001 (GUID) | `source_id` → `holdings_hrid` | Holdings `hrid` | `AxC-holding-4d8f1208-9812-4bb5-84ef-da436b22d9e2` | +| 001 (GUID) | `source_id` → `item_hrid` | Item `hrid` | `AxC-item-4d8f1208-9812-4bb5-84ef-da436b22d9e2` | +| 245$a | `title` | Instance `title` | Daniel Morley, an English Philosopher... | +| 852$b | `location_code` | Holdings `permanentLocationId` (+ Item `permanentLocation`) | `215;B11;MR;84;3;7` | +| 852$c | `call_number_prefix` | Holdings `callNumberPrefix` | `Arch`, `Ref` | +| 852$h | `call_number` | Holdings `callNumber` | (optional) | +| 852$j | `shelving_order` | Holdings `shelvingOrder` | (optional) | +| 949$a | `barcode` | Item `barcode` | (optional) | +| 949$c | `material_type_code` | Item `materialType.id` | `Archives - Non-digital` | +| 949$l | `loan_type_code` | Item `permanentLoanType.id` | `Archives - Requestable` | +| 876$p | `copy_number` | Item `copyNumber` | `copy 1`, `copy 2` | +| 876$t | `volume` | Item `volume` | `v.1`, `disc 1 of 2` | +| 856$u | `electronic_access_uri` | Item `electronicAccess[].uri` | (optional) | +| 852$b | `location_code` | Item `notes[]`, type `Axiell location` | `Axiell location: 215;B11;MR;84;3;7` | + +The AxC current location is therefore written to FOLIO twice: as the resolved `permanentLocation` UUID (for shelving/discovery) and, in human-readable form, as an item **note of type `Axiell location`**. On update, that note is refreshed from the incoming `852$b`, so the FOLIO item always reflects the latest AxC current location. (The note type name resolves to `itemNoteTypeId` via RefCache before the write; note that this note is not currently surfaced on the OAI-PMH feed, see [Item notes not visible on the FOLIO OAI-PMH feed](#item-notes-not-visible-on-the-folio-oai-pmh-feed).) + +--- + +### FOLIO Field Mapping Reference + +For a comprehensive and detailed mapping of all Axiell Collections fields to FOLIO Inventory API fields, see **[folio-axc-fields-mapping.md](folio-axc-fields-mapping.md)**. + +That document provides the complete field-by-field mapping for Instance, Holdings, and Item entities, the transformation rules and MARC source information, the required-versus-optional fields for FOLIO, the reference-data requirements (locations, material types, loan types), and the field validation rules and edge cases. + +--- + +## Change Detection Mechanism + +The FOLIO upsert step leverages the existing Axiell adapter's OAI-PMH change detection to determine which records to sync. + +### How It Works + +OAI-PMH datestamp windows drive the harvest: the Axiell adapter only returns records whose `last_modified` falls within `[window_start, window_end)`. Each harvest window produces a unique `changeset_id`, and the records written to Iceberg in that window are tagged with it. Those Iceberg rows carry `namespace` (record type, e.g. "location", "item"), `id` (external identifier), `content` (the raw MARCXML payload), `changeset` (the changeset ID from the adapter), `last_modified` (the OAI datestamp), and `deleted` (`true` when OAI returns a tombstone). The FOLIO upserter then receives the `changeset_ids` and reads only those rows: + +```python +changed_records = adapter_store.read_changed(changeset_ids) +``` + +### Record selection: only harvest-flagged records + +Only AxC MARC records that have the **harvest flag set in MARC field `980 $a`** are synced to FOLIO. Before mapping, the upserter filters each changeset row on `980 $a`, and records without the flag are skipped entirely (never created, updated, or suppressed in FOLIO). The flag is the source-side switch by which Collection Information controls which Axiell records flow into the LMS, so the sync covers a curated subset rather than the whole changeset. (This selection step is not yet in the prototype, which currently processes every changeset row; it is required behaviour.) + +### Change Detection Signals + +| Signal | Source | Meaning | +|--------|--------|----------| +| Harvest flag (`980 $a`) | AxC MARC record | Record is opted in for FOLIO sync; absent means skip | +| Record in changeset | OAI-PMH datestamp window | Record was created or modified in source | +| `deleted=true` | OAI tombstone | Unreliable; best-effort only, not the authoritative delete signal | +| Payload hash mismatch | XSL output comparison (optional) | FOLIO-relevant fields actually changed | +| Reconciler GUID remap | Axiell reconciler step | Authoritative delete: old GUID superseded; suppress its FOLIO records | + +### Every Record in a Changeset Is Either + +Each record in a loader changeset is treated as **new** (the first time this `id` appears in Iceberg), in which case it is created in FOLIO, or **updated** (an existing `id` with a newer `last_modified`), in which case it is updated in FOLIO. Deletions are handled on the reconciler path (see [Delete detection](#delete-detection-the-reconciler-not-oai-tombstones) below), not by directly suppressing on the loader's `deleted=true` flag. The exact delete action in FOLIO (suppress vs remove, and cascade scope) remains an open policy question (see [Delete semantics](#delete-semantics-what-should-deletedtrue-do-in-folio)). + +### Delete detection: the reconciler, not OAI tombstones + +Axiell's OAI tombstones (`deleted=true`) are unreliable, which is the very reason the adapter has a separate **reconciler** step. The reconciler tracks the `collectId → guid` mapping in its own Iceberg store and, when a `collectId` is remapped to a different work, emits a `DeletedSourceWork` for the superseded GUID. + +This has two consequences for the FOLIO upserter. First, the upserter consumes the **loader's changeset**, so it does not see reconciler-detected deletes, which are produced by a separate, later transformer step. Second, a reconciler delete is keyed by the **old (superseded) GUID**, not by the changeset row being processed. + +Suggested approach: have the **reconciler** step fan out a FOLIO suppression path that mirrors the loader's fan-out to the upsert path, reusing the existing reconciler rather than building a parallel `collectId → guid` mapping store. On a reconciler delete, suppress the FOLIO records for the old GUID (`AxC-instance-{old-guid}`) and cascade to its holdings and item, in line with the [Delete semantics](#delete-semantics-what-should-deletedtrue-do-in-folio) question raised earlier. The loader-changeset `deleted=true` can remain a best-effort secondary signal, but the reconciler fan-out is the authoritative delete path. --- @@ -232,22 +320,32 @@ The reference cache maintains in-memory lookups for static FOLIO configuration d ### Key Design Considerations -We propose a **Step Functions + Lambda + S3 architecture** with synchronous invocation, per-record error isolation, and 90-day audit retention. This approach balances **operational visibility**, **fault resilience**, and **simplicity**. +We propose a **Step Functions + Lambda + S3 architecture** with event-driven (asynchronous) invocation, per-record error isolation, and 90-day audit retention. This approach balances **operational visibility**, **fault resilience**, and **simplicity**. | Aspect | Choice | Primary Benefit | |--------|--------|-----------------| +| **Mapping** | Typed Pydantic models (`mapping.py`) | Build-time validation; fail fast before OKAPI, not as a FOLIO 422 | | **Orchestration** | AWS Step Functions | Configurable retries + rich execution history for audit | | **Compute** | Lambda (ECR container) | Stateless, scales to 0, integrates with Step Functions | -| **Data Storage** | S3 NDJSON manifests (90-day TTL) | Cost-efficient batch writes + queryable via S3 Select | +| **Data Storage** | S3 NDJSON manifests (90-day TTL) | Cost-efficient batch writes + queryable (download + `jq`, or optionally S3 Select) | | **Trigger** | EventBridge on adapter completion | Event-driven (not polling); decoupled from adapter | -| **Invocation** | Synchronous Lambda (via Step Function) | Natural backpressure; clear visibility on success/failure | +| **Invocation** | Async event → Step Function (`StartExecution`); Lambda runs synchronously within it | Decoupled from adapter; ordering via source-timestamp watermark + concurrency=1 | | **Error Handling** | Per-record isolation | Batch completes even if individual records fail | -**Expected outcomes:** -- Safe replay without data corruption (idempotent upserts via FOLIO HRIDs) -- Complete audit trail: every decision (create/update/suppress/skip) logged in S3 + CloudWatch -- Low operational overhead: ~$3–5/month for typical volume -- Clear mental model: no eventual consistency puzzles, ordered execution +**Expected outcomes:** replay is safe without data corruption (idempotent upserts via FOLIO HRIDs); there is a complete audit trail, with every decision (create/update/suppress/skip) logged in S3 and CloudWatch; operational overhead stays low at ~$3–5/month for typical volume; and the mental model remains clear, with no eventual-consistency puzzles and ordered execution. + +#### Scaling the reference cache (if the dataset grows) + +Reload-per-run assumes the reference set is small enough to bulk-load into Lambda memory cheaply (see [Reference Data Cache](#reference-data-cache-ref_cachepy)). If we ever need **much more** reference data (many more types, or large per-record lookups), the choice of caching architecture matters. Options, cheapest first: + +| Option | What it is | Best when | Cost / overhead | Freshness control | +|--------|-----------|-----------|-----------------|-------------------| +| **Warm singleton + reload-on-miss** | Module-global cache reused across warm invocations; fetch-and-memoize a code on a cache miss | Set still fits in memory; modest growth | None (in-process) | Self-healing on miss; lost on cold start | +| **S3 snapshot + scheduled refresher** | A cron (EventBridge) Lambda rebuilds a serialized snapshot (JSON/Parquet) from FOLIO every N hours; the sync Lambda reads it at startup (one GET) instead of N FOLIO calls | Whole set scanned each run; want to decouple sync from FOLIO availability | ~pennies/mo + one extra scheduled Lambda | Snapshot age = refresh cadence; pair with reload-on-miss | +| **DynamoDB lookup table** | Refresher populates `(type, code) → uuid`; sync Lambda `BatchGetItem`s only the codes a changeset needs | Set too big for memory, or only a subset is needed per run | On-demand $/read; native TTL | TTL / refresher cadence; pair with reload-on-miss | +| **ElastiCache (Redis)** | Shared in-memory cache across all invocations | Very large + hot lookups at high concurrency | Always-on (~$12+/mo), **requires VPC** (ENI cold-start + NAT to reach FOLIO/AWS APIs) | TTL | + +**Recommended progression:** stay on reload-per-run → **S3 snapshot + scheduled refresher** (cheap, no VPC, scales via columnar formats, decouples from FOLIO reference endpoints) → **DynamoDB** only once the set outgrows Lambda memory or you genuinely need selective point lookups → **ElastiCache** only for high-concurrency hot-lookup workloads that justify always-on infra and the VPC tax. In every tier, keep a **reload-on-miss fallback to FOLIO** so a newly-added code resolves on first sight rather than failing the record. --- @@ -316,6 +414,8 @@ flowchart TD } ``` +`dry_run` (default `false`) makes the Lambda resolve and **log** every planned create/update/suppress action and still write the manifests, but issue **no** FOLIO writes. It is the safe way to validate a changeset (or the initial backfill) before going live. `sample_limit` caps the number of records processed, for smoke-testing against dev FOLIO. + **IAM Permissions**: `states:StartExecution` on Step Function ARN --- @@ -326,23 +426,14 @@ flowchart TD **Type**: STANDARD **Flow**: -``` -Input (from EventBridge) - ↓ -[Task] Invoke Lambda (pass event detail) - ├─ Input path: $.detail - ├─ Output: result JSON with counts, manifest URIs, errors - │ - ├─ [Retry] - │ • MaxAttempts: 3 (configurable) - │ • BackoffRate: 2.0 - │ • IntervalSeconds: 2 - │ - └─ [Catch] - • States.TaskFailed → SyncFailed (terminal) - • Log to CloudWatch - ↓ -Output (Success) → SyncComplete +```mermaid +flowchart TD + input[Input from EventBridge] --> invoke[Task: Invoke Lambda] + invoke --> output[Output: counts, manifest URIs, errors] + invoke -. Retry: MaxAttempts=3, BackoffRate=2.0, IntervalSeconds=2 .-> invoke + invoke -->|States.TaskFailed| failed[SyncFailed (terminal)] + failed --> failedLog[Log to CloudWatch] + output --> success[SyncComplete] ``` **Execution History**: Stored in CloudWatch Logs (`/aws/states/axiell-folio-sync-sfn`, 30-day retention) @@ -353,119 +444,27 @@ Output (Success) → SyncComplete --- -### RefCache Resolution - -All reference fields (`permanentLocationId`, `instanceTypeId`, `materialTypeId`, `permanentLoanTypeId`) are resolved at sync time via **RefCache**, which caches FOLIO reference data (locations, material types, loan types, instance types). - -**Resolution Process**: -1. On Lambda startup, RefCache initializes by querying FOLIO reference APIs (one-time call per invocation) -2. For each record, mapper looks up codes against cached values (in-memory, O(1) lookup) -3. If code not found: upsert fails with `MappingError` and record is skipped -4. Resolved UUIDs are embedded in payload - -**Current Normalisations**: -- **Material type**: AxC `Object_category` → FOLIO material type (e.g., `"Archives - Non-digital"` → `"unspecified"`; audio → `"sound recording"`) -- **Location**: AxC hierarchical location code (e.g., `"215;B11;MR;84;3;7"`) → FOLIO location UUID -- **Loan type**: AxC `OrderingCodes` (e.g., `"Archives - Requestable"`) → FOLIO loan type UUID -- **Instance type**: Typically mapped to `"text"` or domain-specific type - -**Implementation**: Can be checked in the prototype - ----` - -Benefits: -- **A/B audit**: Compare records created with different mapper versions -- **Rollback**: If a mapping change introduces errors, identify affected records by version -- **Metadata tracking**: Historical record of mapping logic used for each record - -### Sample Field Mapping - -| MARC Source (Axiell) | mapping.yaml key | FOLIO Target | Example | -|---------------------|------------------|--------------|----------| -| 001 (GUID) | `external_id` | Instance/Item `hrid` | `axiell:PP/CJS/B.3/9` | -| 245$a | `title` | Instance title | Daniel Morley, an English Philosopher... | -| 852$b | `location_code` | Holdings permanent location | `215;B11;MR;84;3;7` | -| 852$c | `shelving_location` | Holdings shelving location | `Arch`, `Ref` | -| 852$h | `call_number` | Holdings call number | — | -| 949$a | `barcode` | Item barcode | — | -| 949$c | `item_type` | Item material type | `Archives - Non-digital` | -| 949$l | `loan_type` | Item loan type | `Archives - Requestable` | -| 876$p | `copy_number` | Item copy number | `copy 1`, `copy 2` | -| 876$t | `volume` | Item volume designation | `v.1`, `disc 1 of 2` | -| 856$u | `electronic_access_uri` | Electronic access URL | — | - ---- - -## FOLIO Field Mapping Reference - -For a comprehensive and detailed mapping of all Axiell Collections fields to FOLIO Inventory API fields, see **[folio-axc-fields-mapping.md](folio-axc-fields-mapping.md)**. - -This document provides: -- Complete field-by-field mapping for Instance, Holdings, and Item entities -- Transformation rules and MARC source information -- Required vs. optional fields for FOLIO -- Reference data requirements (locations, material types, loan types) -- Field validation rules and edge cases - ---- - -## Change Detection Mechanism - -The FOLIO upsert step leverages the existing Axiell adapter's OAI-PMH change detection to determine which records to sync. - -### How It Works - -1. **OAI-PMH Datestamp Windows**: Axiell adapter harvests records modified within a time window. Only records with `last_modified` within `[window_start, window_end)` are returned from OAI-PMH. - -2. **Changeset IDs**: Each harvest window produces a unique `changeset_id`. Records written to Iceberg in that window are tagged with it. - -3. **Iceberg Columns**: Records stored with: - - `namespace` — record type (e.g., "location", "item") - - `id` — external identifier - - `content` — JSON-serialized record - - `changeset` — changeset ID from adapter - - `last_modified` — timestamp from OAI datestamp header - - `deleted` — `true` if OAI returned a tombstone - -4. **FOLIO Upsert Input**: Lambda receives `changeset_ids` and reads only those rows: - ```python - changed_records = adapter_store.read_changed(changeset_ids) - ``` - -### Change Detection Signals - -| Signal | Source | Meaning | -|--------|--------|----------| -| Record in changeset | OAI-PMH datestamp window | Record was created or modified in source | -| `deleted=true` | OAI tombstone | Record was removed from source | -| Payload hash mismatch | XSL output comparison (optional) | FOLIO-relevant fields actually changed | -| Reconciler GUID remap | Axiell reconciler step | Old identity superseded, emit delete for old | - -### Every Record in Changeset Is Either: - -- **New** (first time this `id` appears in Iceberg) → create in FOLIO -- **Updated** (existing `id`, newer `last_modified`) → update in FOLIO -- **Deleted** (`deleted=true`) → suppress/remove in FOLIO - ---- - ### 3. Lambda Function: axiell-folio-sync **Docker Image**: ECR (`uk.ac.wellcome/axiell-folio-sync:TAG`) +**IAM & secrets (least privilege):** the Lambda's execution role needs `ssm:GetParameter` (+ KMS decrypt) for the OKAPI credential SecureString, read access to the Iceberg/S3 Tables data, `s3:PutObject` to the manifests bucket, `cloudwatch:PutMetricData`, and CloudWatch Logs write. FOLIO credentials live only in SSM as a SecureString and are never baked into the image or environment. + **Execution Steps**: #### Step 1: Authenticate with FOLIO + +Auth is delegated to the shared `folio-client` (see [FOLIO API Client](#folio-api-client)): a `FolioClient` is built with a credentials provider that reads SSM, and it logs in lazily on first use. ``` SSM Parameter Store (SecureString) Path: /axiell-folio-sync/okapi-creds Value: {"username": "service_account", "password": "..."} -POST /authn/login +FolioClient → POST /authn/login (lazy, on first request) x-okapi-tenant: wellcome Body: credentials Response: x-okapi-token (valid 24h) - → Reused for all subsequent API calls in this invocation + → Cached for the invocation; auto re-auth once on a 401 ``` #### Step 2: Scan Iceberg Changesets @@ -478,30 +477,37 @@ SELECT [namespace, id, content, changeset, last_modified, deleted] WHERE changeset IN ({changeset_ids from event}) Schema: - namespace string — e.g., "location", "item" - id string — external identifier from Axiell - content string — JSON-serialized record - changeset string — changeset ID from adapter - last_modified timestamp — when record changed - deleted boolean — true if marked deleted + namespace string e.g., "location", "item" + id string external identifier from Axiell + content string raw MARCXML payload from OAI-PMH + changeset string changeset ID from adapter + last_modified timestamp when record changed + deleted boolean true if marked deleted ``` #### Step 3: Map & Validate ```python -# Load YAML mapping rules (from mapping.yaml) -mapper = YamlMapper("mapping.yaml") +from axiell_folio_sync.mapping import build_payloads, MappingError +from axiell_folio_sync.ref_cache import RefCache + +# Load tenant reference data once per run (locations, material/loan types, etc.). +ref_cache = RefCache(folio_get).load() for row in iceberg_records: + # Record selection: only records flagged for harvest (MARC 980 $a) are synced. + # Unflagged records are skipped before mapping: no create/update/suppress. + if not has_harvest_flag(row["content"]): # 980 $a present + continue try: - record = json.loads(row['content']) - - # Build Instance, Holdings, Item payloads - instance_payload = mapper.build_instance_payload(record) - holdings_payload = mapper.build_holdings_payload(record) - item_payload = mapper.build_item_payload(record) - - # Validate required fields per FOLIO schema - payloads.append({...}) + # build_payloads() parses the MARCXML (mapper.parse_marcxml → CanonicalRecord), + # builds the typed Pydantic Instance/Holdings/Item models with RefCache + # resolution + defaults, and returns JSON-ready dicts. Pydantic validates on + # construction (extra="forbid"), so a malformed record fails here, before + # any OKAPI call, rather than as a FOLIO 422. + mapped = build_payloads(row["content"], ref_cache, deleted=bool(row["deleted"])) + # → {"instance": {...}, "holdings": {...}, "item": {...}, "meta": {...}} + # meta carries source_id, the three hrids, and mapping_version. + payloads.append(mapped) except MappingError as e: # Capture error, continue to next record errors.append({ @@ -514,51 +520,52 @@ for row in iceberg_records: #### Step 4: Upsert to FOLIO (Per Record) +Every entity is matched by its **hrid** (from `meta`) and created or updated (no +blind creates), which is what makes replay idempotent. On an existing record the +update is additionally gated by the **stale-write guard**: it applies only if the +incoming Axiell `last_modified` is strictly newer than the watermark on the FOLIO +record, otherwise the record is skipped (see [Invocation Pattern](#invocation-pattern-eventbridge-trigger--ordering)). + ``` -For each record: - Execute: Instance → Holdings → Item (dependencies) - +For each mapped record: + Execute in order: Instance → Holdings → Item (each references the prior id) + Instance: - If deleted=true: - GET /inventory/instances/{instance_id} - PUT with discoverySuppress=true - Action: "suppress" - Else: - GET /inventory/instances?query=(hrid=={hrid}) - If exists: PUT (update) - Else: POST (create) - Action: "create" or "update" - - Holdings (if Instance succeeded): - POST /holdings-storage/holdings (create linked to Instance) - Action: "create" - - Item (if Holdings succeeded): - POST /item-storage/items (create linked to Holdings) - Action: "create" + Resolve by hrid: GET /inventory/instances?query=(hrid=={instance_hrid}) + If exists: PUT /inventory/instances/{id} → "update" + Else: POST /inventory/instances → "create" + (some POSTs return 201 with an empty body → re-resolve id by hrid) + + Holdings (linked to the Instance id): + Resolve by hrid in /holdings-storage/holdings → PUT (update) or POST (create) + + Item (linked to the Holdings id): + Resolve by hrid in /inventory/items → PUT (update) or POST (create) + (note types are resolved to itemNoteTypeId before the write) + + Loader tombstones (`deleted=true`) are treated as advisory only: + • record and metric the signal for audit/debug + • do not perform suppress/remove from this path + • apply delete actions only from reconciler-generated delete events Per-record error handling: • HTTP 4xx/5xx → captured, batch continues - • All errors logged to CloudWatch AND accumulated + • All errors logged to CloudWatch AND accumulated in the S3 failures manifest ``` ### Upsert Key Strategy (Idempotency) -**Priority order for matching existing records**: +**Matching**: All entities are matched against existing FOLIO records by `hrid`: -1. **External source identifier**: Axiell GUID from MARC 001 (most stable) -2. **Barcode**: From transformed record (949$a) -3. **Composite fallback**: `axiell:{record_id}` +- Instance: `GET /inventory/instances?query=(hrid==AxC-instance-{guid})` +- Holdings: `GET /holdings-storage/holdings?query=(hrid==AxC-holding-{guid})` +- Item: `GET /inventory/items?query=(hrid==AxC-item-{guid})` -**Behavior**: -- **Found**: Update mutable fields, preserve FOLIO-internal metadata -- **Not found**: Create new item, link to holding/instance via location -- **Missing required fields**: Skip with structured error logged +The `hrid` is derived from the Axiell GUID (MARC 001), not the `collectId` (object number). Axiell reuses `collectId`s; keying on GUID prevents a reused id from overwriting the wrong FOLIO record. The reconciler emits `DeletedSourceWork` events when a GUID is superseded, so old FOLIO records are cleaned up automatically. -**Replay Safety**: -- Upserts are idempotent by external identifier -- Same changeset processed twice produces same outcome -- Manifest deduplication: check if changeset already processed successfully before running +**Behavior:** when a record is found, its mutable fields are updated while FOLIO-internal metadata is preserved; when it isn't found, a new record is created; and when required fields are missing, the record is skipped with a structured error logged. An update applies only if it passes the stale-write guard (incoming Axiell `last_modified` strictly newer than the watermark on the existing FOLIO record), so out-of-order or replayed changesets never overwrite newer state (see [Invocation Pattern](#invocation-pattern-eventbridge-trigger--ordering)). + +**Replay safety:** upserts are idempotent by `hrid`, so processing the same changeset twice produces the same outcome, and manifest deduplication checks whether a changeset was already processed successfully before running it again. #### Step 5: Batch & Write Manifests ```python @@ -611,6 +618,7 @@ cloudwatch.put_metric_data( {"MetricName": "RecordsCreated", "Value": metadata['counts']['created']}, {"MetricName": "RecordsUpdated", "Value": metadata['counts']['updated']}, {"MetricName": "RecordsSuppressed", "Value": metadata['counts']['suppressed']}, + {"MetricName": "RecordsSkipped", "Value": metadata['counts']['skipped']}, {"MetricName": "RecordsFailed", "Value": metadata['counts']['failed']}, ] ) @@ -622,16 +630,18 @@ return metadata ### 4. S3 Manifest Storage +The NDJSON-manifest output is **the existing catalogue-pipeline pattern**: the adapter platform already writes per-job NDJSON id manifests to S3, and this step deliberately reuses that convention (one record per line, an `.ids.ndjson` success file plus a `.ids.failures.ndjson` error file and a `.manifest.json` summary per job). Adopting it keeps downstream tooling, backup/recovery, and operational query patterns consistent with the rest of the pipeline. + **Bucket**: `axiell-folio-sync-manifests-{account-id}-{region}` **File Structure**: ``` manifests/ - ├─ adapter-job-xyz-20260622-10-15.ids.ndjson - ├─ adapter-job-xyz-20260622-10-15.ids.failures.ndjson - ├─ adapter-job-xyz-20260622-10-15.manifest.json - ├─ adapter-job-xyz-20260622-10-30.ids.ndjson - └─ ... + - adapter-job-xyz-20260622-10-15.ids.ndjson + - adapter-job-xyz-20260622-10-15.ids.failures.ndjson + - adapter-job-xyz-20260622-10-15.manifest.json + - adapter-job-xyz-20260622-10-30.ids.ndjson + - ... ``` **Sample Success Record** (NDJSON): @@ -644,7 +654,7 @@ manifests/ "hrid": "HRID-LOC-123" }, "holdings": { - "action": "create", + "action": "createupdate", "id": "hold-xyz-uvw" }, "item": { @@ -659,8 +669,8 @@ manifests/ ```json { "source_id": "location_item_456", - "error": "Mapping error: required field 'barcode' missing", - "detail": "YAML rule for barcode returned null", + "error": "Mapping error: missing 245$a (title)", + "detail": "Pydantic build raised MappingError before any OKAPI call", "type": "mapping" } ``` @@ -671,35 +681,32 @@ manifests/ ## Design Rationale: Why These Choices? -### Orchestration: Step Functions vs. Alternatives +### Mapping: Pydantic Models vs. YAML Mapper -| Approach | Pros | Cons | Cost/mo | -|----------|------|------|---------| -| **A: Direct Lambda** (EventBridge → Lambda, fire-and-forget) | Fewer services, lower overhead, fastest time-to-invocation | No built-in retry logic, no execution history, hard to extend, silent failures possible | ~$2 | -| **B: Step Functions + Lambda** ⭐ **CHOSEN** | Explicit retry policy + audit trail, easy to parallelize later, clear visibility into success/failure | Extra service, marginal latency added | ~$3–5 | -| **C: Async Queue** (EventBridge → SQS → Lambda workers) | Decouples producer/consumer, handles bursts, resilient to adapter restarts | Complex ordering semantics, harder to reason about, no clear success/failure signal, monitoring overhead | ~$10–15 | +| Approach | Pros | Cons | +|----------|------|------| +| **A: YAML mapper** (`mapping.yaml` + `YamlMapper`) | Mapping config separated from code; in principle non-programmers can read/adjust rules; changes are config-only | Errors surface only at runtime (often as a FOLIO 422); no type safety, so a typo'd key or wrong shape passes silently; a second engine (YAML schema + interpreter) to build, test, and document; awkward for normalization tables, per-field defaults, and composite hrids | +| **B: Typed Pydantic models** (`mapping.py`, chosen) | Validation at build time before any OKAPI call; payload shape is a typed contract (`extra="forbid"`); rules, defaults, and contracts in one reviewable module; full Python for normalization/conditionals; payloads stamped with `mapping_version` | Mapping changes need a Python edit + redeploy, not a config tweak; reading the rules assumes Python familiarity | -**Why we chose B (Step Functions + Lambda):** +We chose the typed Pydantic models (B). The decisive factor is *when* errors surface: with typed models a missing or ill-typed required field (or an unknown key, since the models are `extra="forbid"`) raises `MappingError`/`ValidationError` during the build step, before any OKAPI call, whereas the YAML approach deferred the same failures to a FOLIO 422 mid-batch that is far harder to trace. Making the payload a typed contract also means a typo'd key is a build error rather than a silently dropped field, so the mapping and the FOLIO contract cannot drift apart unnoticed. Beyond correctness, it collapses two engines into one: ordinary Python replaces a bespoke YAML schema and its interpreter, so there is less to build, test, and document, and the normalization tables, defaults, and hrid scheme live beside the models. Every payload is also stamped with `mapping_version`, so any record in FOLIO or a manifest traces back to the exact rules that produced it. -1. **Extension point for scale**: When volume grows beyond 500 records/event, we swap the Lambda invocation for a Step Function Map state (fan-out per changeset). Same manifest format, no breaking changes. +The YAML mapper's headline appeal, letting non-programmers edit mappings as config, every rule change still went through review, tests, and a deploy, so config-versus-code made little practical difference, while the absence of type safety let malformed output reach FOLIO and fail there. As the rules grew (normalization tables, per-field defaults, composite GUID-based hrids) they simply read more clearly as Python than as declarative YAML. -2. **Execution history = audit trail**: Every state transition is logged to CloudWatch. If a sync fails partway through, we know exactly where and why. No need to rebuild state from Lambda logs. +> The YAML mapper (`mapping.yaml` + `YamlMapper`) was the original prototype design and has been **superseded** by the Pydantic approach. Remaining `mapping.yaml` references elsewhere in this RFC are historical. -3. **Explicit retry policy**: Max attempts and backoff are declarative. Operations can tweak `max_retries` without code changes (configure in Step Function definition). +--- -4. **Synchronous invocation provides backpressure**: Adapter waits for sync to complete → prevents queue buildup if FOLIO is slow. If sync times out, adapter knows to retry. +### Orchestration: Step Functions vs. Alternatives -5. **Cost remains minimal**: At 1K records/day (~80 invocations), we're talking $0.50/month for Step Function state transitions. Not a material driver. +| Approach | Pros | Cons | Cost/mo | +|----------|------|------|---------| +| **A: Direct Lambda** (EventBridge → Lambda, fire-and-forget) | Fewer services, lower overhead, fastest time-to-invocation | No built-in retry logic, no execution history, hard to extend, silent failures possible | ~$2 | +| **B: Step Functions + Lambda** (chosen) | Explicit retry policy + audit trail, easy to parallelize later, clear visibility into success/failure | Extra service, marginal latency added | ~$3–5 | +| **C: Async Queue** (EventBridge → SQS → Lambda workers) | Decouples producer/consumer, handles bursts, resilient to adapter restarts | Complex ordering semantics, harder to reason about, no clear success/failure signal, monitoring overhead | ~$10–15 | -**Why not A (Direct Lambda)?** -- Silent failures: EventBridge doesn't tell us if the invocation succeeded or failed. -- No retry: If Lambda times out, there's no automatic retry. We'd have to implement our own (add complexity). -- No execution history: Debugging a failed sync means parsing Lambda logs. Step Function history is cleaner. +We chose Step Functions + Lambda (B): it buys an explicit retry policy and a full execution history. Every state transition is logged to CloudWatch, so a sync that fails partway is diagnosable from the execution history rather than by reconstructing state from Lambda logs, and max attempts and backoff are declarative, so operations can tune retries in the state-machine definition without code changes. The adapter triggers the sync asynchronously (`StartExecution`) and does not block; ordering safety comes from source-timestamp watermarking plus a Step Functions concurrency limit of 1 (see [Invocation Pattern](#invocation-pattern-eventbridge-trigger--ordering)), not from backpressure. The design also leaves a clean extension point for scale: when volume outgrows a single invocation we swap in a Step Functions `Map` state to fan out per changeset, with no change to the manifest format. The cost is immaterial: pennies a month in state transitions at ~80 syncs/day (~2,400/month). -**Why not C (Async Queue)?** -- Ordering: SQS doesn't guarantee processing order within a batch. We'd need to handle out-of-order upserts or add logic to sort before processing. -- Complexity: Workers need to coordinate (who's processing which changeset?). Adds operational burden. -- Cost: For current volume, async queue is cheaper per invocation, but the monitoring overhead (error tracking, retry logic, dead-letter queue management) makes it not worth it. +A direct EventBridge→Lambda trigger (A) was rejected because it gives no failure signal, no automatic retry, and no execution history, so debugging would mean parsing Lambda logs. An async SQS queue (C) is cheaper per invocation but doesn't guarantee processing order within a batch, needs workers to coordinate which changeset they own, and carries monitoring overhead (error tracking, retries, dead-letter management) that isn't justified at current volume. --- @@ -708,102 +715,64 @@ manifests/ | Approach | Pros | Cons | Cost/mo (90-day) | |----------|------|------|------------------| | **A: DynamoDB** (per-record writes, TTL) | Query-friendly, real-time dashboards possible, strong consistency | O(n) write cost, schema evolution painful, expensive for batches, not auditable | ~$20–50 | -| **B: S3 NDJSON** ⭐ **CHOSEN** | Batched writes (low cost), queryable (S3 Select), matches org patterns, audit via versioning, easy to compress/archive | Requires S3 Select for queries (not real-time), not ideal for high-cardinality point lookups | ~$1–2 | +| **B: S3 NDJSON** (chosen) | Batched writes (low cost), queryable (download + `jq`; Athena or S3 Select optional), matches the pipeline's NDJSON-manifest pattern, audit via versioning, easy to compress/archive | Requires S3 Select for queries (not real-time), not ideal for high-cardinality point lookups | ~$1–2 | | **C: Streaming** (Kinesis/Firehose → Parquet) | High throughput, good for ML pipelines, efficient compression | Overkill for current volume, adds infrastructure complexity, higher cost | ~$5–15 | -**Why we chose B (S3 NDJSON):** - -1. **Cost scales with volume**: Per-record writes scale O(n). At 5K records/day, you'd pay for 5K DynamoDB writes. With S3 NDJSON, we batch-write every 5K records → 1 PUT per 5K records. Cost dominance flips at ~100 records/day (already exceeded). +We chose S3 NDJSON (B), and write-scaling is the main driver: per-record DynamoDB writes scale O(n) with record count, whereas the upserter batches its manifest writes into one S3 PUT per run (up to 5,000 records per object). At the current ~1,000 records/day across ~80 syncs that is about one object per run, so the batched model stays flat as volume grows rather than scaling per record. It is also explicitly the existing pipeline pattern rather than a new one: the adapter platform already emits per-job **NDJSON manifests** to S3, so reusing that format (and S3 generally) keeps backup, recovery, query, and downstream tooling consistent across the pipeline. And it stays queryable after the fact without any ETL. **S3 Select** is one convenient option (e.g. `SELECT * FROM s3://bucket/manifest.ndjson WHERE errors > 0`, ~500 ms on a 100 KB file), but note S3 Select is **not currently used elsewhere in the org**, so we don't depend on it: because the manifests are plain NDJSON, a failed-record list is equally available by just downloading the object and piping it through `jq`/`grep`, or by pointing Athena at the prefix if ad-hoc SQL is ever wanted. S3 Select is a nicety, not a load-bearing part of the design; whether to adopt it is an open question (below). S3 versioning provides an audit trail (every write is a recoverable object version, with the metadata JSON tracking historical jobs), and after 90 days manifests archive cheaply to Glacier, something DynamoDB has no low-cost equivalent for. -2. **Matches existing patterns**: Axiell adapter platform already uses S3 for changesets. This keeps the operational model consistent (known backup, recovery, and query patterns). +DynamoDB (A) was rejected on fit rather than headline cost (at ~1,000 writes/day either store is cheap), because its per-record write model scales O(n), schema changes require a scan-and-rewrite, and there's no clean way to answer audit questions like "what was deleted last week?". Streaming via Kinesis/Firehose (C) is built for thousands of records per second (overkill for 10–500-record changesets) and adds buffer/flush operational complexity we don't need. -3. **Queryable post-hoc**: S3 Select allows SQL-like queries without ETL: - ```sql - SELECT * FROM s3://bucket/manifest.ndjson WHERE errors > 0 - ``` - Takes ~500ms on 100 KB file. Good enough for operational investigation. +--- -4. **Audit trail via versioning**: S3 versioning enabled → every write creates a new object version. If someone accidentally deletes a manifest, we can recover it. Metadata JSON tracks all historical jobs. +### Invocation Pattern: EventBridge Trigger & Ordering -5. **Easy to compress/archive**: After 90 days, we can archive NDJSON to Glacier for long-term audit retention (pennies/month). DynamoDB has no cheap archive option. +EventBridge publishes changeset events to Step Functions via `StartExecution` (asynchronous, fire-and-forget); the adapter does not block waiting for the sync to complete. Step Functions provides the execution visibility and retry policy, and all invocations (initial or retry) are decoupled from the adapter. -**Why not A (DynamoDB)?** -- Per-record cost: 5K records/day = 5K write units (on-demand) = ~$2.50/day = ~$75/month. S3 is ~$0.05/month for the same data. -- Schema evolution: If we add a new field to the record (e.g., `retry_count`), DynamoDB requires a scan-and-rewrite. S3 NDJSON: just add the field to new records. -- Not suitable for audit: While DynamoDB TTL works, there's no clean way to query "what was deleted from this table last week?" +However, concurrent or retried executions create an **ordering risk**: if two changesets arrive within ~45 s of each other, or if a failed changeset is retried while a newer one is in flight, a naive last-write-wins upsert could apply an older state after a newer one, corrupting the FOLIO record. Similarly, idempotent replays (re-running failed changesets after a fix) must be no-ops if a newer changeset has already updated the same record. -**Why not C (Streaming)?** -- Overkill: Typical changeset is 10–500 records. Kinesis + Firehose is designed for high-throughput (1000s records/sec). We'd be paying for infrastructure we don't use. -- Operational complexity: Firehose auto-flushes based on buffer size or timeout. Extra operational knowledge needed. +**Solution: Source-Timestamp Watermarking.** The upsert layer compares each incoming record's Axiell `last_modified` timestamp (from the source data) against a watermark stored on the FOLIO record. The write proceeds only if the incoming timestamp is strictly newer; otherwise the record is skipped, leaving the FOLIO state intact. Where exactly that watermark lives in FOLIO (an administrative note, a custom property, or a version field) is itself an open question; see [Watermark storage](#watermark-storage-where-does-the-source-timestamp-live). This ensures: +- Out-of-order concurrent executions apply updates in source-timestamp order, not invocation order. +- Replays (re-running failed changesets) are idempotent: an older changeset re-run after a newer one succeeds does not overwrite the newer state. +- No coupling between adapter and sync (the adapter remains fire-and-forget). ---- +**Concurrency control:** A Step Functions concurrency limit of 1 (via the state machine definition) is a secondary guard, preventing the GET–PUT gap between a query and an upsert; with a limit of 1, only one execution can be in the `Lambda` state at a time, so even if two changesets arrive within milliseconds, the second waits for the first to commit. This is loose (the adapter cadence is 15 minutes, so concurrency is naturally rare) but provides an extra safeguard against read-skew during retries. -### Invocation Pattern: Synchronous vs. Asynchronous +Combined, these two mechanisms ensure ordering safety without blocking the adapter or introducing a queue. -| Approach | Pros | Cons | -|----------|------|------| -| **A: Async** (EventBridge → Lambda, fire-and-forget) | Decoupled, adapter doesn't wait, low latency | No backpressure; if adapter emits 10 events in quick succession, Lambda might queue up (can overwhelm FOLIO); no failure signal | -| **B: Synchronous** ⭐ **CHOSEN** | Backpressure (adapter waits), clear success/failure, enables replay on failure | Adapter must wait ~45 sec for sync to complete before next event; if FOLIO is slow, adapter is blocked | +--- -**Why we chose B (Synchronous):** +### Error Handling: Per-Record Isolation -1. **Natural backpressure**: If FOLIO is slow or unavailable, Step Function times out → adapter pauses before emitting next event. No need for queue management. +The guiding policy is that one bad record must never halt the batch. -2. **Failure visibility**: If sync fails, EventBridge knows → can retry the same changeset. No silent failures. +Per-record errors are non-blocking. A mapping error (Pydantic validation, a missing required field, or an unresolved reference) is captured as `{"source_id": "...", "type": "mapping", "detail": "..."}` and an API error (FOLIO returns 4xx/5xx) as `{"source_id": "...", "type": "api", "detail": "..."}`; in either case the record is written to the S3 error manifest and processing moves on to the next record. Batch-level errors, by contrast, are blocking: an OKAPI auth failure, an Iceberg connection failure, or an S3 write failure raises an exception that the Step Function retries up to `max_retries`, and if those are exhausted the execution ends in the terminal `SyncFailed` state and raises an alert (see the alert path below). -3. **Replay support**: If sync fails partway (e.g., 100 of 200 records succeeded before timeout), we can: - - Query S3 manifest to see which records failed - - Fix the issue (e.g., restart FOLIO, fix mapping rule) - - Re-trigger sync with same changeset_ids - - Idempotent upserts (via FOLIO HRID lookup) prevent duplication +Failures are observable across four channels: CloudWatch Logs (`/aws/lambda/axiell-folio-sync`) for detailed per-record execution, the S3 `.ids.failures.ndjson` manifest for a queryable error list (via a plain download + `jq`, or optionally S3 Select; see the storage note), CloudWatch Metrics (`RecordsFailed`, `RecordsCreated`, and the like) for alerting on a high failure rate, and the Step Function history (`/aws/states/axiell-folio-sync-sfn`) for state transitions and retry events. -4. **Adapter is designed for it**: Axiell adapter runs on 15-min cycles. Waiting 45 sec for sync fits within the cadence. +**Skipped records are also accounted for, not dropped silently.** Records skipped before a write (no `980 $a` harvest flag, `Level` gating, or the stale-write guard) are counted in a `RecordsSkipped` metric and recorded in the run's metadata manifest, so "why didn't record X appear in FOLIO?" is answerable after the fact rather than being an invisible no-op. -**Why not A (Async)?** -- Queue buildup: If adapter emits 10 changesets in quick succession, and each sync takes 45 sec, we'd have a 7-minute backlog. Async doesn't naturally handle this (we'd need manual queue monitoring). -- Failure not visible: If a sync fails, the adapter doesn't know. No built-in retry or alerting. +**Alert path:** alerting follows the existing pipeline pattern: a **CloudWatch metric alarm** (on `RecordsFailed`/the failure rate, and on `SyncFailed` Step Function executions) fires into the team's **Slack** channel via **Amazon Q Developer in chat applications**. No new notification mechanism is introduced; the alarms are wired to the same SNS-topic → Amazon Q → Slack route already used elsewhere in the catalogue pipeline. --- -### Error Handling: Per-Record Isolation - -**Policy**: One bad record must not halt the entire batch. +## SRS-backed Instances and the Update Path -**Error Levels**: +A consideration worth making explicit for the update path. Some FOLIO instances are backed by **Source Record Storage (SRS)** as MARC, for example anything migrated or loaded as MARC. For those records the bibliographic fields are controlled by the underlying SRS MARC record and **cannot be updated through the mod-inventory instance API**; only administrative data is editable there. MARC edits go through **quickMARC**, which writes to SRS and syncs the Inventory record. A `PUT` to mod-inventory may therefore fail or be silently ignored for an SRS-backed instance, whereas the current update logic assumes the PUT takes effect. -1. **Per-record errors** (non-blocking): - - Mapping error (YAML validation fails): `{"source_id": "...", "type": "mapping", "detail": "..."}` - - API error (FOLIO returns 4xx/5xx): `{"source_id": "...", "type": "api", "detail": "..."}` - - Action: Log to S3 error manifest, continue to next record +**which storage do we create records in, **Inventory-native** (FOLIO source) or **MARC/SRS**?** +Records created the two ways behave differently on update, so a mixed estate is harder to reason about and maintain. This sync currently creates Inventory-native instances (`Instance.source = "FOLIO"`, with no linked SRS record), which keeps the bibliographic fields editable through mod-inventory and keeps the PUT-based update path valid. -2. **Batch-level errors** (blocking): - - Auth failure (OKAPI login fails): Exception raised → Step Function retries up to `max_retries` - - Iceberg connection failure: Exception raised → Step Function retries - - S3 write failure: Exception raised → Step Function retries - - Action: If retries exhausted → SyncFailed (terminal state), alert operations - -**Observable via**: -- **CloudWatch Logs**: `/aws/lambda/axiell-folio-sync` (detailed per-record execution) -- **S3 Manifests**: `.ids.failures.ndjson` (queryable error list via S3 Select) -- **CloudWatch Metrics**: `RecordsFailed`, `RecordsCreated`, etc. (can alert on high failure rate) -- **Step Function History**: `/aws/states/axiell-folio-sync-sfn` (state transitions, retry events) +**Catalogue-pipeline impact:** the catalogue pipeline harvests FOLIO over OAI-PMH using the `marc21_withholdings` prefix (see `catalogue_graph/src/adapters/extractors/oai_pmh/folio/config.py`). Under that prefix the instance bib comes from SRS when an SRS record is present, or is generated on the fly from Inventory depending on the **mod-oai-pmh record-source** setting, while holdings and items come from Inventory. So whether the records this sync creates appear in that feed, and in what form, depends on the storage type we choose together with the mod-oai-pmh configuration. This has now been confirmed via the prototype for Inventory-native records: a `source = "FOLIO"` instance created here is updatable through mod-inventory and is received on the FOLIO adapter in the catalogue pipeline. One gap remains under investigation, the item notes not appearing on the OAI-PMH feed; see [Item notes not visible on the FOLIO OAI-PMH feed](#item-notes-not-visible-on-the-folio-oai-pmh-feed). --- ## FOLIO API Client -**Location**: `prototypes/axiell-folio-sync/axiell_folio_sync/upsert.py` +OKAPI traffic goes through the shared **`folio-client`** package (`prototypes/folio-client/`), the same dependency-free OKAPI client used by the folio-api Lambda and the CLI scripts. It is pure standard library (`urllib` + `ssl`), so it adds no third-party dependency to the bundle, and the build vendors its source into the image. The Lambda declares it as a path dependency (`[tool.uv.sources]` in `axiell-folio-sync/pyproject.toml`) and wires it up in `axiell_folio_sync.py`: a `FolioClient` is constructed with a `credentials_provider` (the SSM lookup) and `ssl_context_from_env()`, then a thin adapter (`_make_folio_callables`) wraps it into the `folio_get` / `folio_post` / `folio_put` callables that `ref_cache.py` and `upsert.py` consume. -**Responsibilities**: -- Authenticate to FOLIO via OKAPI (`/authn/login`), token refreshed automatically on 401 -- Resolve existing records by CQL query (GET before PUT) -- Create (POST), update (PUT), or suppress deleted records per entity -- Handle rate limiting and retries +**Responsibilities:** the client authenticates to FOLIO via OKAPI (`/authn/login`) and refreshes the token automatically on a 401; the upsert layer (`upsert.py`) resolves existing records by CQL query (a GET before each write) and creates (POST), updates (PUT), or suppresses deleted records per entity. -**Token Management**: -- Initial auth: `POST /authn/login` with credentials from SSM -- Token lifetime: 24 hours (sufficient for 5-min Lambda execution) -- Reused across all API calls in single invocation (no refresh overhead) +**Token management:** `FolioClient` logs in lazily on the first request using credentials from SSM, caches the resulting token (valid 24 hours, comfortably longer than the 5-minute Lambda execution) for the whole invocation (and across warm invocations on the same instance), and re-authenticates once automatically on a 401, so there is no per-call refresh overhead. --- @@ -812,36 +781,56 @@ manifests/ ### Current Volume: ~1,000 records/day +At ~80 syncs/day the pipeline runs ~2,400 times/month, so the monthly quantities below derive from that: + | Service | Operation | Qty/mo | Rate | Cost/mo | |---------|-----------|--------|------|---------| -| Lambda | 80 invocations × 300s × 1 GB | 80 invocations | $0.0000167/GB-s | ~$1.20 | -| Step Functions | 80 state transitions | 80 transitions | $0.000025/transition | ~$0.02 | -| EventBridge | 80 events | 80 events | $1/M events | ~$0.00 | -| S3 (manifests) | ~80 objects written, 90-day retention | 80 objects + storage | $0.023/K objects + $0.023/GB/mo | ~$1.50 | -| CloudWatch Logs | ~80 × 5 KB = 400 KB/month | 400 KB ingested | $0.50/GB ingested | ~$0.20 | +| Lambda | ~2,400 invocations × ~60s × 512 MB | ~2,400 invocations | $0.0000167/GB-s | ~$1.20 | +| Step Functions | ~2,400 state transitions | ~2,400 transitions | $0.000025/transition | ~$0.06 | +| EventBridge | ~2,400 events | ~2,400 events | $1/M events | ~$0.00 | +| S3 (manifests) | ~2,400 objects written, 90-day retention | ~2,400 objects + storage | $0.005/K PUTs + $0.023/GB/mo | ~$1.50 | +| CloudWatch Logs | ~2,400 × 5 KB ≈ 12 MB/month | 12 MB ingested | $0.50/GB ingested | ~$0.20 | | **Total** | | | | **~$3–5** | -## Assumptions & Constraints - -- **FOLIO HRID requirement**: Records must have a stable, unique HRID per type (Instance, Holdings, Item) for idempotent upserts. Axiell must provide this in mapping.yaml. -- **No real-time requirements**: 15-minute cadence is acceptable. If sub-minute sync becomes required, architecture must change (streaming Kinesis, real-time database). -- **FOLIO API stability**: Assumes FOLIO API is available and stable. If FOLIO is down for hours, sync manifests will accumulate (90-day retention allows manual replay after recovery). -- **Axiell stability**: Assumes Axiell adapter completes consistently. If adapter fails, no changeset is emitted → no sync triggered (not our problem, but should monitor). - --- ## Open Questions ### Field Mapping from Axc to Folio Instance, Holdings and Items needs to be defined -- Sample file **[folio-axc-fields-mapping.md](folio-axc-fields-mapping.md)**, File shared with Collection Information for feedback -- Minimum Stub Record needs to be defined. -- Source of data Marc/Folio for inventory types. Test with both the options +The field mapping still needs to be finalised with Collection Information; the sample file **[folio-axc-fields-mapping.md](folio-axc-fields-mapping.md)** has been shared with them for feedback. A minimum stub record also needs to be defined, and the source of data (MARC vs FOLIO) for inventory types remains open and should be tested with both options. + +### Delete semantics: what should reconciler-detected deletes do in FOLIO? + +Raised here for an answer, not settled. In FOLIO there are different operations: *suppress* (`discoverySuppress=true`, optionally `staffSuppress=true`) hides the record while preserving it and its links, so it is reversible and audit-friendly; *remove* (a hard `DELETE`) deletes it outright, which is irreversible, and FOLIO refuses to delete a parent that still has dependents. As a starting point we'd **suggest `discoverySuppress=true`** (preserve-and-hide), but that is a recommendation to confirm with Collection Information, not a decision. A second part of the question is whether the action **cascades** from instance → holdings → item or acts on the item only; this interacts with the [1:1-vs-multi-item question](#item-creation-should-likely-be-gated-on-record-level), since an instance shared by other items must not be suppressed or removed when one item is deleted. + +Specific points to resolve with Collection Information: whether suppressed items should remain queryable by staff (i.e. whether to also set `staffSuppress`); how an item-level deletion behaves when its holdings or instance still have other dependents; whether deletion should propagate in either direction (item → holdings → instance, or the reverse); and the retention policy for suppressed records in the audit log. + +### Watermark storage: where does the source timestamp live? +The stale-write guard (see [Invocation Pattern](#invocation-pattern-eventbridge-trigger--ordering)) needs the Axiell `last_modified` of the last applied change stored **on the FOLIO record** so the next write can compare against it. FOLIO does not offer an obvious home for this: `_version` is mod-inventory's optimistic-locking counter (not a source timestamp), and `discoverySuppress` is a boolean. Candidates are an **administrative note**, a **custom property**, or a dedicated field, each with trade-offs for visibility, OAI-PMH leakage, and whether it survives a quickMARC/SRS round-trip. The storage location needs to be decided (and confirmed not to pollute the catalogue feed) before the guard can be implemented. + +### Reference-data caching: when and how to cache across Lambda runs + +Today `RefCache` reloads all six reference sets on every invocation, which is the right trade-off at current volume (see [Reference Data Cache](#reference-data-cache-ref_cachepy)). The open question is what happens if the reference set grows substantially. We would need to decide the size or load-latency threshold at which caching across runs is worth the added complexity. Staleness is unlikely to be the blocker: the reference data changes rarely, so a long TTL would seldom be stale, and a reload-on-miss fallback would keep a newly-added code from failing as a `MappingError`. So the question is mostly about whether the saved reload justifies the extra infrastructure. We would also need to pick a tier when we get there: the progression runs from a warm-singleton with reload-on-miss, to an S3 snapshot rebuilt by a scheduled refresher, to DynamoDB, and only then to ElastiCache (see [Scaling the reference cache](#scaling-the-reference-cache-if-the-dataset-grows) for the trade-offs). + +### Manifest query mechanism: S3 Select is not an established org pattern + +The design mentions **S3 Select** as a convenient way to query the NDJSON manifests (e.g. list failed records), but S3 Select is **not used anywhere else in the org today**, so adopting it would be a new pattern to learn, permission, and maintain. The design deliberately does not depend on it: the manifests are plain NDJSON, so the same questions can be answered by downloading the object and using `jq`/`grep`, or by pointing Athena at the manifests prefix. Open question: do we standardise on a query mechanism (plain download + local tooling, S3 Select, or Athena/Glue), and is it worth introducing S3 Select solely for this pipeline? Until decided, treat S3 Select as optional and lead with download + `jq` for operational investigation. + +### Item creation should likely be gated on record `Level` + +The legacy CALM→Sierra transform (`docs/discovery/CalmInnopac.xsl`) only attaches an item (its MARC `949`) when the record's `Level` is "Item"; higher levels (Collection, Series, File, …) get a bib record with no item. The current prototype instead builds an instance, holdings, **and** item for every record, regardless of level. We need to confirm with Collection Information whether the AxC → FOLIO mapping should branch the same way (item only for Item-level records, instance/holdings-only above that) and, if so, how `Level` arrives in the harvested MARCXML so the mapper can read it. This also backs the **1:1 instance-to-item** assumption: `CalmInnopac.xsl` produces at most one item per record (no `for-each` over copies/locations, and a multi-location attempt was left disabled), so multiple items per record is not established behaviour and would be a deliberate future extension rather than something to match on day one. + +### Item notes not visible on the FOLIO OAI-PMH feed + +Confirmed via the prototype: FOLIO-native instances (`source = "FOLIO"`) created by this sync **can** be updated through mod-inventory, and the records **are** received on the FOLIO adapter in the catalogue pipeline (so the storage-type and catalogue-feed concerns are largely settled for Inventory-native records). One issue remains open: the item **notes list does not appear on the OAI-PMH feed** (`marc21_withholdings`). The records otherwise come through, but the notes we set (for example the `Axiell location` note) are missing from the harvested output. This needs further investigation, likely into how mod-oai-pmh renders Inventory item notes in the MARC output and whether a note type, `staffOnly`, or suppression setting hides them. + +--- ## Next Steps -- Prototype adding another lambda in the step function for moving the read of data from iceberg and write of data to Folio in separare lambdas and check on the approach +- **Initial backfill / cutover (to be worked out).** The design above handles the ongoing 15-minute changeset flow, but the first task is populating FOLIO with the **entire existing harvest-flagged (`980 $a`) corpus**, not a single window. We need to work out how the one-off initial load runs (replay all historical changesets, or a full Iceberg scan filtered on the harvest flag), how it is validated against FOLIO, how it is throttled so it does not overwhelm OKAPI, and how it sequences with the switch-over to the steady-state event-driven sync. --- @@ -850,4 +839,4 @@ manifests/ - **Axiell Adapter Platform**: Emits changesets to Iceberg; documentation in location-movement-control-docs - **FOLIO API**: https://api-wellcome.folio.ebsco.com (OKAPI auth required) - **AWS S3 Tables**: Iceberg catalog on S3; managed via Terraform -- **YAML Mapping Rules**: mapping.yaml (stored in Lambda container or S3) +- **Mapping rules**: `mapping.py`, the typed Pydantic Instance/Holdings/Item models + builders (bundled in the Lambda image) diff --git a/rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md b/rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md index 1910f2a4..d27d59a2 100644 --- a/rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md +++ b/rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md @@ -1,70 +1,84 @@ -# FOLIO Minimum Fields Mapping (AxC → FOLIO) +# AxC → FOLIO Field Mapping -**Date:** 2026-06-17 -**Status:** Reference guide for AxC to FOLIO data integration -**Related:** [axc-folio-field-mapping.md](axc-folio-field-mapping.md) · [axiell-to-folio-upsert.md](axiell-to-folio-upsert.md) -**Code:** `prototypes/axiell-folio-sync/axiell_folio_sync/mapper.py` +**Date:** 2026-06-30 +**Status:** Reference guide for AxC to FOLIO data integration +**Code:** `prototypes/axiell-folio-sync/axiell_folio_sync/mapping.py` (typed Pydantic +models + builders, the source of truth) · `mapper.py` (MARC extraction via the +`MARC_SOURCE` table) · `ref_cache.py` (reference-data resolution) + +> The payloads are **Pydantic models with `extra="forbid"`** (`mapping.py`), so only +> the fields listed as *(in model)* are emitted today; an unmodelled or typo'd key +> fails at build time. Rows marked *(not in model)* are recommended future additions. +> Every hrid is derived from the **Axiell GUID** (MARC `001`); `mapping_version` is +> stamped on each payload (currently `2.1.0`). --- ## Instance -Minimum recommended fields to create a discoverable, functional instance record in FOLIO Inventory. +`Instance` model: `hrid`, `title`, `source` (default `"FOLIO"`), `instanceTypeId`. -| FOLIO Field | Type | Status | OIM Field (AxC) | MARC Tag | Notes | -|-------------|------|--------|-----------------|----------|-------| -| `hrid` | string | Required | `object_number` | `001` | Hierarchical record identifier. Format: `axiell:{source_id}`. Example: `axiell:PP/CJS/B.3/9` | -| `title` | string | Required | `Titles/title` | `245$a` | Primary title from object. Essential for discoverability; fails API if absent. Example: `Daniel Morley, an English Philosopher of the XIIth Century. Isis, 1920, iii, 263-9` | -| `source` | string | Required | — | — | Hardcoded to `"Axiell"` to mark origin system. | -| `instanceTypeId` | UUID | Required | — | — | Resolved from RefCache. Typically `"text"` or domain type (e.g., `"unspecified"` for mixed media). | -| `formerIds` | array | Recommended | `guid` + `Alternative_number` (Calm RefNo) | `035$a` + `244$a` | Preserve alternate identifiers for reference and migration. Includes AxC GUID and legacy Calm RefNo. Example: `["axiell:PP/CJS/B.3/9", "PPCJS/B/3/9"]` | -| `discoverySuppress` | boolean | Recommended | `accession_status` | — | Set `true` for RESTRICTED items or draft records. Default `false` for OPEN items. | +| FOLIO Field | Type | Status | AxC source | MARC Tag | Notes | +|-------------|------|--------|------------|----------|-------| +| `hrid` | string | Required *(in model)* | `guid` | `001` | Format `AxC-instance-{guid}`. Example: `AxC-instance-4d8f1208-9812-4bb5-84ef-da436b22d9e2` | +| `title` | string | Required *(in model)* | `Titles/title` | `245$a` | Fails the build with `MappingError` if absent. | +| `source` | string | Required *(in model)* | — | — | Hardcoded `"FOLIO"`. Inventory-native (no linked SRS MARC record), so bibliographic fields stay editable via mod-inventory. | +| `instanceTypeId` | UUID | Required *(in model)* | — | — | Resolved from RefCache; defaults to the `"text"` type. | +| `formerIds` | array | Recommended *(not in model)* | `guid` + `Alternative_number` | `035$a` + `244$a` | Preserve alternate identifiers for reference/migration. | +| `discoverySuppress` | boolean | Recommended *(not in model)* | `accession_status` | — | Today suppression is applied only to items on delete. | --- ## Holdings -Minimum recommended fields to link instance to location and call numbers. +`Holdings` model: `hrid`, `instanceId`, `sourceId`, `permanentLocationId`, `callNumber`, `callNumberPrefix`, `shelvingOrder`. -| FOLIO Field | Type | Status | OIM Field (AxC) | MARC Tag | Notes | -|-------------|------|--------|-----------------|----------|-------| -| `hrid` | string | Required | `object_number` + `location.default.name` | `001` + `852$b` | Composite identifier linking record to location. Format: `axiell:{source_id}-holding-{location_slug}`. Example: `axiell:PP/CJS/B.3/9-holding-215-b11-mr-84-3-7` where `location_slug` is lowercased, hyphenated. | -| `instanceId` | UUID | Required | — | — | FOLIO instance UUID. Resolved after instance POST succeeds. | -| `permanentLocationId` | UUID | Required | `location.default.name` | `852$b` | Hierarchical shelf location from AxC. Resolved from RefCache using location code string. Example: `215;B11;MR;84;3;7` → UUID lookup. | -| `callNumber` | string | Recommended | — | `852$h` | Call number suffix. Optional but aids patron retrieval. | -| `callNumberPrefix` | string | Optional | — | `852$c` | Call number prefix or classification. Example: `"Arch"`, `"Ref"`. | -| `shelvingOrder` | string | Optional | — | `852$j` | Sort/shelving sequence for physical ordering on shelf. | -| `formerIds` | array | Recommended | `guid` + `Alternative_number` | `035$a` + `244$a` | Preserve AxC identifiers. Example: `["axiell:PP/CJS/B.3/9"]` | -| `discoverySuppress` | boolean | Recommended | `accession_status` | — | Set `true` if holdings should not appear in public discovery. Inherits from instance or may override. | +| FOLIO Field | Type | Status | AxC source | MARC Tag | Notes | +|-------------|------|--------|------------|----------|-------| +| `hrid` | string | Required *(in model)* | `guid` | `001` | Format `AxC-holding-{guid}`. | +| `instanceId` | UUID | Required *(in model)* | — | — | Injected by the upsert orchestrator after the instance resolves. | +| `sourceId` | UUID | Required *(in model)* | — | — | Holdings source; resolved from RefCache; default `"MARC"` (`DEFAULT_HOLDINGS_SOURCE`). | +| `permanentLocationId` | UUID | Required *(in model)* | `location` | `852$b` | Resolved from RefCache by code/name; default `"History of Medicine"`. Example: `215;B11;MR;84;3;7` → UUID. | +| `callNumber` | string | Optional *(in model)* | — | `852$h` | Call number suffix. | +| `callNumberPrefix` | string | Optional *(in model)* | — | `852$c` | Example: `"Arch"`, `"Ref"`. | +| `shelvingOrder` | string | Optional *(in model)* | — | `852$j` | Shelving sequence. | +| `formerIds` / `discoverySuppress` | — | Recommended *(not in model)* | — | — | Future additions. | --- ## Item -Minimum recommended fields to represent the physical/digital object in circulation. - -| FOLIO Field | Type | Status | OIM Field (AxC) | MARC Tag | Notes | -|-------------|------|--------|-----------------|----------|-------| -| `hrid` | string | Required | `object_number` | `001` | Item record identifier. Format: `axiell:{source_id}`. Example: `axiell:PP/CJS/B.3/9` | -| `holdingsRecordId` | UUID | Required | — | — | FOLIO holdings UUID. Resolved after holdings POST succeeds. | -| `status.name` | string | Required | — | — | Hardcoded to `"Available"` for initial sync. Valid values: `"Available"`, `"Awaiting pickup"`, `"Checked out"`, etc. | -| `materialTypeId` | UUID | Required | `Object_category/object_category` | `949$c` | Material type (format) of the item. Resolved from RefCache. Example: `Archives - Non-digital` → `unspecified` UUID. See material type normalisation table. | -| `permanentLoanTypeId` | UUID | Required | `Free_texts` (type=`OrderingCodes`) | `949$l` | Loan policy. Resolved from RefCache. Example: `"Archives - Requestable"` → loan type UUID. | -| `barcode` | string | Optional | — | `949$a` | Machine-readable barcode. Used for physical item tracking. May be absent for archival items. | -| `copyNumber` | string | Optional | — | `876$p` | Distinguishes multiple copies of the same item. Example: `"copy 1"`, `"copy 2"`. | -| `volume` | string | Optional | — | `876$t` | Volume designation for multi-part works. Example: `"v.1"`, `"disc 1 of 2"`. | -| `formerIds` | array | Recommended | `guid` | `035$a` | Preserve AxC GUID for reference. Example: `["axiell:4d8f1208-9812-4bb5-84ef-da436b22d9e2"]` | -| `discoverySuppress` | boolean | Recommended | `accession_status` | — | Set `true` for RESTRICTED items. Hides item from public search results. | +`Item` model: `hrid`, `holdingsRecordId`, `status`, `materialType` `{id}`, `permanentLoanType` `{id}`, `permanentLocation` `{id}`, `barcode`, `copyNumber`, `volume`, `electronicAccess`, `notes`. + +| FOLIO Field | Type | Status | AxC source | MARC Tag | Notes | +|-------------|------|--------|------------|----------|-------| +| `hrid` | string | Required *(in model)* | `guid` | `001` | Format `AxC-item-{guid}`. | +| `holdingsRecordId` | UUID | Required *(in model)* | — | — | Injected by the upsert orchestrator after the holdings resolves. | +| `status.name` | string | Required *(in model)* | — | — | Defaults to `"Available"` (`status` is a nested object, `default_factory=Status`). | +| `materialType.id` | UUID | Required *(in model)* | `Object_category` | `949$c` | Inventory `{ "id": "" }` reference. RefCache + `MATERIAL_TYPE` table; default `"Books"`. Example: `"archives"` → `"unspecified"`. | +| `permanentLoanType.id` | UUID | Required *(in model)* | `Free_texts` (`OrderingCodes`) | `949$l` | `{ "id": "" }`. RefCache; default `"Can Circulate"`. | +| `permanentLocation.id` | UUID | Required *(in model)* | `location` | `852$b` | `{ "id": "" }`; same location as holdings. RefCache; default `"History of Medicine"`. | +| `barcode` | string | Optional *(in model)* | — | `949$a` | May be absent for archival items. | +| `copyNumber` | string | Optional *(in model)* | — | `876$p` | Example: `"copy 1"`. | +| `volume` | string | Optional *(in model)* | — | `876$t` | Example: `"v.1"`. | +| `electronicAccess[].uri` | array | Optional *(in model)* | — | `856$u` | Omitted when no `856$u`. | +| `notes[]` (type `Axiell location`) | array | Optional *(in model)* | `location` | `852$b` | One note `{ note: "Axiell location: ", noteType: "Axiell location", staffOnly: false }`. The AxC current location in human-readable form; **refreshed from `852$b` on every update**. `noteType` resolves to `itemNoteTypeId` during upsert. | +| `formerIds` / `discoverySuppress` | — | Recommended *(not in model)* | — | — | Future additions; on delete the upsert sets `discoverySuppress` and `staffSuppress` on the existing item. | --- ## RefCache Resolution -All reference fields (`permanentLocationId`, `instanceTypeId`, `materialTypeId`, `permanentLoanTypeId`) are resolved at sync time via **RefCache**, which caches FOLIO reference data (locations, material types, loan types, instance types). If a code cannot be resolved, the upsert fails with `MappingError`. +All reference lookups (location, material type, loan type, holdings source, item note type, and the default instance type) are resolved at sync time via **RefCache** (`ref_cache.py`), which loads that FOLIO reference data once per run. Resolution is case-insensitive: each raw AxC value is normalised (optional lookup table), falls back to a default, then resolves to a UUID; if the resolved name is unknown to the tenant, the build fails with `MappingError`. + +**Defaults** (used when the MARC record carries no value): material type `"Books"`, loan type `"Can Circulate"`, location `"History of Medicine"`, holdings source `"MARC"`. The item location note always uses note type `"Axiell location"` (`AXIELL_LOCATION_NOTE_TYPE`). + +**Material-type normalisations** (`MATERIAL_TYPE` table, AxC `Object_category` → FOLIO material-type name, case-insensitive): +- `"archives"` → `"unspecified"` +- `"published material"` → `"Books"` +- `"sound only"` / `"audio-visual material - e-sound only"` → `"sound recording"` +- `"audio-visual material - visual"` / `"audio-visual material - e-visual only"` → `"video recording"` -**Current normalisations:** -- **Material type:** AxC `Object_category` → FOLIO material type (e.g., `"Archives - Non-digital"` → `"unspecified"`; audio → `"sound recording"`). -- **Location:** AxC hierarchical location code (e.g., `"215;B11;MR;84;3;7"`) → FOLIO location UUID. -- **Loan type:** AxC `OrderingCodes` (e.g., `"Archives - Requestable"`) → FOLIO loan type UUID. +Location and loan type resolve directly by code/name (no normalisation table): a location code such as `"215;B11;MR;84;3;7"` and an `OrderingCodes` value such as `"Archives - Requestable"` are looked up against the cached FOLIO records. -See `prototypes/axiell-folio-sync/axiell_folio_sync/ref_cache.py` for lookup implementation. +See `mapping.py` for the `MARC_SOURCE` table, `MATERIAL_TYPE`, and the `DEFAULT_*` constants, and `ref_cache.py` for the lookup implementation. diff --git a/rfcs/090-axiell-folio-sync/mapping.py b/rfcs/090-axiell-folio-sync/mapping.py new file mode 100644 index 00000000..e3865b80 --- /dev/null +++ b/rfcs/090-axiell-folio-sync/mapping.py @@ -0,0 +1,302 @@ +""" +AxC → FOLIO mapping: payload models, normalization rules, and builders. + +This is the single home for everything that decides *what an Axiell record +becomes in FOLIO*. It replaces the former mapping.yaml + yaml_mapper.py pair: +the rules are now ordinary Python and the payloads are typed Pydantic models, +so a malformed payload (missing/ill-typed required field, typo'd key) fails at +build time — before any OKAPI call — instead of surfacing as a FOLIO 422. + +Everything a reviewer needs to understand or change the mapping lives here: + + • MATERIAL_TYPE AxC source code → FOLIO standard material-type name + • DEFAULTS / hrid scheme module-level constants + • Instance/Holdings/Item Pydantic payload contracts + • build_instance/holdings/item the field-by-field mapping logic + • build_payloads orchestration → the dict shape upsert consumes + +MARC *extraction* (which subfield holds what) stays in mapper.parse_marcxml, +which yields the CanonicalRecord these builders read from. + +Output contract is unchanged from the YAML mapper: ``build_payloads`` returns +``{"instance": {...}, "holdings": {...}, "item": {...}, "meta": {...}}`` ready +for :func:`axiell_folio_sync.upsert.upsert_from_payloads`. +""" + +from __future__ import annotations + +import re +from typing import Callable, Optional + +from pydantic import BaseModel, ConfigDict, Field + +from .mapper import CanonicalRecord, MappingError, extract, parse_xml +from .ref_cache import RefCache + +# Bumped whenever the rules below change; stamped into every payload's meta. +VERSION = "2.1.0" + + +# ── inbound: which MARC field feeds which CanonicalRecord attribute ───────────── +# +# This table is the single source of truth for the Axiell/MARC *source* side. +# Spec syntax: "TAG$subfield" for datafields, "TAG" for controlfields. +MARC_SOURCE: dict[str, str] = { + "source_id": "001", # Axiell GUID — identifies the record + "title": "245$a", + "location_code": "852$b", + "call_number": "852$h", + "call_number_prefix": "852$c", + "shelving_order": "852$j", + "barcode": "949$a", + "material_type_code": "949$c", + "loan_type_code": "949$l", + "copy_number": "876$p", + "volume": "876$t", + "electronic_access_uri": "856$u", +} + + +def _safe_segment(value: str) -> str: + """Lowercase slug from an arbitrary string — used in composite hrids.""" + return re.sub(r"[^a-z0-9]+", "-", value.strip().lower()).strip("-") or "unknown" + + +def parse_marcxml(xml_content: str, *, deleted: bool = False) -> CanonicalRecord: + """Parse a single MARCXML string into a :class:`CanonicalRecord` via MARC_SOURCE. + + Raises :class:`MappingError` if MARC 001 is absent (record cannot be identified). + """ + root = parse_xml(xml_content) + values = {field: extract(root, spec) for field, spec in MARC_SOURCE.items()} + + source_id = values["source_id"] + if not source_id: + raise MappingError("Missing MARC 001 — cannot identify record") + + instance_hrid = f"axiell:{source_id}" + holdings_hrid = ( + f"{instance_hrid}-holding-{_safe_segment(values['location_code'] or 'unknown')}" + ) + return CanonicalRecord( + instance_hrid=instance_hrid, + holdings_hrid=holdings_hrid, + deleted=deleted, + **values, + ) + + +# ── normalization tables & defaults ───────────────────────────────────────────── + +# Axiell source code → FOLIO standard material-type name. Case-insensitive keys. +MATERIAL_TYPE: dict[str, str] = { + "sound only": "sound recording", + "audio-visual material - visual": "video recording", + "audio-visual material - e-sound only": "sound recording", + "audio-visual material - e-visual only": "video recording", + "published material": "Books", + "archives": "unspecified", +} + +# Fallbacks used when the MARC record carries no value for a resolved field. +DEFAULT_MATERIAL_TYPE = "Books" +DEFAULT_LOAN_TYPE = "Can Circulate" +DEFAULT_LOCATION = "History of Medicine" +DEFAULT_HOLDINGS_SOURCE = "MARC" +AXIELL_LOCATION_NOTE_TYPE = "Axiell location" + + +def _instance_hrid(source_id: str) -> str: + return f"AxC-instance-{source_id}" + + +def _holdings_hrid(source_id: str) -> str: + return f"AxC-holding-{source_id}" + + +def _item_hrid(source_id: str) -> str: + return f"AxC-item-{source_id}" + + +# ── payload models (the FOLIO contract) ───────────────────────────────────────── + +class IdRef(BaseModel): + """A FOLIO {"id": ""} reference object.""" + id: str + + +class Status(BaseModel): + name: str = "Available" + + +class Note(BaseModel): + # noteType is resolved to itemNoteTypeId later by upsert._resolve_item_note_types. + model_config = ConfigDict(extra="allow") + note: str + noteType: Optional[str] = None + staffOnly: bool = False + + +class ElectronicAccess(BaseModel): + uri: str + + +class Instance(BaseModel): + model_config = ConfigDict(extra="forbid") # guard against typo'd keys in our code + hrid: str + title: str + source: str = "FOLIO" + instanceTypeId: str + + +class Holdings(BaseModel): + model_config = ConfigDict(extra="forbid") + hrid: str + instanceId: Optional[str] = None # injected by the upsert orchestrator + sourceId: str + permanentLocationId: str + callNumber: Optional[str] = None + callNumberPrefix: Optional[str] = None + shelvingOrder: Optional[str] = None + + +class Item(BaseModel): + model_config = ConfigDict(extra="forbid") + hrid: str + holdingsRecordId: Optional[str] = None # injected by the upsert orchestrator + status: Status = Field(default_factory=Status) + materialType: IdRef + permanentLoanType: IdRef + permanentLocation: IdRef + barcode: Optional[str] = None + copyNumber: Optional[str] = None + volume: Optional[str] = None + electronicAccess: Optional[list[ElectronicAccess]] = None + notes: Optional[list[Note]] = None + + +# ── lookup helper (mirrors the YAML map → default → lookup chain) ──────────────── + +def _resolve( + resolver: Callable[[Optional[str]], Optional[str]], + raw: Optional[str], + *, + label: str, + default: str, + table: Optional[dict[str, str]] = None, +) -> str: + """Normalize a raw AxC code, fall back to ``default``, then resolve to a UUID. + + Raises MappingError if the resolved name is unknown to the FOLIO tenant. + """ + value = (raw or "").strip() + if table: + value = table.get(value.lower(), value) + if not value: + value = default + uuid = resolver(value) + if uuid is None: + raise MappingError( + f"Unresolved {label} {value!r} — add it to the FOLIO tenant or fix the MARC" + ) + return uuid + + +# ── builders (the mapping) ────────────────────────────────────────────────────── + +def build_instance(rec: CanonicalRecord, ref: RefCache) -> Instance: + if not rec.title: + raise MappingError(f"Missing 245$a (title) for source_id={rec.source_id}") + return Instance( + hrid=_instance_hrid(rec.source_id), + title=rec.title.strip(), + # FOLIO-native: we create the instance via the Inventory API with no + # linked SRS MARC record, so the source is FOLIO, not MARC. + source="FOLIO", + instanceTypeId=ref.instance_type_id(), + ) + + +def build_holdings(rec: CanonicalRecord, ref: RefCache) -> Holdings: + return Holdings( + hrid=_holdings_hrid(rec.source_id), + sourceId=_resolve( + ref.resolve_holdings_source, DEFAULT_HOLDINGS_SOURCE, + label="holdings source", default=DEFAULT_HOLDINGS_SOURCE, + ), + permanentLocationId=_resolve( + ref.resolve_location, rec.location_code, + label="location", default=DEFAULT_LOCATION, + ), + callNumber=rec.call_number, + callNumberPrefix=rec.call_number_prefix, + shelvingOrder=rec.shelving_order, + ) + + +def build_item(rec: CanonicalRecord, ref: RefCache) -> Item: + location_id = _resolve( + ref.resolve_location, rec.location_code, + label="location", default=DEFAULT_LOCATION, + ) + return Item( + hrid=_item_hrid(rec.source_id), + materialType=IdRef(id=_resolve( + ref.resolve_material_type, rec.material_type_code, + label="material type", default=DEFAULT_MATERIAL_TYPE, table=MATERIAL_TYPE, + )), + permanentLoanType=IdRef(id=_resolve( + ref.resolve_loan_type, rec.loan_type_code, + label="loan type", default=DEFAULT_LOAN_TYPE, + )), + permanentLocation=IdRef(id=location_id), + barcode=rec.barcode, + copyNumber=rec.copy_number, + volume=rec.volume, + electronicAccess=( + [ElectronicAccess(uri=rec.electronic_access_uri)] + if rec.electronic_access_uri else None + ), + notes=[Note( + note=f"Axiell location: {rec.location_code or 'unknown'}", + noteType=AXIELL_LOCATION_NOTE_TYPE, + staffOnly=False, + )], + ) + + +# ── orchestration ─────────────────────────────────────────────────────────────── + +def build_payloads( + xml_content: str, + ref_cache: RefCache, + *, + deleted: bool = False, +) -> dict: + """Parse MARCXML and build the three FOLIO payloads as JSON-ready dicts. + + Drop-in replacement for ``YamlMapper.build_payloads`` — returns the same + ``{"instance", "holdings", "item", "meta"}`` shape that + :func:`axiell_folio_sync.upsert.upsert_from_payloads` consumes. + + Raises :class:`MappingError` for required-field violations or unresolved lookups. + """ + rec = parse_marcxml(xml_content, deleted=deleted) + + instance = build_instance(rec, ref_cache) + holdings = build_holdings(rec, ref_cache) + item = build_item(rec, ref_cache) + + return { + "instance": instance.model_dump(exclude_none=True), + "holdings": holdings.model_dump(exclude_none=True), + "item": item.model_dump(exclude_none=True), + "meta": { + "source_id": rec.source_id, + "instance_hrid": _instance_hrid(rec.source_id), + "holdings_hrid": _holdings_hrid(rec.source_id), + "item_hrid": _item_hrid(rec.source_id), + "mapping_version": VERSION, + "deleted": deleted, + }, + } diff --git a/rfcs/090-axiell-folio-sync/mapping.yaml b/rfcs/090-axiell-folio-sync/mapping.yaml deleted file mode 100644 index bcdf089b..00000000 --- a/rfcs/090-axiell-folio-sync/mapping.yaml +++ /dev/null @@ -1,86 +0,0 @@ -# AxC → FOLIO field mapping -# Version is stamped into every produced payload for traceability. -# Field rules: -# from_marc: "TAG$subfield" — extract first matching subfield value -# from_marc: "TAG" — extract control field value (no $) -# default: — use this literal if from_marc is absent or empty -# lookup: — resolve value via RefCache (location|materialType|loanType|instanceType) -# map: — apply value mapping from mapping_tables section -# template: "" — interpolate {source_id}, {instance_hrid}, {holdings_hrid} -# as_list: true — wrap result in a JSON array -# as_electronic_access: true — wrap URI as [{uri: }] -# transforms: [trim|upper|lower] -# required: true — MappingError raised if value is null after resolution -# Fields with no value after all rules are omitted from the payload (not null). - -version: "1.0.0" - -mapping_tables: - # Material type normalization: Axiell source codes → FOLIO standard names - material_type: - "sound only": "sound recording" - "audio-visual material - visual": "video recording" - "audio-visual material - e-sound only": "sound recording" - "audio-visual material - e-visual only": "video recording" - "published material": "Books" - # Archives variants default to "unspecified" (handled via pattern matching below) - "archives": "unspecified" - -identity: - # MARC controlfield used as the source record identifier - marc_controlfield: "001" - # Prefix prepended to form the instance hrid: AxC:{source_id} - hrid_prefix: "AxC" - -instance: - fields: - hrid: - template: "AxC:{source_id}" - title: - from_marc: "245$a" - transforms: [trim] - required: true - source: - default: "AxC" - instanceType.id: - lookup: instanceType - required: true - -holdings: - fields: - hrid: - template: "AxC:{source_id}-holding-{location_slug}" - # FOLIO requires holdings sourceId; resolve it by name via RefCache. - sourceId: - default: "MARC" - lookup: holdingsSource - required: true - permanentLocationId: - from_marc: "852$b" - default: "History of Medicine" - lookup: location - required: true - -item: - fields: - hrid: - template: "AxC:{source_id}" - materialType.id: - from_marc: "949$c" - map: material_type - default: "Books" - lookup: materialType - required: true - permanentLoanType.id: - from_marc: "949$l" - default: "Can Circulate" - lookup: loanType - required: true - permanentLocation.id: - from_marc: "852$b" - default: "History of Medicine" - lookup: location - required: true - status: - default: - name: "Available" From fbbce5596f3190c58677b830603712f103ad1468 Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Tue, 30 Jun 2026 12:35:33 +0100 Subject: [PATCH 07/15] Fix mermaid diagrams and clarify Axiell Location --- rfcs/090-axiell-folio-sync/README.md | 103 +++++++++++++-------------- 1 file changed, 48 insertions(+), 55 deletions(-) diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index 3d20d843..be9a3c8b 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -10,7 +10,6 @@ This RFC proposes an automated pipeline to synchronize data from **Axiell Collec - [Purpose](#purpose) - [Background](#background) -- [Scope](#scope) - [System Architecture](#system-architecture) - [Current Data Feeds for AxC and Folio Data](#current-data-feeds-for-axc-and-folio-data) - [Proposed Change: FOLIO Item Upsert on Every Adapter Run](#proposed-change-folio-item-upsert-on-every-adapter-run) @@ -71,12 +70,6 @@ The library systems migration is replacing both layers. **CALM** is being migrat CALM-sourced data in Sierra is not being migrated to FOLIO. Therefore, AxC data must be synced to FOLIO to enable requesting and circulation workflows for items that were not included in the initial FOLIO migration. Library staff rely on FOLIO for real-time item availability and location data. Axiell-to-FOLIO sync is a core data pipeline for the new catalogue system and must be reliable, auditable, and manually recoverable. -## Scope - -**In scope:** the idempotent upsert of AxC bibliographic records into FOLIO Inventory as instance → holdings → item, for records flagged for harvest in MARC `980 $a`. Records are keyed by GUID-based hrids (`AxC-{instance,holding,item}-{guid}`), created Inventory-native (`source = "FOLIO"`), updated in place, and suppressed on delete. - -**Out of scope:** patron data and circulation transactions (loans, holds, requests); real-time sync (the 15-minute adapter cadence is sufficient, sub-minute latency is not required); multiple items per AxC record (a 1:1 instance-to-item mapping is assumed for now, see [Item creation should likely be gated on record `Level`](#item-creation-should-likely-be-gated-on-record-level)); updating SRS/MARC-backed instances (their bibliographic fields are owned by quickMARC/SRS, not mod-inventory, see [SRS-backed Instances and the Update Path](#srs-backed-instances-and-the-update-path)); and discovery indexing (that stays with the existing ES transformer, this pipeline writes only to the LMS). - ## System Architecture ### Current Data Feeds for AxC and Folio Data @@ -128,11 +121,11 @@ flowchart TD loader --> transformer[Transformer (ES)] transformer --> es[Elasticsearch] loader --> iceberg[Iceberg Table] - loader --> upserter[FOLIO Upserter (new step)] + loader --> upserter[FOLIO Upserter] iceberg --> upserter upserter --> folio[FOLIO Inventory] - upserterSteps["1. Authenticate to FOLIO
2. Load reference data cache
3. Read changeset rows from Iceberg
4. Pydantic mapping (mapping.py) -> instance/holdings/item
5. Upsert: create/update/suppress"] + upserterSteps["1. Authenticate
2. Load ref cache
3. Read changesets
4. Map: mapping.py
5. Upsert: create/update/suppress"] upserter -. process .-> upserterSteps ``` @@ -147,21 +140,21 @@ So after the loader emits `changeset_ids`, the event fans out to **two** indepen ```mermaid flowchart LR - scheduler[EventBridge Scheduler
15-minute cadence] --> trigger[Trigger step] - trigger --> loader[Loader step
OAI-PMH harvest] - loader --> iceberg[(Iceberg adapter table)] - loader --> event[changeset_ids event] + scheduler[EventBridge
15-min cadence] --> trigger[Trigger] + trigger --> loader[Loader
OAI-PMH] + loader --> iceberg[(Iceberg)] + loader --> event[changeset_ids] - event --> es_transformer[Transformer step
AxiellTransformer] + event --> es_transformer[Transformer
AxiellTransformer] es_transformer --> es[(Elasticsearch)] - event --> folio_sync[Lambda: axiell-folio-sync
prototypes/axiell-folio-sync] - folio_sync --> ssm[SSM SecureStrings
FOLIO credentials] - folio_sync --> refcache[ref_cache.py
locations/material/loan types] - folio_sync --> read_changes[Read Iceberg rows
by changeset_ids] - read_changes --> mapper[mapping.py (Pydantic models)
MARCXML -> instance/holdings/item] - mapper --> upsert[upsert.py
create/update/suppress] - upsert --> okapi[(FOLIO OKAPI Inventory APIs)] + event --> folio_sync[Lambda:
axiell-folio-sync] + folio_sync --> ssm[SSM Secrets] + folio_sync --> refcache[ref_cache.py] + folio_sync --> read_changes[Read Iceberg] + read_changes --> mapper[mapping.py
Pydantic models] + mapper --> upsert[upsert.py] + upsert --> okapi[(FOLIO OKAPI)] ``` @@ -177,10 +170,10 @@ This replaces an earlier YAML-driven mapper (`mapping.yaml` + `YamlMapper`); see ```mermaid graph TD - A["Raw MARCXML
(from Iceberg)"] --> B["parse_marcxml (mapper.py)
MARC_SOURCE table → CanonicalRecord"] - B --> C["build_instance/holdings/item (mapping.py)
map + RefCache resolve → typed Pydantic models"] - C --> D["Pydantic validation
required fields, types, no unknown keys (extra=forbid)"] - D --> E["model_dump(exclude_none=True)
JSON-ready instance / holdings / item dicts"] + A["Raw MARCXML
(from Iceberg)"] --> B["parse_marcxml (mapper.py)
MARC SOURCE -> CanonicalRecord"] + B --> C["build_instance/holdings/item
map + RefCache resolve"] + C --> D["Pydantic validation
required fields, types"] + D --> E["model_dump
JSON-ready dicts"] ``` ### Pydantic Mapping (`mapping.py`) @@ -252,7 +245,7 @@ table in `mapping.py`; the hrids are derived from the GUID (MARC `001`). | 001 (GUID) | `source_id` → `holdings_hrid` | Holdings `hrid` | `AxC-holding-4d8f1208-9812-4bb5-84ef-da436b22d9e2` | | 001 (GUID) | `source_id` → `item_hrid` | Item `hrid` | `AxC-item-4d8f1208-9812-4bb5-84ef-da436b22d9e2` | | 245$a | `title` | Instance `title` | Daniel Morley, an English Philosopher... | -| 852$b | `location_code` | Holdings `permanentLocationId` (+ Item `permanentLocation`) | `215;B11;MR;84;3;7` | +| 852$b | `location_code` | Holdings `permanentLocationId` | resolved FOLIO location UUID | | 852$c | `call_number_prefix` | Holdings `callNumberPrefix` | `Arch`, `Ref` | | 852$h | `call_number` | Holdings `callNumber` | (optional) | | 852$j | `shelving_order` | Holdings `shelvingOrder` | (optional) | @@ -262,9 +255,9 @@ table in `mapping.py`; the hrids are derived from the GUID (MARC `001`). | 876$p | `copy_number` | Item `copyNumber` | `copy 1`, `copy 2` | | 876$t | `volume` | Item `volume` | `v.1`, `disc 1 of 2` | | 856$u | `electronic_access_uri` | Item `electronicAccess[].uri` | (optional) | -| 852$b | `location_code` | Item `notes[]`, type `Axiell location` | `Axiell location: 215;B11;MR;84;3;7` | +| 852$b | `location_code` | Item `notes[]`, type `Axiell location` | raw AxC location code: `215;B11;MR;84;3;7` | -The AxC current location is therefore written to FOLIO twice: as the resolved `permanentLocation` UUID (for shelving/discovery) and, in human-readable form, as an item **note of type `Axiell location`**. On update, that note is refreshed from the incoming `852$b`, so the FOLIO item always reflects the latest AxC current location. (The note type name resolves to `itemNoteTypeId` via RefCache before the write; note that this note is not currently surfaced on the OAI-PMH feed, see [Item notes not visible on the FOLIO OAI-PMH feed](#item-notes-not-visible-on-the-folio-oai-pmh-feed).) +The AxC location code is therefore written to FOLIO in two different fields: the Holdings `permanentLocationId` is set to the resolved FOLIO location UUID (for shelving/discovery and circulation), while the raw AxC location code is preserved as an item **note of type `Axiell location`** (for audit trail and historical reference). On update, the note is refreshed from the incoming `852$b`, so the FOLIO item always carries the latest AxC location code. (The note type name resolves to `itemNoteTypeId` via RefCache before the write; note that this note is not currently surfaced on the OAI-PMH feed, see [Item notes not visible on the FOLIO OAI-PMH feed](#item-notes-not-visible-on-the-folio-oai-pmh-feed).) --- @@ -353,39 +346,39 @@ Reload-per-run assumes the reference set is small enough to bulk-load into Lambd ```mermaid flowchart TD - subgraph upstream["Axiell Adapter Platform - upstream, not in this stack"] - adapter["Adapter - 15-min cadence
writes changesets to Iceberg
emits axiell.adapter.completed"] - bus(["EventBridge bus
catalogue-pipeline-adapter-event-bus"]) - adapter -->|"PutEvents - detail:
changeset_ids, job_id, transformer_type"| bus + subgraph upstream["Adapter Platform"] + adapter["Adapter
15-min cadence
writes Iceberg"] + bus(["EventBridge bus"]) + adapter -->|"changeset_ids
job_id"| bus end - subgraph sync["axiell-folio-sync - this Terraform stack"] - rule["EventBridge Rule
axiell-folio-sync-axiell-adapter-completed
pattern: source=axiell.adapter +
detail-type=axiell.adapter.completed +
detail.transformer_type=axiell"] - sfn["Step Function - STANDARD
axiell-folio-sync-sfn
InvokeSyncLambda then Retry 3x (backoff 2.0)
Catch States.ALL then SyncFailed"] - lambda["Lambda - ECR image
axiell-folio-sync
timeout 300s, mem 512MB
auth, scan, map, upsert, manifests"] - rule -->|"target: StartExecution
role: eventbridge-exec"| sfn - sfn -->|"lambda InvokeFunction
payload: detail.* plus dry_run"| lambda + subgraph sync["axiell-folio-sync"] + rule["EventBridge Rule
axiell-adapter-completed"] + sfn["Step Function
axiell-folio-sync-sfn
Retry 3x"] + lambda["Lambda ECR
axiell-folio-sync
timeout 300s"] + rule --> sfn + sfn --> lambda end bus --> rule - subgraph data["Data stores and observability"] - ssm[("SSM SecureString
okapi-credentials")] - iceberg[("S3 Tables / Iceberg
wellcomecollection-platform-axiell-adapter")] - ecr[("ECR
uk.ac.wellcome/axiell-folio-sync")] - s3man[("S3 manifests
axiell-folio-sync-manifests-acct-region
90-day lifecycle, versioned, SSE-S3")] - cw["CloudWatch
Logs: /aws/lambda + /aws/states (30d)
Metrics: AxiellFolioSync"] + subgraph data["Data and Observability"] + ssm[("SSM
Secrets")] + iceberg[("S3 Iceberg")] + ecr[("ECR
Image")] + s3man[("S3
Manifests")] + cw["CloudWatch
Logs+Metrics"] end - folio(["FOLIO OKAPI - external
api-wellcome.folio.ebsco.com"]) + folio(["FOLIO OKAPI"]) - ecr -.->|"pull image"| lambda - ssm -.->|"GetParameter + KMS decrypt"| lambda - iceberg -.->|"scan changeset rows"| lambda - lambda -->|"authn/login + upsert
instance, holdings, item"| folio - lambda -->|"NDJSON: ids / failures / manifest"| s3man - lambda -->|"metrics when not dry_run"| cw - sfn -.->|"execution history"| cw + ecr -.-> lambda + ssm -.-> lambda + iceberg -.-> lambda + lambda --> folio + lambda --> s3man + lambda --> cw + sfn -.-> cw ``` ### 1. EventBridge Trigger @@ -429,9 +422,9 @@ flowchart TD ```mermaid flowchart TD input[Input from EventBridge] --> invoke[Task: Invoke Lambda] - invoke --> output[Output: counts, manifest URIs, errors] - invoke -. Retry: MaxAttempts=3, BackoffRate=2.0, IntervalSeconds=2 .-> invoke - invoke -->|States.TaskFailed| failed[SyncFailed (terminal)] + invoke --> output[Output: counts, URIs, errors] + invoke -. Retry: MaxAttempts=3
BackoffRate=2.0 .-> invoke + invoke -->|TaskFailed| failed[SyncFailed] failed --> failedLog[Log to CloudWatch] output --> success[SyncComplete] ``` From c89265552bed481c72f206e2f787a2bb568bb11e Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Tue, 30 Jun 2026 12:43:15 +0100 Subject: [PATCH 08/15] Fix mermaid diagrams --- rfcs/090-axiell-folio-sync/README.md | 33 ++++++++++++++-------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index be9a3c8b..0d226d64 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -43,6 +43,7 @@ This RFC proposes an automated pipeline to synchronize data from **Axiell Collec - [Storage: S3 NDJSON vs. Alternatives](#storage-s3-ndjson-vs-alternatives) - [Invocation Pattern: EventBridge Trigger & Ordering](#invocation-pattern-eventbridge-trigger--ordering) - [Error Handling: Per-Record Isolation](#error-handling-per-record-isolation) + - [Caching Strategy: Reference Data Scaling Options](#caching-strategy-reference-data-scaling-options) - [SRS-backed Instances and the Update Path](#srs-backed-instances-and-the-update-path) - [FOLIO API Client](#folio-api-client) - [Cost Analysis](#cost-analysis) @@ -89,7 +90,7 @@ and also exposes its own OAI-PMH feed every 15 minutes. flowchart TD scheduler[EventBridge Scheduler] --> trigger[Trigger] trigger --> loader[Loader] - loader --> transformer[Transformer (ES)] + loader --> transformer[Transformer ES] loader --> reconciler[Reconciler] transformer --> es[Elasticsearch] loader --> iceberg[Iceberg Adapter Table] @@ -118,7 +119,7 @@ The Axiell adapter harvests over OAI-PMH using the `oai_marcxml` metadata prefix flowchart TD scheduler[EventBridge Scheduler] --> trigger[Trigger] trigger --> loader[Loader] - loader --> transformer[Transformer (ES)] + loader --> transformer[Transformer ES] transformer --> es[Elasticsearch] loader --> iceberg[Iceberg Table] loader --> upserter[FOLIO Upserter] @@ -327,19 +328,6 @@ We propose a **Step Functions + Lambda + S3 architecture** with event-driven (as **Expected outcomes:** replay is safe without data corruption (idempotent upserts via FOLIO HRIDs); there is a complete audit trail, with every decision (create/update/suppress/skip) logged in S3 and CloudWatch; operational overhead stays low at ~$3–5/month for typical volume; and the mental model remains clear, with no eventual-consistency puzzles and ordered execution. -#### Scaling the reference cache (if the dataset grows) - -Reload-per-run assumes the reference set is small enough to bulk-load into Lambda memory cheaply (see [Reference Data Cache](#reference-data-cache-ref_cachepy)). If we ever need **much more** reference data (many more types, or large per-record lookups), the choice of caching architecture matters. Options, cheapest first: - -| Option | What it is | Best when | Cost / overhead | Freshness control | -|--------|-----------|-----------|-----------------|-------------------| -| **Warm singleton + reload-on-miss** | Module-global cache reused across warm invocations; fetch-and-memoize a code on a cache miss | Set still fits in memory; modest growth | None (in-process) | Self-healing on miss; lost on cold start | -| **S3 snapshot + scheduled refresher** | A cron (EventBridge) Lambda rebuilds a serialized snapshot (JSON/Parquet) from FOLIO every N hours; the sync Lambda reads it at startup (one GET) instead of N FOLIO calls | Whole set scanned each run; want to decouple sync from FOLIO availability | ~pennies/mo + one extra scheduled Lambda | Snapshot age = refresh cadence; pair with reload-on-miss | -| **DynamoDB lookup table** | Refresher populates `(type, code) → uuid`; sync Lambda `BatchGetItem`s only the codes a changeset needs | Set too big for memory, or only a subset is needed per run | On-demand $/read; native TTL | TTL / refresher cadence; pair with reload-on-miss | -| **ElastiCache (Redis)** | Shared in-memory cache across all invocations | Very large + hot lookups at high concurrency | Always-on (~$12+/mo), **requires VPC** (ENI cold-start + NAT to reach FOLIO/AWS APIs) | TTL | - -**Recommended progression:** stay on reload-per-run → **S3 snapshot + scheduled refresher** (cheap, no VPC, scales via columnar formats, decouples from FOLIO reference endpoints) → **DynamoDB** only once the set outgrows Lambda memory or you genuinely need selective point lookups → **ElastiCache** only for high-concurrency hot-lookup workloads that justify always-on infra and the VPC tax. In every tier, keep a **reload-on-miss fallback to FOLIO** so a newly-added code resolves on first sight rather than failing the record. - --- ### System Diagram @@ -423,7 +411,7 @@ flowchart TD flowchart TD input[Input from EventBridge] --> invoke[Task: Invoke Lambda] invoke --> output[Output: counts, URIs, errors] - invoke -. Retry: MaxAttempts=3
BackoffRate=2.0 .-> invoke + invoke -. Retry: Max 3
Backoff 2.0 .-> invoke invoke -->|TaskFailed| failed[SyncFailed] failed --> failedLog[Log to CloudWatch] output --> success[SyncComplete] @@ -701,6 +689,19 @@ We chose Step Functions + Lambda (B): it buys an explicit retry policy and a ful A direct EventBridge→Lambda trigger (A) was rejected because it gives no failure signal, no automatic retry, and no execution history, so debugging would mean parsing Lambda logs. An async SQS queue (C) is cheaper per invocation but doesn't guarantee processing order within a batch, needs workers to coordinate which changeset they own, and carries monitoring overhead (error tracking, retries, dead-letter management) that isn't justified at current volume. +### Caching Strategy: Reference Data Scaling Options + +Reload-per-run assumes the reference set is small enough to bulk-load into Lambda memory cheaply (see [Reference Data Cache](#reference-data-cache-ref_cachepy)). If we ever need **much more** reference data (many more types, or large per-record lookups), the choice of caching architecture matters. Options, cheapest first: + +| Option | What it is | Best when | Cost / overhead | Freshness control | +|--------|-----------|-----------|-----------------|-------------------| +| **Warm singleton + reload-on-miss** | Module-global cache reused across warm invocations; fetch-and-memoize a code on a cache miss | Set still fits in memory; modest growth | None (in-process) | Self-healing on miss; lost on cold start | +| **S3 snapshot + scheduled refresher** | A cron (EventBridge) Lambda rebuilds a serialized snapshot (JSON/Parquet) from FOLIO every N hours; the sync Lambda reads it at startup (one GET) instead of N FOLIO calls | Whole set scanned each run; want to decouple sync from FOLIO availability | ~pennies/mo + one extra scheduled Lambda | Snapshot age = refresh cadence; pair with reload-on-miss | +| **DynamoDB lookup table** | Refresher populates `(type, code) → uuid`; sync Lambda `BatchGetItem`s only the codes a changeset needs | Set too big for memory, or only a subset is needed per run | On-demand $/read; native TTL | TTL / refresher cadence; pair with reload-on-miss | +| **ElastiCache (Redis)** | Shared in-memory cache across all invocations | Very large + hot lookups at high concurrency | Always-on (~$12+/mo), **requires VPC** (ENI cold-start + NAT to reach FOLIO/AWS APIs) | TTL | + +**Recommended progression:** stay on reload-per-run → **S3 snapshot + scheduled refresher** (cheap, no VPC, scales via columnar formats, decouples from FOLIO reference endpoints) → **DynamoDB** only once the set outgrows Lambda memory or you genuinely need selective point lookups → **ElastiCache** only for high-concurrency hot-lookup workloads that justify always-on infra and the VPC tax. In every tier, keep a **reload-on-miss fallback to FOLIO** so a newly-added code resolves on first sight rather than failing the record. + --- ### Storage: S3 NDJSON vs. Alternatives From e37231ce360df49625b8da1f396dc227c61e64d7 Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Tue, 30 Jun 2026 12:51:59 +0100 Subject: [PATCH 09/15] Fix the diagram --- rfcs/090-axiell-folio-sync/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index 0d226d64..49b5d8f0 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -411,7 +411,8 @@ flowchart TD flowchart TD input[Input from EventBridge] --> invoke[Task: Invoke Lambda] invoke --> output[Output: counts, URIs, errors] - invoke -. Retry: Max 3
Backoff 2.0 .-> invoke + invoke --> retryPolicy[Retry policy
Max attempts 3
Backoff 2.0] + retryPolicy --> invoke invoke -->|TaskFailed| failed[SyncFailed] failed --> failedLog[Log to CloudWatch] output --> success[SyncComplete] From d8276d15315c12494d96f1d10ec047dc383063f1 Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Tue, 30 Jun 2026 12:58:15 +0100 Subject: [PATCH 10/15] More changes to diagrams --- rfcs/090-axiell-folio-sync/README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index 49b5d8f0..f0c2d904 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -119,10 +119,12 @@ The Axiell adapter harvests over OAI-PMH using the `oai_marcxml` metadata prefix flowchart TD scheduler[EventBridge Scheduler] --> trigger[Trigger] trigger --> loader[Loader] - loader --> transformer[Transformer ES] - transformer --> es[Elasticsearch] loader --> iceberg[Iceberg Table] - loader --> upserter[FOLIO Upserter] + loader --> event[changeset_ids event] + event --> transformer[Transformer ES] + transformer --> es[Elasticsearch] + event --> upserter[FOLIO Upserter] + iceberg --> transformer iceberg --> upserter upserter --> folio[FOLIO Inventory] @@ -411,9 +413,9 @@ flowchart TD flowchart TD input[Input from EventBridge] --> invoke[Task: Invoke Lambda] invoke --> output[Output: counts, URIs, errors] - invoke --> retryPolicy[Retry policy
Max attempts 3
Backoff 2.0] - retryPolicy --> invoke - invoke -->|TaskFailed| failed[SyncFailed] + invoke -->|TaskFailed| retryPolicy[Retry policy
Max attempts 3
Backoff 2.0] + retryPolicy --> invoke + retryPolicy -->|Retries exhausted| failed[SyncFailed] failed --> failedLog[Log to CloudWatch] output --> success[SyncComplete] ``` From 183e8fc6424ec24ed80f71cce50dfdc020f7bad7 Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Tue, 30 Jun 2026 13:10:51 +0100 Subject: [PATCH 11/15] Fix a few more things --- rfcs/090-axiell-folio-sync/README.md | 8 +++---- .../folio-axc-fields-mapping.md | 4 ++-- rfcs/090-axiell-folio-sync/mapping.py | 21 +++++++------------ 3 files changed, 13 insertions(+), 20 deletions(-) diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index f0c2d904..8bbf1743 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -40,10 +40,10 @@ This RFC proposes an automated pipeline to synchronize data from **Axiell Collec - [Design Rationale: Why These Choices?](#design-rationale-why-these-choices) - [Mapping: Pydantic Models vs. YAML Mapper](#mapping-pydantic-models-vs-yaml-mapper) - [Orchestration: Step Functions vs. Alternatives](#orchestration-step-functions-vs-alternatives) + - [Caching Strategy: Reference Data Scaling Options](#caching-strategy-reference-data-scaling-options) - [Storage: S3 NDJSON vs. Alternatives](#storage-s3-ndjson-vs-alternatives) - [Invocation Pattern: EventBridge Trigger & Ordering](#invocation-pattern-eventbridge-trigger--ordering) - [Error Handling: Per-Record Isolation](#error-handling-per-record-isolation) - - [Caching Strategy: Reference Data Scaling Options](#caching-strategy-reference-data-scaling-options) - [SRS-backed Instances and the Update Path](#srs-backed-instances-and-the-update-path) - [FOLIO API Client](#folio-api-client) - [Cost Analysis](#cost-analysis) @@ -300,7 +300,7 @@ Only AxC MARC records that have the **harvest flag set in MARC field `980 $a`** ### Every Record in a Changeset Is Either -Each record in a loader changeset is treated as **new** (the first time this `id` appears in Iceberg), in which case it is created in FOLIO, or **updated** (an existing `id` with a newer `last_modified`), in which case it is updated in FOLIO. Deletions are handled on the reconciler path (see [Delete detection](#delete-detection-the-reconciler-not-oai-tombstones) below), not by directly suppressing on the loader's `deleted=true` flag. The exact delete action in FOLIO (suppress vs remove, and cascade scope) remains an open policy question (see [Delete semantics](#delete-semantics-what-should-deletedtrue-do-in-folio)). +Each record in a loader changeset is treated as **new** (the first time this `id` appears in Iceberg), in which case it is created in FOLIO, or **updated** (an existing `id` with a newer `last_modified`), in which case it is updated in FOLIO. Deletions are handled on the reconciler path (see [Delete detection](#delete-detection-the-reconciler-not-oai-tombstones) below), not by directly suppressing on the loader's `deleted=true` flag. The exact delete action in FOLIO (suppress vs remove, and cascade scope) remains an open policy question (see [Delete semantics](#delete-semantics-what-should-reconciler-detected-deletes-do-in-folio)). ### Delete detection: the reconciler, not OAI tombstones @@ -308,7 +308,7 @@ Axiell's OAI tombstones (`deleted=true`) are unreliable, which is the very reaso This has two consequences for the FOLIO upserter. First, the upserter consumes the **loader's changeset**, so it does not see reconciler-detected deletes, which are produced by a separate, later transformer step. Second, a reconciler delete is keyed by the **old (superseded) GUID**, not by the changeset row being processed. -Suggested approach: have the **reconciler** step fan out a FOLIO suppression path that mirrors the loader's fan-out to the upsert path, reusing the existing reconciler rather than building a parallel `collectId → guid` mapping store. On a reconciler delete, suppress the FOLIO records for the old GUID (`AxC-instance-{old-guid}`) and cascade to its holdings and item, in line with the [Delete semantics](#delete-semantics-what-should-deletedtrue-do-in-folio) question raised earlier. The loader-changeset `deleted=true` can remain a best-effort secondary signal, but the reconciler fan-out is the authoritative delete path. +Suggested approach: have the **reconciler** step fan out a FOLIO suppression path that mirrors the loader's fan-out to the upsert path, reusing the existing reconciler rather than building a parallel `collectId → guid` mapping store. On a reconciler delete, suppress the FOLIO records for the old GUID (`AxC-instance-{old-guid}`) and cascade to its holdings and item, in line with the [Delete semantics](#delete-semantics-what-should-reconciler-detected-deletes-do-in-folio) question raised earlier. The loader-changeset `deleted=true` can remain a best-effort secondary signal, but the reconciler fan-out is the authoritative delete path. --- @@ -809,7 +809,7 @@ The stale-write guard (see [Invocation Pattern](#invocation-pattern-eventbridge- ### Reference-data caching: when and how to cache across Lambda runs -Today `RefCache` reloads all six reference sets on every invocation, which is the right trade-off at current volume (see [Reference Data Cache](#reference-data-cache-ref_cachepy)). The open question is what happens if the reference set grows substantially. We would need to decide the size or load-latency threshold at which caching across runs is worth the added complexity. Staleness is unlikely to be the blocker: the reference data changes rarely, so a long TTL would seldom be stale, and a reload-on-miss fallback would keep a newly-added code from failing as a `MappingError`. So the question is mostly about whether the saved reload justifies the extra infrastructure. We would also need to pick a tier when we get there: the progression runs from a warm-singleton with reload-on-miss, to an S3 snapshot rebuilt by a scheduled refresher, to DynamoDB, and only then to ElastiCache (see [Scaling the reference cache](#scaling-the-reference-cache-if-the-dataset-grows) for the trade-offs). +Today `RefCache` reloads all six reference sets on every invocation, which is the right trade-off at current volume (see [Reference Data Cache](#reference-data-cache-ref_cachepy)). The open question is what happens if the reference set grows substantially. We would need to decide the size or load-latency threshold at which caching across runs is worth the added complexity. Staleness is unlikely to be the blocker: the reference data changes rarely, so a long TTL would seldom be stale, and a reload-on-miss fallback would keep a newly-added code from failing as a `MappingError`. So the question is mostly about whether the saved reload justifies the extra infrastructure. We would also need to pick a tier when we get there: the progression runs from a warm-singleton with reload-on-miss, to an S3 snapshot rebuilt by a scheduled refresher, to DynamoDB, and only then to ElastiCache (see [Caching Strategy: Reference Data Scaling Options](#caching-strategy-reference-data-scaling-options) for the trade-offs). ### Manifest query mechanism: S3 Select is not an established org pattern diff --git a/rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md b/rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md index d27d59a2..b68e3dac 100644 --- a/rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md +++ b/rfcs/090-axiell-folio-sync/folio-axc-fields-mapping.md @@ -48,7 +48,7 @@ models + builders, the source of truth) · `mapper.py` (MARC extraction via the ## Item -`Item` model: `hrid`, `holdingsRecordId`, `status`, `materialType` `{id}`, `permanentLoanType` `{id}`, `permanentLocation` `{id}`, `barcode`, `copyNumber`, `volume`, `electronicAccess`, `notes`. +`Item` model: `hrid`, `holdingsRecordId`, `status`, `materialType` `{id}`, `permanentLoanType` `{id}`, `permanentLocationId`, `barcode`, `copyNumber`, `volume`, `electronicAccess`, `notes`. | FOLIO Field | Type | Status | AxC source | MARC Tag | Notes | |-------------|------|--------|------------|----------|-------| @@ -57,7 +57,7 @@ models + builders, the source of truth) · `mapper.py` (MARC extraction via the | `status.name` | string | Required *(in model)* | — | — | Defaults to `"Available"` (`status` is a nested object, `default_factory=Status`). | | `materialType.id` | UUID | Required *(in model)* | `Object_category` | `949$c` | Inventory `{ "id": "" }` reference. RefCache + `MATERIAL_TYPE` table; default `"Books"`. Example: `"archives"` → `"unspecified"`. | | `permanentLoanType.id` | UUID | Required *(in model)* | `Free_texts` (`OrderingCodes`) | `949$l` | `{ "id": "" }`. RefCache; default `"Can Circulate"`. | -| `permanentLocation.id` | UUID | Required *(in model)* | `location` | `852$b` | `{ "id": "" }`; same location as holdings. RefCache; default `"History of Medicine"`. | +| `permanentLocationId` | UUID | Required *(in model)* | `location` | `852$b` | Bare UUID (the writable field; `permanentLocation` is read-only in FOLIO); same location as holdings. RefCache; default `"History of Medicine"`. | | `barcode` | string | Optional *(in model)* | — | `949$a` | May be absent for archival items. | | `copyNumber` | string | Optional *(in model)* | — | `876$p` | Example: `"copy 1"`. | | `volume` | string | Optional *(in model)* | — | `876$t` | Example: `"v.1"`. | diff --git a/rfcs/090-axiell-folio-sync/mapping.py b/rfcs/090-axiell-folio-sync/mapping.py index e3865b80..b88d1d2f 100644 --- a/rfcs/090-axiell-folio-sync/mapping.py +++ b/rfcs/090-axiell-folio-sync/mapping.py @@ -25,7 +25,6 @@ from __future__ import annotations -import re from typing import Callable, Optional from pydantic import BaseModel, ConfigDict, Field @@ -57,11 +56,6 @@ } -def _safe_segment(value: str) -> str: - """Lowercase slug from an arbitrary string — used in composite hrids.""" - return re.sub(r"[^a-z0-9]+", "-", value.strip().lower()).strip("-") or "unknown" - - def parse_marcxml(xml_content: str, *, deleted: bool = False) -> CanonicalRecord: """Parse a single MARCXML string into a :class:`CanonicalRecord` via MARC_SOURCE. @@ -74,13 +68,12 @@ def parse_marcxml(xml_content: str, *, deleted: bool = False) -> CanonicalRecord if not source_id: raise MappingError("Missing MARC 001 — cannot identify record") - instance_hrid = f"axiell:{source_id}" - holdings_hrid = ( - f"{instance_hrid}-holding-{_safe_segment(values['location_code'] or 'unknown')}" - ) + # Single GUID-based hrid scheme (see _instance_hrid / _holdings_hrid): every + # hrid is derived purely from the Axiell GUID, never from location, so the + # idempotency key stays stable even if a record's location changes. return CanonicalRecord( - instance_hrid=instance_hrid, - holdings_hrid=holdings_hrid, + instance_hrid=_instance_hrid(source_id), + holdings_hrid=_holdings_hrid(source_id), deleted=deleted, **values, ) @@ -167,7 +160,7 @@ class Item(BaseModel): status: Status = Field(default_factory=Status) materialType: IdRef permanentLoanType: IdRef - permanentLocation: IdRef + permanentLocationId: str # bare UUID; `permanentLocation` is read-only in FOLIO barcode: Optional[str] = None copyNumber: Optional[str] = None volume: Optional[str] = None @@ -249,7 +242,7 @@ def build_item(rec: CanonicalRecord, ref: RefCache) -> Item: ref.resolve_loan_type, rec.loan_type_code, label="loan type", default=DEFAULT_LOAN_TYPE, )), - permanentLocation=IdRef(id=location_id), + permanentLocationId=location_id, barcode=rec.barcode, copyNumber=rec.copy_number, volume=rec.volume, From c03fef7b2cfd8368cd108bccf38fc6df60230ba8 Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Tue, 30 Jun 2026 13:33:58 +0100 Subject: [PATCH 12/15] Fix build --- rfcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/README.md b/rfcs/README.md index e97c811d..f80f36b8 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -74,7 +74,7 @@ _This is generated from the RFCs in this directory using `.scripts/create_table_ | RFC ID | Summary | Next Line | Last Modified | |--------|---------|-----------|---------------| -| [090-axiell-folio-sync](090-axiell-folio-sync/README.md) | RFC 090: CMS to LMS Sync | This is proposal to synchronize library location and item holdings from Axiell Collections(Content Management System) into FOLIO (Library Management system) with strict requirements for idempotency, auditability, and graceful error isolation. | 23 Jun 2026 | +| [090-axiell-folio-sync](090-axiell-folio-sync/README.md) | RFC 090: CMS to LMS Sync | This RFC proposes an automated pipeline to synchronize data from **Axiell Collections (AxC)**, the new Content Management System (CMS), into **FOLIO**, the new Library Management System (LMS). The records from Axiell Collections need to exist in FOLIO so they can be requested and circulated in the LMS. It is designed for **idempotency** (safe to replay without duplication), **auditability** (every create / update / suppress is recorded), and **graceful error isolation** (one bad record never halts the batch). | 30 Jun 2026 | | [088-folio-identity-requesting-migration](088-folio-identity-requesting-migration/README.md) | RFC 088: Migrating identity, requesting and items APIs from Sierra to FOLIO | This RFC describes how we move the identity, requesting and item-availability APIs that power `wellcomecollection.org` from our current Library Management System (LMS), **Sierra**, to its replacement, **FOLIO**. It sets out the proposed architecture (a parallel, FOLIO-backed **v2** identity API fronted by Auth0), the embedded API contract, the migration plan (a per-request website toggle plus lazy patron migration, culminating in a single coordinated cutover), and the questions still open before cutover. | 22 Jun 2026 | | [089-identifiers-api](089-identifiers-api/README.md) | RFC 089: Identifiers API | This RFC proposes a small, read-only **Identifiers API** that resolves a **canonical** catalogue identifier to its **source** identifier(s) and back, served from the catalogue ID Registry (the same store the ID Minter writes to, per [RFC 083](../083-stable_identifiers/README.md)). It provides that translation in one place, between the canonical ids the public surface uses and the source ids (Sierra numbers, FOLIO UUIDs, CALM/Axiell refs) that the underlying systems require across the Sierra/CALM → FOLIO/Axiell migration. It sets out the contract, the AWS architecture, the authentication and cost model, the caching strategy, and what a working prototype has already established. | 18 Jun 2026 | | [087-kiosk-mode](087-kiosk-mode/README.md) | RFC 087: wellcomecollection.org in kiosk mode | This RFC serves to outline how we propose to offer in-venue experiences using our current website, while optimising it for a different experience than usual. | 13 May 2026 | From 6c14b3972cf9f5be55e3b6a1db5d2a9ced73e6ab Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Wed, 1 Jul 2026 09:38:13 +0100 Subject: [PATCH 13/15] Move Design rationale in its own md and general trimming --- rfcs/090-axiell-folio-sync/README.md | 295 +++--------------- .../090-axiell-folio-sync/design-rationale.md | 134 ++++++++ 2 files changed, 183 insertions(+), 246 deletions(-) create mode 100644 rfcs/090-axiell-folio-sync/design-rationale.md diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index 8bbf1743..b553788c 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -14,40 +14,27 @@ This RFC proposes an automated pipeline to synchronize data from **Axiell Collec - [Current Data Feeds for AxC and Folio Data](#current-data-feeds-for-axc-and-folio-data) - [Proposed Change: FOLIO Item Upsert on Every Adapter Run](#proposed-change-folio-item-upsert-on-every-adapter-run) - [Fan-out Mechanism](#fan-out-mechanism) - - [Target Upsert/Mapping Architecture](#target-upsertmapping-architecture) - [Transformation Design](#transformation-design) - [Approach](#approach) - [Transformation Pipeline](#transformation-pipeline) - [Pydantic Mapping](#pydantic-mapping-mappingpy) - [Reference Data Cache](#reference-data-cache-ref_cachepy) - - [RefCache Resolution](#refcache-resolution) - [Sample Field Mapping](#sample-field-mapping) - [FOLIO Field Mapping Reference](#folio-field-mapping-reference) - [Change Detection Mechanism](#change-detection-mechanism) - [How It Works](#how-it-works) - [Record selection: only harvest-flagged records](#record-selection-only-harvest-flagged-records) - [Change Detection Signals](#change-detection-signals) - - [Every Record in a Changeset Is Either](#every-record-in-a-changeset-is-either) - - [Delete detection: the reconciler, not OAI tombstones](#delete-detection-the-reconciler-not-oai-tombstones) + - [How records are applied: create, update, delete](#how-records-are-applied-create-update-delete) + - [Upsert Key Strategy (Idempotency)](#upsert-key-strategy-idempotency) - [Proposed AWS Architecture](#proposed-aws-architecture) - [Key Design Considerations](#key-design-considerations) - [System Diagram](#system-diagram) - [1. EventBridge Trigger](#1-eventbridge-trigger) - [2. Step Function State Machine](#2-step-function-state-machine) - [3. Lambda Function: axiell-folio-sync](#3-lambda-function-axiell-folio-sync) - - [Upsert Key Strategy (Idempotency)](#upsert-key-strategy-idempotency) - [4. S3 Manifest Storage](#4-s3-manifest-storage) -- [Design Rationale: Why These Choices?](#design-rationale-why-these-choices) - - [Mapping: Pydantic Models vs. YAML Mapper](#mapping-pydantic-models-vs-yaml-mapper) - - [Orchestration: Step Functions vs. Alternatives](#orchestration-step-functions-vs-alternatives) - - [Caching Strategy: Reference Data Scaling Options](#caching-strategy-reference-data-scaling-options) - - [Storage: S3 NDJSON vs. Alternatives](#storage-s3-ndjson-vs-alternatives) - - [Invocation Pattern: EventBridge Trigger & Ordering](#invocation-pattern-eventbridge-trigger--ordering) - - [Error Handling: Per-Record Isolation](#error-handling-per-record-isolation) -- [SRS-backed Instances and the Update Path](#srs-backed-instances-and-the-update-path) -- [FOLIO API Client](#folio-api-client) -- [Cost Analysis](#cost-analysis) - - [Current Volume: ~1,000 records/day](#current-volume-1000-recordsday) +- [Design Decisions](#design-decisions) - [Open Questions](#open-questions) - [Field Mapping from Axc to Folio Instance, Holdings and Items needs to be defined](#field-mapping-from-axc-to-folio-instance-holdings-and-items-needs-to-be-defined) - [Delete semantics: what should reconciler-detected deletes do in FOLIO?](#delete-semantics-what-should-reconciler-detected-deletes-do-in-folio) @@ -56,8 +43,6 @@ This RFC proposes an automated pipeline to synchronize data from **Axiell Collec - [Manifest query mechanism: S3 Select is not an established org pattern](#manifest-query-mechanism-s3-select-is-not-an-established-org-pattern) - [Item creation should likely be gated on record `Level`](#item-creation-should-likely-be-gated-on-record-level) - [Item notes not visible on the FOLIO OAI-PMH feed](#item-notes-not-visible-on-the-folio-oai-pmh-feed) -- [Next Steps](#next-steps) -- [References](#references) --- @@ -107,7 +92,7 @@ flowchart TD #### Key characteristics -The Axiell adapter harvests over OAI-PMH using the `oai_marcxml` metadata prefix and the `collect` OAI set, authenticating with a custom `Token` header rather than the standard `Authorization` header. Record identity comes from the `axiell-guid` extracted from MARC `001`, while visibility is controlled by the `InvisibleSourceWork` flag (MimsyWorksAreNotVisible). Harvesting runs in 15-minute windows with a 7-day lookback and a 360-minute maximum lag. All records land in a single Iceberg table per adapter, whose schema is `namespace`, `id`, `content`, `changeset`, `last_modified`, and `deleted`. +The Axiell adapter harvests over OAI-PMH using the `oai_marcxml` metadata prefix and the `collect` OAI set, authenticating with a custom `Token` header rather than the standard `Authorization` header. Record identity comes from the `axiell-guid` extracted from MARC `001`, while visibility is controlled by the `InvisibleSourceWork` flag (MimsyWorksAreNotVisible). Harvesting runs in 15-minute windows with a 7-day lookback and a 360-minute maximum lag. All records land in a single Iceberg table per adapter (row schema described under [Change Detection](#how-it-works)). --- @@ -138,28 +123,9 @@ The FOLIO upserter follows the same pattern but targets a different sink, the ** So after the loader emits `changeset_ids`, the event fans out to **two** independent transformers: the existing ES transformer (`AxiellTransformer`, unchanged) writing to Elasticsearch for discovery, and the new FOLIO upserter writing to FOLIO Inventory for circulation and requesting in the LMS. Both consume the same `changeset_ids` independently, and either path can fail and retry without affecting the other, so adding the LMS sink does not put the existing discovery feed at risk. ---- -### Target Upsert/Mapping Architecture - -```mermaid -flowchart LR - scheduler[EventBridge
15-min cadence] --> trigger[Trigger] - trigger --> loader[Loader
OAI-PMH] - loader --> iceberg[(Iceberg)] - loader --> event[changeset_ids] - - event --> es_transformer[Transformer
AxiellTransformer] - es_transformer --> es[(Elasticsearch)] - - event --> folio_sync[Lambda:
axiell-folio-sync] - folio_sync --> ssm[SSM Secrets] - folio_sync --> refcache[ref_cache.py] - folio_sync --> read_changes[Read Iceberg] - read_changes --> mapper[mapping.py
Pydantic models] - mapper --> upsert[upsert.py] - upsert --> okapi[(FOLIO OKAPI)] -``` +The FOLIO upserter's internal module flow (`ref_cache` → read Iceberg → `mapping.py` → `upsert.py` → OKAPI) is detailed under [Transformation Pipeline](#transformation-pipeline) and the [Lambda execution steps](#3-lambda-function-axiell-folio-sync); its AWS deployment wiring is in the [System Diagram](#system-diagram). +--- ## Transformation Design @@ -167,7 +133,7 @@ flowchart LR The upserter converts raw MARCXML from the AxC adapter into FOLIO payloads using **typed Python with Pydantic models** (`mapping.py`). The three payloads (Instance, Holdings, and Item) are Pydantic models with `extra="forbid"`, so the field-mapping rules are ordinary, reviewable Python and the payloads are typed contracts: a malformed payload (missing/ill-typed required field, typo'd key) **fails at build time, before any OKAPI call**, instead of surfacing as a FOLIO 422 mid-batch. -This replaces an earlier YAML-driven mapper (`mapping.yaml` + `YamlMapper`); see [Mapping: Pydantic Models vs. YAML Mapper](#mapping-pydantic-models-vs-yaml-mapper) for the trade-off. +This replaces an earlier YAML-driven mapper (`mapping.yaml` + `YamlMapper`); see [Mapping: Pydantic Models vs. YAML Mapper](design-rationale.md#mapping-pydantic-models-vs-yaml-mapper) for the trade-off. ### Transformation Pipeline @@ -183,17 +149,16 @@ graph TD `mapping.py` is the single home for everything that decides *what an Axiell record becomes in FOLIO*: the MARC source table, normalization tables, defaults, the hrid scheme, the typed payload contracts, and the field-by-field builders. -The flow runs in three steps. `parse_marcxml()` (in `mapper.py`) first extracts MARC fields via the `MARC_SOURCE` table (e.g. `title: 245$a`, `location_code: 852$b`, `barcode: 949$a`) into a typed `CanonicalRecord`, where MARC `001` is the Axiell GUID and the basis for every hrid. `build_instance/holdings/item()` then map that record into the `Instance`, `Holdings`, and `Item` Pydantic models, resolving reference data (location, material type, loan type, holdings source, instance type) through **RefCache** and applying defaults. Finally, Pydantic validates on construction (`extra="forbid"` rejects unknown keys) and `build_payloads()` returns `{"instance", "holdings", "item", "meta"}` as JSON-ready dicts, with `meta` carrying `source_id`, the three hrids, and `mapping_version`. +The flow runs in three steps. `parse_marcxml()` (in `mapper.py`) first extracts MARC fields via the `MARC_SOURCE` table (e.g. `title: 245$a`, `location_code: 852$b`, `barcode: 949$a`) into a typed `CanonicalRecord`, where MARC `001` is the Axiell GUID and the basis for every hrid. `build_instance/holdings/item()` then map that record into the `Instance`, `Holdings`, and `Item` Pydantic models, resolving reference data (location, material type, loan type, holdings source, instance type) through **RefCache** and applying defaults. Finally, Pydantic validates on construction (`extra="forbid"` rejects unknown keys) and `build_payloads()` returns `{"instance", "holdings", "item", "meta"}` as JSON-ready dicts, with `meta` carrying `source_id`, the three hrids, and `mapping_version` (used for audit, rollback, and tracing a record back to the rules that produced it). -This buys several things. Invalid payloads raise `MappingError`/`ValidationError` at build time, before any FOLIO write, so there are no round-trip 422s; the payload shape is an enforced typed contract (`extra="forbid"`), so a typo'd key is a build error rather than a silently dropped field; the rules, defaults, and contracts live in one reviewable Python module with full language expressiveness for normalization and conditionals; and every payload is stamped with `mapping_version` for traceability. - -**hrid scheme** (the idempotency key for every upsert): `AxC-instance-{guid}`, `AxC-holding-{guid}`, `AxC-item-{guid}`. `Instance.source = "FOLIO"`. The sync creates no linked SRS MARC record, so the instance is FOLIO-native and its bibliographic fields stay editable via mod-inventory. +**hrid scheme** (the idempotency key for every upsert): `AxC-instance-{guid}`, `AxC-holding-{guid}`, `AxC-item-{guid}`. `Instance.source = "FOLIO"` (FOLIO-native, no linked SRS record; see [Design Decisions](#design-decisions)). --- ### Reference Data Cache (`ref_cache.py`) -The reference cache maintains in-memory lookups for static FOLIO configuration data that rarely changes. `RefCache.load()` fetches and indexes **six** reference sets: +FOLIO APIs require UUIDs/IDs, but Axiell supplies human-readable names, so `RefCache` translates names → FOLIO IDs and caches them to avoid repeated API calls within a run. `RefCache.load()` runs once per invocation and fetches and indexes **six** reference sets: + - **Locations**: indexed by both code *and* name → UUID - **Material types**: by name → UUID (e.g., "Books", "video recording") - **Loan types**: by name → UUID (e.g., "Can Circulate", "Reference") @@ -201,42 +166,12 @@ The reference cache maintains in-memory lookups for static FOLIO configuration d - **Item note types**: by name → UUID (resolves the "Axiell location" note's `itemNoteTypeId`) - **Instance types**: the resolved default type id (e.g., "text"), applied to every instance -(It also retains the full record list per set, but those are used only by the `summary()` / introspection helpers, not by the mapping.) - -FOLIO APIs require UUIDs/IDs rather than human-readable names, but Axiell supplies names, so `ref_cache` translates names to FOLIO IDs; caching them avoids repeated API calls during a sync run, which helps both performance and resilience. - -**Workflow:** on Lambda startup, `ref_cache.load()` fetches from the FOLIO reference endpoints; the mapper then resolves names against the cache (e.g. `location_uuid = ref_cache.location["Wellcome Science"]`); and if a name isn't found, the error is logged and the record is marked for manual review. - -**Cache persistence & freshness:** `RefCache.load()` runs once at the start of each -invocation and is **not** carried across invocations. Reloading every run -mostly re-fetches identical data; the point is simply that doing so is trivially -cheap (about six reference-endpoint GETs, a few hundred KB, ~1–3 s on a job that runs -every ~15 minutes, ~80 syncs/day), negligible next to the per-record OKAPI upserts, -and needs no extra infrastructure: no DynamoDB/ElastiCache/S3 layer, no TTL, no -cache-invalidation logic. Reading fresh each run also means that on the rare occasion -a new code does appear it is picked up on the very next sync instead of failing as a -`MappingError`, but that is a bonus rather than the reason. - -The trigger to revisit is reference-load latency genuinely dominating runtime (much -higher frequency or far larger reference sets); even then the order is -**warm-singleton + reload-on-miss** first (the pattern the FOLIO token already uses), -and an external cross-run cache only beyond that. - ---- - -### RefCache Resolution - -All reference lookups (location, material type, loan type, holdings source, item note type, and the default instance type) are resolved at sync time via **RefCache**, which caches that FOLIO reference data. - -**Resolution process:** on Lambda startup RefCache initializes by querying the FOLIO reference APIs once per invocation; for each record the mapper looks codes up against the cached values (an in-memory, O(1) lookup); if a code isn't found the upsert fails with `MappingError` and the record is skipped; and the resolved UUIDs are embedded directly in the payload. - -**Current normalisations:** material types map from AxC `Object_category` to a FOLIO material-type name (e.g. `"archives"` → `"unspecified"`, audio → `"sound recording"`); locations map from an AxC hierarchical location code (e.g. `"215;B11;MR;84;3;7"`) to a FOLIO location UUID; loan types map from AxC `OrderingCodes` (e.g. `"Archives - Requestable"`) to a FOLIO loan type UUID; and the instance type is typically `"text"` or a domain-specific type. +**Resolution:** for each record the mapper looks names up against the cache (an in-memory, O(1) lookup, e.g. `ref_cache.location["Wellcome Science"]`) and embeds the resolved UUIDs directly in the payload; if a name isn't found the record fails with `MappingError` and is skipped for manual review. **Current normalisations:** material types map from AxC `Object_category` to a FOLIO material-type name (e.g. `"archives"` → `"unspecified"`, audio → `"sound recording"`); locations from an AxC hierarchical code (e.g. `"215;B11;MR;84;3;7"`) to a location UUID; loan types from AxC `OrderingCodes` (e.g. `"Archives - Requestable"`) to a loan-type UUID; instance type is typically `"text"`. +**Caching decision:** the cache is reloaded every run and **not** carried across invocations — about six reference-endpoint GETs (~1–3 s) is negligible next to the per-record upserts and needs no extra infrastructure (no DynamoDB/ElastiCache/TTL). For when this stops being the right trade-off and the scale-up progression, see [Caching Strategy](design-rationale.md#caching-strategy-reference-data-scaling-options). --- -Stamping each payload with `mapping_version` supports audit (comparing records created with different mapper versions), rollback (identifying the records affected by a mapping change by version), and metadata tracking (a historical record of the mapping logic used for each record). - ### Sample Field Mapping `CanonicalRecord` field = the attribute parsed from MARC via the `MARC_SOURCE` @@ -298,17 +233,27 @@ Only AxC MARC records that have the **harvest flag set in MARC field `980 $a`** | Payload hash mismatch | XSL output comparison (optional) | FOLIO-relevant fields actually changed | | Reconciler GUID remap | Axiell reconciler step | Authoritative delete: old GUID superseded; suppress its FOLIO records | -### Every Record in a Changeset Is Either +### How records are applied: create, update, delete -Each record in a loader changeset is treated as **new** (the first time this `id` appears in Iceberg), in which case it is created in FOLIO, or **updated** (an existing `id` with a newer `last_modified`), in which case it is updated in FOLIO. Deletions are handled on the reconciler path (see [Delete detection](#delete-detection-the-reconciler-not-oai-tombstones) below), not by directly suppressing on the loader's `deleted=true` flag. The exact delete action in FOLIO (suppress vs remove, and cascade scope) remains an open policy question (see [Delete semantics](#delete-semantics-what-should-reconciler-detected-deletes-do-in-folio)). +Each record in a loader changeset is **new** (the first time this `id` appears in Iceberg) and is created in FOLIO, or **updated** (an existing `id` with a newer `last_modified`) and is updated in FOLIO. -### Delete detection: the reconciler, not OAI tombstones +**Deletes are driven by the reconciler, not the loader's `deleted=true` tombstone** (which is unreliable — the very reason the adapter has a separate reconciler step). The reconciler tracks the `collectId → guid` mapping in its own Iceberg store and, when a `collectId` is remapped, emits a `DeletedSourceWork` for the superseded GUID. Two things follow for the upserter: it consumes the **loader's** changeset, so it never sees these deletes directly, and a reconciler delete is keyed by the **old (superseded) GUID**, not the changeset row being processed. -Axiell's OAI tombstones (`deleted=true`) are unreliable, which is the very reason the adapter has a separate **reconciler** step. The reconciler tracks the `collectId → guid` mapping in its own Iceberg store and, when a `collectId` is remapped to a different work, emits a `DeletedSourceWork` for the superseded GUID. +**Suggested approach:** have the reconciler fan out a FOLIO suppression path mirroring the loader's fan-out to the upsert path (reusing the existing reconciler rather than building a parallel mapping store). On a reconciler delete, suppress the records for the old GUID (`AxC-instance-{old-guid}`) and cascade to its holdings and item; the loader `deleted=true` stays a best-effort secondary signal only. The exact delete action (suppress vs remove, and cascade scope) is an open policy question — see [Delete semantics](#delete-semantics-what-should-reconciler-detected-deletes-do-in-folio). -This has two consequences for the FOLIO upserter. First, the upserter consumes the **loader's changeset**, so it does not see reconciler-detected deletes, which are produced by a separate, later transformer step. Second, a reconciler delete is keyed by the **old (superseded) GUID**, not by the changeset row being processed. +### Upsert Key Strategy (Idempotency) + +Every entity is matched against its existing FOLIO record by `hrid` (a GET before each write), then created or updated — no blind creates, which is what makes replay idempotent: + +- Instance: `GET /inventory/instances?query=(hrid==AxC-instance-{guid})` +- Holdings: `GET /holdings-storage/holdings?query=(hrid==AxC-holding-{guid})` +- Item: `GET /inventory/items?query=(hrid==AxC-item-{guid})` -Suggested approach: have the **reconciler** step fan out a FOLIO suppression path that mirrors the loader's fan-out to the upsert path, reusing the existing reconciler rather than building a parallel `collectId → guid` mapping store. On a reconciler delete, suppress the FOLIO records for the old GUID (`AxC-instance-{old-guid}`) and cascade to its holdings and item, in line with the [Delete semantics](#delete-semantics-what-should-reconciler-detected-deletes-do-in-folio) question raised earlier. The loader-changeset `deleted=true` can remain a best-effort secondary signal, but the reconciler fan-out is the authoritative delete path. +The `hrid` is derived from the Axiell GUID (MARC 001), not the `collectId` (object number). Axiell reuses `collectId`s; keying on GUID prevents a reused id from overwriting the wrong FOLIO record. + +**Behavior:** when a record is found, its mutable fields are updated while FOLIO-internal metadata is preserved; when it isn't found, a new record is created; and when required fields are missing, the record is skipped with a structured error logged. An update applies only if it passes the stale-write guard (incoming Axiell `last_modified` strictly newer than the watermark on the existing FOLIO record), so out-of-order or replayed changesets never overwrite newer state (see [Invocation Pattern](design-rationale.md#invocation-pattern-ordering--concurrency)). + +**Replay safety:** upserts are idempotent by `hrid`, so processing the same changeset twice produces the same outcome, and manifest deduplication checks whether a changeset was already processed successfully before running it again. --- @@ -438,7 +383,7 @@ flowchart TD #### Step 1: Authenticate with FOLIO -Auth is delegated to the shared `folio-client` (see [FOLIO API Client](#folio-api-client)): a `FolioClient` is built with a credentials provider that reads SSM, and it logs in lazily on first use. +Auth is delegated to the shared `folio-client` package: a `FolioClient` is built with a credentials provider that reads SSM, logs in lazily on first use, caches the token (valid 24h) for the invocation, and re-authenticates once automatically on a 401. ``` SSM Parameter Store (SecureString) Path: /axiell-folio-sync/okapi-creds @@ -456,17 +401,10 @@ FolioClient → POST /authn/login (lazy, on first request) Query S3 Tables Iceberg catalog Bucket: {S3_TABLE_BUCKET_ARN} Table: {ICEBERG_TABLE_NAME} (e.g., default.axiell_changesets) - + SELECT [namespace, id, content, changeset, last_modified, deleted] WHERE changeset IN ({changeset_ids from event}) - -Schema: - namespace string e.g., "location", "item" - id string external identifier from Axiell - content string raw MARCXML payload from OAI-PMH - changeset string changeset ID from adapter - last_modified timestamp when record changed - deleted boolean true if marked deleted + # row schema described under Change Detection > How It Works ``` #### Step 3: Map & Validate @@ -508,7 +446,7 @@ Every entity is matched by its **hrid** (from `meta`) and created or updated (no blind creates), which is what makes replay idempotent. On an existing record the update is additionally gated by the **stale-write guard**: it applies only if the incoming Axiell `last_modified` is strictly newer than the watermark on the FOLIO -record, otherwise the record is skipped (see [Invocation Pattern](#invocation-pattern-eventbridge-trigger--ordering)). +record, otherwise the record is skipped (see [Invocation Pattern](design-rationale.md#invocation-pattern-ordering--concurrency)). ``` For each mapped record: @@ -537,20 +475,6 @@ Per-record error handling: • All errors logged to CloudWatch AND accumulated in the S3 failures manifest ``` -### Upsert Key Strategy (Idempotency) - -**Matching**: All entities are matched against existing FOLIO records by `hrid`: - -- Instance: `GET /inventory/instances?query=(hrid==AxC-instance-{guid})` -- Holdings: `GET /holdings-storage/holdings?query=(hrid==AxC-holding-{guid})` -- Item: `GET /inventory/items?query=(hrid==AxC-item-{guid})` - -The `hrid` is derived from the Axiell GUID (MARC 001), not the `collectId` (object number). Axiell reuses `collectId`s; keying on GUID prevents a reused id from overwriting the wrong FOLIO record. The reconciler emits `DeletedSourceWork` events when a GUID is superseded, so old FOLIO records are cleaned up automatically. - -**Behavior:** when a record is found, its mutable fields are updated while FOLIO-internal metadata is preserved; when it isn't found, a new record is created; and when required fields are missing, the record is skipped with a structured error logged. An update applies only if it passes the stale-write guard (incoming Axiell `last_modified` strictly newer than the watermark on the existing FOLIO record), so out-of-order or replayed changesets never overwrite newer state (see [Invocation Pattern](#invocation-pattern-eventbridge-trigger--ordering)). - -**Replay safety:** upserts are idempotent by `hrid`, so processing the same changeset twice produces the same outcome, and manifest deduplication checks whether a changeset was already processed successfully before running it again. - #### Step 5: Batch & Write Manifests ```python # Success NDJSON (one record per line, 5K records per batch) @@ -663,134 +587,28 @@ manifests/ --- -## Design Rationale: Why These Choices? +## Design Decisions -### Mapping: Pydantic Models vs. YAML Mapper +The decisions below are summarized in [Key Design Considerations](#key-design-considerations); the full trade-off analysis for each (alternatives weighed, why this option won) lives in **[design-rationale.md](design-rationale.md)**. -| Approach | Pros | Cons | -|----------|------|------| -| **A: YAML mapper** (`mapping.yaml` + `YamlMapper`) | Mapping config separated from code; in principle non-programmers can read/adjust rules; changes are config-only | Errors surface only at runtime (often as a FOLIO 422); no type safety, so a typo'd key or wrong shape passes silently; a second engine (YAML schema + interpreter) to build, test, and document; awkward for normalization tables, per-field defaults, and composite hrids | -| **B: Typed Pydantic models** (`mapping.py`, chosen) | Validation at build time before any OKAPI call; payload shape is a typed contract (`extra="forbid"`); rules, defaults, and contracts in one reviewable module; full Python for normalization/conditionals; payloads stamped with `mapping_version` | Mapping changes need a Python edit + redeploy, not a config tweak; reading the rules assumes Python familiarity | +| Decision | Choice | Rationale | +|----------|--------|-----------| +| **Mapping** | Typed Pydantic models (`mapping.py`), not a YAML mapper | [Mapping](design-rationale.md#mapping-pydantic-models-vs-yaml-mapper) | +| **Orchestration** | Step Functions + Lambda, not direct Lambda or SQS | [Orchestration](design-rationale.md#orchestration-step-functions-vs-alternatives) | +| **Reference caching** | Reload-per-run; defined scale-up progression if it grows | [Caching](design-rationale.md#caching-strategy-reference-data-scaling-options) | +| **Manifest storage** | S3 NDJSON (90-day TTL), not DynamoDB or streaming | [Storage](design-rationale.md#storage-s3-ndjson-vs-alternatives) | +| **Ordering safety** | Source-timestamp stale-write guard + Step Functions concurrency = 1 | [Invocation Pattern](design-rationale.md#invocation-pattern-ordering--concurrency) | +| **Error handling** | Per-record isolation; batch-level errors retry then fail | [Error Handling](design-rationale.md#error-handling-per-record-isolation) | +| **Instance storage type** | Inventory-native (`Instance.source = "FOLIO"`), no linked SRS record | [SRS-backed Instances](design-rationale.md#srs-backed-instances-and-the-update-path) | -We chose the typed Pydantic models (B). The decisive factor is *when* errors surface: with typed models a missing or ill-typed required field (or an unknown key, since the models are `extra="forbid"`) raises `MappingError`/`ValidationError` during the build step, before any OKAPI call, whereas the YAML approach deferred the same failures to a FOLIO 422 mid-batch that is far harder to trace. Making the payload a typed contract also means a typo'd key is a build error rather than a silently dropped field, so the mapping and the FOLIO contract cannot drift apart unnoticed. Beyond correctness, it collapses two engines into one: ordinary Python replaces a bespoke YAML schema and its interpreter, so there is less to build, test, and document, and the normalization tables, defaults, and hrid scheme live beside the models. Every payload is also stamped with `mapping_version`, so any record in FOLIO or a manifest traces back to the exact rules that produced it. +The **ordering** and **error-handling** mechanisms referenced above are load-bearing for correctness, so they are stated as behaviour in [Step 4: Upsert to FOLIO](#step-4-upsert-to-folio-per-record) and the [Upsert Key Strategy](#upsert-key-strategy-idempotency); design-rationale.md explains *why* they take the form they do. -The YAML mapper's headline appeal, letting non-programmers edit mappings as config, every rule change still went through review, tests, and a deploy, so config-versus-code made little practical difference, while the absence of type safety let malformed output reach FOLIO and fail there. As the rules grew (normalization tables, per-field defaults, composite GUID-based hrids) they simply read more clearly as Python than as declarative YAML. +> **Instance storage type (decision).** This sync creates Inventory-native instances (`Instance.source = "FOLIO"`, no linked SRS record), which keeps the bibliographic fields editable through mod-inventory and keeps the PUT-based update path valid. SRS-backed instances cannot be updated through the mod-inventory API, so a mixed estate is avoided. Confirmed via the prototype: these records are updatable through mod-inventory and are received on the FOLIO adapter in the catalogue pipeline; one gap (item notes not appearing on the OAI-PMH feed) remains open, see [Item notes not visible on the FOLIO OAI-PMH feed](#item-notes-not-visible-on-the-folio-oai-pmh-feed). Full reasoning: [design-rationale.md](design-rationale.md#srs-backed-instances-and-the-update-path). -> The YAML mapper (`mapping.yaml` + `YamlMapper`) was the original prototype design and has been **superseded** by the Pydantic approach. Remaining `mapping.yaml` references elsewhere in this RFC are historical. +For the cost evidence behind the "~$3–5/month" figure, see [Cost Analysis](design-rationale.md#cost-analysis). --- -### Orchestration: Step Functions vs. Alternatives - -| Approach | Pros | Cons | Cost/mo | -|----------|------|------|---------| -| **A: Direct Lambda** (EventBridge → Lambda, fire-and-forget) | Fewer services, lower overhead, fastest time-to-invocation | No built-in retry logic, no execution history, hard to extend, silent failures possible | ~$2 | -| **B: Step Functions + Lambda** (chosen) | Explicit retry policy + audit trail, easy to parallelize later, clear visibility into success/failure | Extra service, marginal latency added | ~$3–5 | -| **C: Async Queue** (EventBridge → SQS → Lambda workers) | Decouples producer/consumer, handles bursts, resilient to adapter restarts | Complex ordering semantics, harder to reason about, no clear success/failure signal, monitoring overhead | ~$10–15 | - -We chose Step Functions + Lambda (B): it buys an explicit retry policy and a full execution history. Every state transition is logged to CloudWatch, so a sync that fails partway is diagnosable from the execution history rather than by reconstructing state from Lambda logs, and max attempts and backoff are declarative, so operations can tune retries in the state-machine definition without code changes. The adapter triggers the sync asynchronously (`StartExecution`) and does not block; ordering safety comes from source-timestamp watermarking plus a Step Functions concurrency limit of 1 (see [Invocation Pattern](#invocation-pattern-eventbridge-trigger--ordering)), not from backpressure. The design also leaves a clean extension point for scale: when volume outgrows a single invocation we swap in a Step Functions `Map` state to fan out per changeset, with no change to the manifest format. The cost is immaterial: pennies a month in state transitions at ~80 syncs/day (~2,400/month). - -A direct EventBridge→Lambda trigger (A) was rejected because it gives no failure signal, no automatic retry, and no execution history, so debugging would mean parsing Lambda logs. An async SQS queue (C) is cheaper per invocation but doesn't guarantee processing order within a batch, needs workers to coordinate which changeset they own, and carries monitoring overhead (error tracking, retries, dead-letter management) that isn't justified at current volume. - -### Caching Strategy: Reference Data Scaling Options - -Reload-per-run assumes the reference set is small enough to bulk-load into Lambda memory cheaply (see [Reference Data Cache](#reference-data-cache-ref_cachepy)). If we ever need **much more** reference data (many more types, or large per-record lookups), the choice of caching architecture matters. Options, cheapest first: - -| Option | What it is | Best when | Cost / overhead | Freshness control | -|--------|-----------|-----------|-----------------|-------------------| -| **Warm singleton + reload-on-miss** | Module-global cache reused across warm invocations; fetch-and-memoize a code on a cache miss | Set still fits in memory; modest growth | None (in-process) | Self-healing on miss; lost on cold start | -| **S3 snapshot + scheduled refresher** | A cron (EventBridge) Lambda rebuilds a serialized snapshot (JSON/Parquet) from FOLIO every N hours; the sync Lambda reads it at startup (one GET) instead of N FOLIO calls | Whole set scanned each run; want to decouple sync from FOLIO availability | ~pennies/mo + one extra scheduled Lambda | Snapshot age = refresh cadence; pair with reload-on-miss | -| **DynamoDB lookup table** | Refresher populates `(type, code) → uuid`; sync Lambda `BatchGetItem`s only the codes a changeset needs | Set too big for memory, or only a subset is needed per run | On-demand $/read; native TTL | TTL / refresher cadence; pair with reload-on-miss | -| **ElastiCache (Redis)** | Shared in-memory cache across all invocations | Very large + hot lookups at high concurrency | Always-on (~$12+/mo), **requires VPC** (ENI cold-start + NAT to reach FOLIO/AWS APIs) | TTL | - -**Recommended progression:** stay on reload-per-run → **S3 snapshot + scheduled refresher** (cheap, no VPC, scales via columnar formats, decouples from FOLIO reference endpoints) → **DynamoDB** only once the set outgrows Lambda memory or you genuinely need selective point lookups → **ElastiCache** only for high-concurrency hot-lookup workloads that justify always-on infra and the VPC tax. In every tier, keep a **reload-on-miss fallback to FOLIO** so a newly-added code resolves on first sight rather than failing the record. - ---- - -### Storage: S3 NDJSON vs. Alternatives - -| Approach | Pros | Cons | Cost/mo (90-day) | -|----------|------|------|------------------| -| **A: DynamoDB** (per-record writes, TTL) | Query-friendly, real-time dashboards possible, strong consistency | O(n) write cost, schema evolution painful, expensive for batches, not auditable | ~$20–50 | -| **B: S3 NDJSON** (chosen) | Batched writes (low cost), queryable (download + `jq`; Athena or S3 Select optional), matches the pipeline's NDJSON-manifest pattern, audit via versioning, easy to compress/archive | Requires S3 Select for queries (not real-time), not ideal for high-cardinality point lookups | ~$1–2 | -| **C: Streaming** (Kinesis/Firehose → Parquet) | High throughput, good for ML pipelines, efficient compression | Overkill for current volume, adds infrastructure complexity, higher cost | ~$5–15 | - -We chose S3 NDJSON (B), and write-scaling is the main driver: per-record DynamoDB writes scale O(n) with record count, whereas the upserter batches its manifest writes into one S3 PUT per run (up to 5,000 records per object). At the current ~1,000 records/day across ~80 syncs that is about one object per run, so the batched model stays flat as volume grows rather than scaling per record. It is also explicitly the existing pipeline pattern rather than a new one: the adapter platform already emits per-job **NDJSON manifests** to S3, so reusing that format (and S3 generally) keeps backup, recovery, query, and downstream tooling consistent across the pipeline. And it stays queryable after the fact without any ETL. **S3 Select** is one convenient option (e.g. `SELECT * FROM s3://bucket/manifest.ndjson WHERE errors > 0`, ~500 ms on a 100 KB file), but note S3 Select is **not currently used elsewhere in the org**, so we don't depend on it: because the manifests are plain NDJSON, a failed-record list is equally available by just downloading the object and piping it through `jq`/`grep`, or by pointing Athena at the prefix if ad-hoc SQL is ever wanted. S3 Select is a nicety, not a load-bearing part of the design; whether to adopt it is an open question (below). S3 versioning provides an audit trail (every write is a recoverable object version, with the metadata JSON tracking historical jobs), and after 90 days manifests archive cheaply to Glacier, something DynamoDB has no low-cost equivalent for. - -DynamoDB (A) was rejected on fit rather than headline cost (at ~1,000 writes/day either store is cheap), because its per-record write model scales O(n), schema changes require a scan-and-rewrite, and there's no clean way to answer audit questions like "what was deleted last week?". Streaming via Kinesis/Firehose (C) is built for thousands of records per second (overkill for 10–500-record changesets) and adds buffer/flush operational complexity we don't need. - ---- - -### Invocation Pattern: EventBridge Trigger & Ordering - -EventBridge publishes changeset events to Step Functions via `StartExecution` (asynchronous, fire-and-forget); the adapter does not block waiting for the sync to complete. Step Functions provides the execution visibility and retry policy, and all invocations (initial or retry) are decoupled from the adapter. - -However, concurrent or retried executions create an **ordering risk**: if two changesets arrive within ~45 s of each other, or if a failed changeset is retried while a newer one is in flight, a naive last-write-wins upsert could apply an older state after a newer one, corrupting the FOLIO record. Similarly, idempotent replays (re-running failed changesets after a fix) must be no-ops if a newer changeset has already updated the same record. - -**Solution: Source-Timestamp Watermarking.** The upsert layer compares each incoming record's Axiell `last_modified` timestamp (from the source data) against a watermark stored on the FOLIO record. The write proceeds only if the incoming timestamp is strictly newer; otherwise the record is skipped, leaving the FOLIO state intact. Where exactly that watermark lives in FOLIO (an administrative note, a custom property, or a version field) is itself an open question; see [Watermark storage](#watermark-storage-where-does-the-source-timestamp-live). This ensures: -- Out-of-order concurrent executions apply updates in source-timestamp order, not invocation order. -- Replays (re-running failed changesets) are idempotent: an older changeset re-run after a newer one succeeds does not overwrite the newer state. -- No coupling between adapter and sync (the adapter remains fire-and-forget). - -**Concurrency control:** A Step Functions concurrency limit of 1 (via the state machine definition) is a secondary guard, preventing the GET–PUT gap between a query and an upsert; with a limit of 1, only one execution can be in the `Lambda` state at a time, so even if two changesets arrive within milliseconds, the second waits for the first to commit. This is loose (the adapter cadence is 15 minutes, so concurrency is naturally rare) but provides an extra safeguard against read-skew during retries. - -Combined, these two mechanisms ensure ordering safety without blocking the adapter or introducing a queue. - ---- - -### Error Handling: Per-Record Isolation - -The guiding policy is that one bad record must never halt the batch. - -Per-record errors are non-blocking. A mapping error (Pydantic validation, a missing required field, or an unresolved reference) is captured as `{"source_id": "...", "type": "mapping", "detail": "..."}` and an API error (FOLIO returns 4xx/5xx) as `{"source_id": "...", "type": "api", "detail": "..."}`; in either case the record is written to the S3 error manifest and processing moves on to the next record. Batch-level errors, by contrast, are blocking: an OKAPI auth failure, an Iceberg connection failure, or an S3 write failure raises an exception that the Step Function retries up to `max_retries`, and if those are exhausted the execution ends in the terminal `SyncFailed` state and raises an alert (see the alert path below). - -Failures are observable across four channels: CloudWatch Logs (`/aws/lambda/axiell-folio-sync`) for detailed per-record execution, the S3 `.ids.failures.ndjson` manifest for a queryable error list (via a plain download + `jq`, or optionally S3 Select; see the storage note), CloudWatch Metrics (`RecordsFailed`, `RecordsCreated`, and the like) for alerting on a high failure rate, and the Step Function history (`/aws/states/axiell-folio-sync-sfn`) for state transitions and retry events. - -**Skipped records are also accounted for, not dropped silently.** Records skipped before a write (no `980 $a` harvest flag, `Level` gating, or the stale-write guard) are counted in a `RecordsSkipped` metric and recorded in the run's metadata manifest, so "why didn't record X appear in FOLIO?" is answerable after the fact rather than being an invisible no-op. - -**Alert path:** alerting follows the existing pipeline pattern: a **CloudWatch metric alarm** (on `RecordsFailed`/the failure rate, and on `SyncFailed` Step Function executions) fires into the team's **Slack** channel via **Amazon Q Developer in chat applications**. No new notification mechanism is introduced; the alarms are wired to the same SNS-topic → Amazon Q → Slack route already used elsewhere in the catalogue pipeline. - ---- - -## SRS-backed Instances and the Update Path - -A consideration worth making explicit for the update path. Some FOLIO instances are backed by **Source Record Storage (SRS)** as MARC, for example anything migrated or loaded as MARC. For those records the bibliographic fields are controlled by the underlying SRS MARC record and **cannot be updated through the mod-inventory instance API**; only administrative data is editable there. MARC edits go through **quickMARC**, which writes to SRS and syncs the Inventory record. A `PUT` to mod-inventory may therefore fail or be silently ignored for an SRS-backed instance, whereas the current update logic assumes the PUT takes effect. - -**which storage do we create records in, **Inventory-native** (FOLIO source) or **MARC/SRS**?** -Records created the two ways behave differently on update, so a mixed estate is harder to reason about and maintain. This sync currently creates Inventory-native instances (`Instance.source = "FOLIO"`, with no linked SRS record), which keeps the bibliographic fields editable through mod-inventory and keeps the PUT-based update path valid. - -**Catalogue-pipeline impact:** the catalogue pipeline harvests FOLIO over OAI-PMH using the `marc21_withholdings` prefix (see `catalogue_graph/src/adapters/extractors/oai_pmh/folio/config.py`). Under that prefix the instance bib comes from SRS when an SRS record is present, or is generated on the fly from Inventory depending on the **mod-oai-pmh record-source** setting, while holdings and items come from Inventory. So whether the records this sync creates appear in that feed, and in what form, depends on the storage type we choose together with the mod-oai-pmh configuration. This has now been confirmed via the prototype for Inventory-native records: a `source = "FOLIO"` instance created here is updatable through mod-inventory and is received on the FOLIO adapter in the catalogue pipeline. One gap remains under investigation, the item notes not appearing on the OAI-PMH feed; see [Item notes not visible on the FOLIO OAI-PMH feed](#item-notes-not-visible-on-the-folio-oai-pmh-feed). - ---- - -## FOLIO API Client - -OKAPI traffic goes through the shared **`folio-client`** package (`prototypes/folio-client/`), the same dependency-free OKAPI client used by the folio-api Lambda and the CLI scripts. It is pure standard library (`urllib` + `ssl`), so it adds no third-party dependency to the bundle, and the build vendors its source into the image. The Lambda declares it as a path dependency (`[tool.uv.sources]` in `axiell-folio-sync/pyproject.toml`) and wires it up in `axiell_folio_sync.py`: a `FolioClient` is constructed with a `credentials_provider` (the SSM lookup) and `ssl_context_from_env()`, then a thin adapter (`_make_folio_callables`) wraps it into the `folio_get` / `folio_post` / `folio_put` callables that `ref_cache.py` and `upsert.py` consume. - -**Responsibilities:** the client authenticates to FOLIO via OKAPI (`/authn/login`) and refreshes the token automatically on a 401; the upsert layer (`upsert.py`) resolves existing records by CQL query (a GET before each write) and creates (POST), updates (PUT), or suppresses deleted records per entity. - -**Token management:** `FolioClient` logs in lazily on the first request using credentials from SSM, caches the resulting token (valid 24 hours, comfortably longer than the 5-minute Lambda execution) for the whole invocation (and across warm invocations on the same instance), and re-authenticates once automatically on a 401, so there is no per-call refresh overhead. - ---- - - -## Cost Analysis - -### Current Volume: ~1,000 records/day - -At ~80 syncs/day the pipeline runs ~2,400 times/month, so the monthly quantities below derive from that: - -| Service | Operation | Qty/mo | Rate | Cost/mo | -|---------|-----------|--------|------|---------| -| Lambda | ~2,400 invocations × ~60s × 512 MB | ~2,400 invocations | $0.0000167/GB-s | ~$1.20 | -| Step Functions | ~2,400 state transitions | ~2,400 transitions | $0.000025/transition | ~$0.06 | -| EventBridge | ~2,400 events | ~2,400 events | $1/M events | ~$0.00 | -| S3 (manifests) | ~2,400 objects written, 90-day retention | ~2,400 objects + storage | $0.005/K PUTs + $0.023/GB/mo | ~$1.50 | -| CloudWatch Logs | ~2,400 × 5 KB ≈ 12 MB/month | 12 MB ingested | $0.50/GB ingested | ~$0.20 | -| **Total** | | | | **~$3–5** | - - ---- ## Open Questions ### Field Mapping from Axc to Folio Instance, Holdings and Items needs to be defined @@ -805,11 +623,11 @@ Specific points to resolve with Collection Information: whether suppressed items ### Watermark storage: where does the source timestamp live? -The stale-write guard (see [Invocation Pattern](#invocation-pattern-eventbridge-trigger--ordering)) needs the Axiell `last_modified` of the last applied change stored **on the FOLIO record** so the next write can compare against it. FOLIO does not offer an obvious home for this: `_version` is mod-inventory's optimistic-locking counter (not a source timestamp), and `discoverySuppress` is a boolean. Candidates are an **administrative note**, a **custom property**, or a dedicated field, each with trade-offs for visibility, OAI-PMH leakage, and whether it survives a quickMARC/SRS round-trip. The storage location needs to be decided (and confirmed not to pollute the catalogue feed) before the guard can be implemented. +The stale-write guard (see [Invocation Pattern](design-rationale.md#invocation-pattern-ordering--concurrency)) needs the Axiell `last_modified` of the last applied change stored **on the FOLIO record** so the next write can compare against it. FOLIO does not offer an obvious home for this: `_version` is mod-inventory's optimistic-locking counter (not a source timestamp), and `discoverySuppress` is a boolean. Candidates are an **administrative note**, a **custom property**, or a dedicated field, each with trade-offs for visibility, OAI-PMH leakage, and whether it survives a quickMARC/SRS round-trip. The storage location needs to be decided (and confirmed not to pollute the catalogue feed) before the guard can be implemented. ### Reference-data caching: when and how to cache across Lambda runs -Today `RefCache` reloads all six reference sets on every invocation, which is the right trade-off at current volume (see [Reference Data Cache](#reference-data-cache-ref_cachepy)). The open question is what happens if the reference set grows substantially. We would need to decide the size or load-latency threshold at which caching across runs is worth the added complexity. Staleness is unlikely to be the blocker: the reference data changes rarely, so a long TTL would seldom be stale, and a reload-on-miss fallback would keep a newly-added code from failing as a `MappingError`. So the question is mostly about whether the saved reload justifies the extra infrastructure. We would also need to pick a tier when we get there: the progression runs from a warm-singleton with reload-on-miss, to an S3 snapshot rebuilt by a scheduled refresher, to DynamoDB, and only then to ElastiCache (see [Caching Strategy: Reference Data Scaling Options](#caching-strategy-reference-data-scaling-options) for the trade-offs). +Today `RefCache` reloads all six reference sets on every invocation, which is the right trade-off at current volume (see [Reference Data Cache](#reference-data-cache-ref_cachepy)). The open question is what happens if the reference set grows substantially. We would need to decide the size or load-latency threshold at which caching across runs is worth the added complexity. Staleness is unlikely to be the blocker: the reference data changes rarely, so a long TTL would seldom be stale, and a reload-on-miss fallback would keep a newly-added code from failing as a `MappingError`. So the question is mostly about whether the saved reload justifies the extra infrastructure. We would also need to pick a tier when we get there: the progression runs from a warm-singleton with reload-on-miss, to an S3 snapshot rebuilt by a scheduled refresher, to DynamoDB, and only then to ElastiCache (see [Caching Strategy: Reference Data Scaling Options](design-rationale.md#caching-strategy-reference-data-scaling-options) for the trade-offs). ### Manifest query mechanism: S3 Select is not an established org pattern @@ -822,18 +640,3 @@ The legacy CALM→Sierra transform (`docs/discovery/CalmInnopac.xsl`) only attac ### Item notes not visible on the FOLIO OAI-PMH feed Confirmed via the prototype: FOLIO-native instances (`source = "FOLIO"`) created by this sync **can** be updated through mod-inventory, and the records **are** received on the FOLIO adapter in the catalogue pipeline (so the storage-type and catalogue-feed concerns are largely settled for Inventory-native records). One issue remains open: the item **notes list does not appear on the OAI-PMH feed** (`marc21_withholdings`). The records otherwise come through, but the notes we set (for example the `Axiell location` note) are missing from the harvested output. This needs further investigation, likely into how mod-oai-pmh renders Inventory item notes in the MARC output and whether a note type, `staffOnly`, or suppression setting hides them. - ---- - -## Next Steps - -- **Initial backfill / cutover (to be worked out).** The design above handles the ongoing 15-minute changeset flow, but the first task is populating FOLIO with the **entire existing harvest-flagged (`980 $a`) corpus**, not a single window. We need to work out how the one-off initial load runs (replay all historical changesets, or a full Iceberg scan filtered on the harvest flag), how it is validated against FOLIO, how it is throttled so it does not overwhelm OKAPI, and how it sequences with the switch-over to the steady-state event-driven sync. - ---- - -## References - -- **Axiell Adapter Platform**: Emits changesets to Iceberg; documentation in location-movement-control-docs -- **FOLIO API**: https://api-wellcome.folio.ebsco.com (OKAPI auth required) -- **AWS S3 Tables**: Iceberg catalog on S3; managed via Terraform -- **Mapping rules**: `mapping.py`, the typed Pydantic Instance/Holdings/Item models + builders (bundled in the Lambda image) diff --git a/rfcs/090-axiell-folio-sync/design-rationale.md b/rfcs/090-axiell-folio-sync/design-rationale.md new file mode 100644 index 00000000..f418fc98 --- /dev/null +++ b/rfcs/090-axiell-folio-sync/design-rationale.md @@ -0,0 +1,134 @@ +# RFC 090: Design Rationale + +This document records *why* the [RFC 090 CMS→LMS sync](README.md) makes the +choices it does. The README states the decisions; this document holds the +trade-off analysis behind them, plus the cost evidence that backs the +"keep it simple" conclusions. + +## Table of Contents + +- [Mapping: Pydantic Models vs. YAML Mapper](#mapping-pydantic-models-vs-yaml-mapper) +- [Orchestration: Step Functions vs. Alternatives](#orchestration-step-functions-vs-alternatives) +- [Caching Strategy: Reference Data Scaling Options](#caching-strategy-reference-data-scaling-options) +- [Storage: S3 NDJSON vs. Alternatives](#storage-s3-ndjson-vs-alternatives) +- [Invocation Pattern: Ordering & Concurrency](#invocation-pattern-ordering--concurrency) +- [Error Handling: Per-Record Isolation](#error-handling-per-record-isolation) +- [SRS-backed Instances and the Update Path](#srs-backed-instances-and-the-update-path) +- [Cost Analysis](#cost-analysis) + +--- + +## Mapping: Pydantic Models vs. YAML Mapper + +| Approach | Pros | Cons | +|----------|------|------| +| **A: YAML mapper** (`mapping.yaml` + `YamlMapper`) | Mapping config separated from code; in principle non-programmers can read/adjust rules; changes are config-only | Errors surface only at runtime (often as a FOLIO 422); no type safety, so a typo'd key or wrong shape passes silently; a second engine (YAML schema + interpreter) to build, test, and document; awkward for normalization tables, per-field defaults, and composite hrids | +| **B: Typed Pydantic models** (`mapping.py`, chosen) | Validation at build time before any OKAPI call; payload shape is a typed contract (`extra="forbid"`); rules, defaults, and contracts in one reviewable module; full Python for normalization/conditionals; payloads stamped with `mapping_version` | Mapping changes need a Python edit + redeploy, not a config tweak; reading the rules assumes Python familiarity | + +We chose the typed Pydantic models (B). The decisive factor is *when* errors surface: with typed models a missing or ill-typed required field (or an unknown key, since the models are `extra="forbid"`) raises `MappingError`/`ValidationError` during the build step, before any OKAPI call, whereas the YAML approach deferred the same failures to a FOLIO 422 mid-batch that is far harder to trace. Making the payload a typed contract also means a typo'd key is a build error rather than a silently dropped field, so the mapping and the FOLIO contract cannot drift apart unnoticed. Beyond correctness, it collapses two engines into one: ordinary Python replaces a bespoke YAML schema and its interpreter, so there is less to build, test, and document, and the normalization tables, defaults, and hrid scheme live beside the models. Every payload is also stamped with `mapping_version`, so any record in FOLIO or a manifest traces back to the exact rules that produced it. + +The YAML mapper's headline appeal, letting non-programmers edit mappings as config, every rule change still went through review, tests, and a deploy, so config-versus-code made little practical difference, while the absence of type safety let malformed output reach FOLIO and fail there. As the rules grew (normalization tables, per-field defaults, composite GUID-based hrids) they simply read more clearly as Python than as declarative YAML. + +> The YAML mapper (`mapping.yaml` + `YamlMapper`) was the original prototype design and has been **superseded** by the Pydantic approach. Remaining `mapping.yaml` references elsewhere in this RFC are historical. + +--- + +## Orchestration: Step Functions vs. Alternatives + +| Approach | Pros | Cons | Cost/mo | +|----------|------|------|---------| +| **A: Direct Lambda** (EventBridge → Lambda, fire-and-forget) | Fewer services, lower overhead, fastest time-to-invocation | No built-in retry logic, no execution history, hard to extend, silent failures possible | ~$2 | +| **B: Step Functions + Lambda** (chosen) | Explicit retry policy + audit trail, easy to parallelize later, clear visibility into success/failure | Extra service, marginal latency added | ~$3–5 | +| **C: Async Queue** (EventBridge → SQS → Lambda workers) | Decouples producer/consumer, handles bursts, resilient to adapter restarts | Complex ordering semantics, harder to reason about, no clear success/failure signal, monitoring overhead | ~$10–15 | + +We chose Step Functions + Lambda (B): it buys an explicit retry policy and a full execution history. Every state transition is logged to CloudWatch, so a sync that fails partway is diagnosable from the execution history rather than by reconstructing state from Lambda logs, and max attempts and backoff are declarative, so operations can tune retries in the state-machine definition without code changes. The adapter triggers the sync asynchronously (`StartExecution`) and does not block; ordering safety comes from source-timestamp watermarking plus a Step Functions concurrency limit of 1 (see [Invocation Pattern](#invocation-pattern-ordering--concurrency)), not from backpressure. The design also leaves a clean extension point for scale: when volume outgrows a single invocation we swap in a Step Functions `Map` state to fan out per changeset, with no change to the manifest format. The cost is immaterial: pennies a month in state transitions at ~80 syncs/day (~2,400/month). + +A direct EventBridge→Lambda trigger (A) was rejected because it gives no failure signal, no automatic retry, and no execution history, so debugging would mean parsing Lambda logs. An async SQS queue (C) is cheaper per invocation but doesn't guarantee processing order within a batch, needs workers to coordinate which changeset they own, and carries monitoring overhead (error tracking, retries, dead-letter management) that isn't justified at current volume. + +--- + +## Caching Strategy: Reference Data Scaling Options + +Reload-per-run assumes the reference set is small enough to bulk-load into Lambda memory cheaply (see [Reference Data Cache](README.md#reference-data-cache-ref_cachepy)). If we ever need **much more** reference data (many more types, or large per-record lookups), the choice of caching architecture matters. Options, cheapest first: + +| Option | What it is | Best when | Cost / overhead | Freshness control | +|--------|-----------|-----------|-----------------|-------------------| +| **Warm singleton + reload-on-miss** | Module-global cache reused across warm invocations; fetch-and-memoize a code on a cache miss | Set still fits in memory; modest growth | None (in-process) | Self-healing on miss; lost on cold start | +| **S3 snapshot + scheduled refresher** | A cron (EventBridge) Lambda rebuilds a serialized snapshot (JSON/Parquet) from FOLIO every N hours; the sync Lambda reads it at startup (one GET) instead of N FOLIO calls | Whole set scanned each run; want to decouple sync from FOLIO availability | ~pennies/mo + one extra scheduled Lambda | Snapshot age = refresh cadence; pair with reload-on-miss | +| **DynamoDB lookup table** | Refresher populates `(type, code) → uuid`; sync Lambda `BatchGetItem`s only the codes a changeset needs | Set too big for memory, or only a subset is needed per run | On-demand $/read; native TTL | TTL / refresher cadence; pair with reload-on-miss | +| **ElastiCache (Redis)** | Shared in-memory cache across all invocations | Very large + hot lookups at high concurrency | Always-on (~$12+/mo), **requires VPC** (ENI cold-start + NAT to reach FOLIO/AWS APIs) | TTL | + +**Recommended progression:** stay on reload-per-run → **S3 snapshot + scheduled refresher** (cheap, no VPC, scales via columnar formats, decouples from FOLIO reference endpoints) → **DynamoDB** only once the set outgrows Lambda memory or you genuinely need selective point lookups → **ElastiCache** only for high-concurrency hot-lookup workloads that justify always-on infra and the VPC tax. In every tier, keep a **reload-on-miss fallback to FOLIO** so a newly-added code resolves on first sight rather than failing the record. + +--- + +## Storage: S3 NDJSON vs. Alternatives + +| Approach | Pros | Cons | Cost/mo (90-day) | +|----------|------|------|------------------| +| **A: DynamoDB** (per-record writes, TTL) | Query-friendly, real-time dashboards possible, strong consistency | O(n) write cost, schema evolution painful, expensive for batches, not auditable | ~$20–50 | +| **B: S3 NDJSON** (chosen) | Batched writes (low cost), queryable (download + `jq`; Athena or S3 Select optional), matches the pipeline's NDJSON-manifest pattern, audit via versioning, easy to compress/archive | Requires S3 Select for queries (not real-time), not ideal for high-cardinality point lookups | ~$1–2 | +| **C: Streaming** (Kinesis/Firehose → Parquet) | High throughput, good for ML pipelines, efficient compression | Overkill for current volume, adds infrastructure complexity, higher cost | ~$5–15 | + +We chose S3 NDJSON (B), and write-scaling is the main driver: per-record DynamoDB writes scale O(n) with record count, whereas the upserter batches its manifest writes into one S3 PUT per run (up to 5,000 records per object). At the current ~1,000 records/day across ~80 syncs that is about one object per run, so the batched model stays flat as volume grows rather than scaling per record. It is also explicitly the existing pipeline pattern rather than a new one: the adapter platform already emits per-job **NDJSON manifests** to S3, so reusing that format (and S3 generally) keeps backup, recovery, query, and downstream tooling consistent across the pipeline. And it stays queryable after the fact without any ETL. **S3 Select** is one convenient option (e.g. `SELECT * FROM s3://bucket/manifest.ndjson WHERE errors > 0`, ~500 ms on a 100 KB file), but note S3 Select is **not currently used elsewhere in the org**, so we don't depend on it: because the manifests are plain NDJSON, a failed-record list is equally available by just downloading the object and piping it through `jq`/`grep`, or by pointing Athena at the prefix if ad-hoc SQL is ever wanted. S3 Select is a nicety, not a load-bearing part of the design; whether to adopt it is an open question (see README). S3 versioning provides an audit trail (every write is a recoverable object version, with the metadata JSON tracking historical jobs), and after 90 days manifests archive cheaply to Glacier, something DynamoDB has no low-cost equivalent for. + +DynamoDB (A) was rejected on fit rather than headline cost (at ~1,000 writes/day either store is cheap), because its per-record write model scales O(n), schema changes require a scan-and-rewrite, and there's no clean way to answer audit questions like "what was deleted last week?". Streaming via Kinesis/Firehose (C) is built for thousands of records per second (overkill for 10–500-record changesets) and adds buffer/flush operational complexity we don't need. + +--- + +## Invocation Pattern: Ordering & Concurrency + +The README specifies the mechanism as behaviour (the source-timestamp stale-write guard in [Step 4: Upsert to FOLIO](README.md#step-4-upsert-to-folio-per-record) and the [Upsert Key Strategy](README.md#upsert-key-strategy-idempotency), plus the Step Functions concurrency limit of 1). This is the reasoning behind it. + +EventBridge publishes changeset events to Step Functions via `StartExecution` (asynchronous, fire-and-forget); the adapter does not block waiting for the sync to complete. Step Functions provides the execution visibility and retry policy, and all invocations (initial or retry) are decoupled from the adapter. + +However, concurrent or retried executions create an **ordering risk**: if two changesets arrive within ~45 s of each other, or if a failed changeset is retried while a newer one is in flight, a naive last-write-wins upsert could apply an older state after a newer one, corrupting the FOLIO record. Similarly, idempotent replays (re-running failed changesets after a fix) must be no-ops if a newer changeset has already updated the same record. + +**Source-timestamp watermarking.** The upsert layer compares each incoming record's Axiell `last_modified` timestamp (from the source data) against a watermark stored on the FOLIO record. The write proceeds only if the incoming timestamp is strictly newer; otherwise the record is skipped, leaving the FOLIO state intact. Where exactly that watermark lives in FOLIO (an administrative note, a custom property, or a version field) is itself an open question; see the README. This ensures: +- Out-of-order concurrent executions apply updates in source-timestamp order, not invocation order. +- Replays (re-running failed changesets) are idempotent: an older changeset re-run after a newer one succeeds does not overwrite the newer state. +- No coupling between adapter and sync (the adapter remains fire-and-forget). + +**Concurrency control.** A Step Functions concurrency limit of 1 (via the state machine definition) is a secondary guard, preventing the GET–PUT gap between a query and an upsert; with a limit of 1, only one execution can be in the `Lambda` state at a time, so even if two changesets arrive within milliseconds, the second waits for the first to commit. This is loose (the adapter cadence is 15 minutes, so concurrency is naturally rare) but provides an extra safeguard against read-skew during retries. + +Combined, these two mechanisms ensure ordering safety without blocking the adapter or introducing a queue. + +--- + +## Error Handling: Per-Record Isolation + +The guiding policy is that one bad record must never halt the batch. + +Per-record errors are non-blocking. A mapping error (Pydantic validation, a missing required field, or an unresolved reference) is captured as `{"source_id": "...", "type": "mapping", "detail": "..."}` and an API error (FOLIO returns 4xx/5xx) as `{"source_id": "...", "type": "api", "detail": "..."}`; in either case the record is written to the S3 error manifest and processing moves on to the next record. Batch-level errors, by contrast, are blocking: an OKAPI auth failure, an Iceberg connection failure, or an S3 write failure raises an exception that the Step Function retries up to `max_retries`, and if those are exhausted the execution ends in the terminal `SyncFailed` state and raises an alert. + +Failures are observable across four channels: CloudWatch Logs (`/aws/lambda/axiell-folio-sync`) for detailed per-record execution, the S3 `.ids.failures.ndjson` manifest for a queryable error list (via a plain download + `jq`, or optionally S3 Select; see the storage note), CloudWatch Metrics (`RecordsFailed`, `RecordsCreated`, and the like) for alerting on a high failure rate, and the Step Function history (`/aws/states/axiell-folio-sync-sfn`) for state transitions and retry events. + +**Skipped records are also accounted for, not dropped silently.** Records skipped before a write (no `980 $a` harvest flag, `Level` gating, or the stale-write guard) are counted in a `RecordsSkipped` metric and recorded in the run's metadata manifest, so "why didn't record X appear in FOLIO?" is answerable after the fact rather than being an invisible no-op. + +**Alert path:** alerting follows the existing pipeline pattern: a **CloudWatch metric alarm** (on `RecordsFailed`/the failure rate, and on `SyncFailed` Step Function executions) fires into the team's **Slack** channel via **Amazon Q Developer in chat applications**. No new notification mechanism is introduced; the alarms are wired to the same SNS-topic → Amazon Q → Slack route already used elsewhere in the catalogue pipeline. + +--- + +## SRS-backed Instances and the Update Path + +A consideration worth making explicit for the update path. Some FOLIO instances are backed by **Source Record Storage (SRS)** as MARC, for example anything migrated or loaded as MARC. For those records the bibliographic fields are controlled by the underlying SRS MARC record and **cannot be updated through the mod-inventory instance API**; only administrative data is editable there. MARC edits go through **quickMARC**, which writes to SRS and syncs the Inventory record. A `PUT` to mod-inventory may therefore fail or be silently ignored for an SRS-backed instance, whereas the current update logic assumes the PUT takes effect. + +**Which storage do we create records in, Inventory-native (FOLIO source) or MARC/SRS?** +Records created the two ways behave differently on update, so a mixed estate is harder to reason about and maintain. This sync currently creates Inventory-native instances (`Instance.source = "FOLIO"`, with no linked SRS record), which keeps the bibliographic fields editable through mod-inventory and keeps the PUT-based update path valid. + +**Catalogue-pipeline impact:** the catalogue pipeline harvests FOLIO over OAI-PMH using the `marc21_withholdings` prefix (see `catalogue_graph/src/adapters/extractors/oai_pmh/folio/config.py`). Under that prefix the instance bib comes from SRS when an SRS record is present, or is generated on the fly from Inventory depending on the **mod-oai-pmh record-source** setting, while holdings and items come from Inventory. So whether the records this sync creates appear in that feed, and in what form, depends on the storage type we choose together with the mod-oai-pmh configuration. This has now been confirmed via the prototype for Inventory-native records: a `source = "FOLIO"` instance created here is updatable through mod-inventory and is received on the FOLIO adapter in the catalogue pipeline. One gap remains under investigation, the item notes not appearing on the OAI-PMH feed; see the README open question. + +--- + +## Cost Analysis + +At ~80 syncs/day the pipeline runs ~2,400 times/month, so the monthly quantities below derive from that: + +| Service | Operation | Qty/mo | Rate | Cost/mo | +|---------|-----------|--------|------|---------| +| Lambda | ~2,400 invocations × ~60s × 512 MB | ~2,400 invocations | $0.0000167/GB-s | ~$1.20 | +| Step Functions | ~2,400 state transitions | ~2,400 transitions | $0.000025/transition | ~$0.06 | +| EventBridge | ~2,400 events | ~2,400 events | $1/M events | ~$0.00 | +| S3 (manifests) | ~2,400 objects written, 90-day retention | ~2,400 objects + storage | $0.005/K PUTs + $0.023/GB/mo | ~$1.50 | +| CloudWatch Logs | ~2,400 × 5 KB ≈ 12 MB/month | 12 MB ingested | $0.50/GB ingested | ~$0.20 | +| **Total** | | | | **~$3–5** | From 44680a535c333d4faeb2abfaa6a8075cab2c1c1e Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Wed, 1 Jul 2026 09:47:16 +0100 Subject: [PATCH 14/15] Move cost analysis in Readme --- rfcs/090-axiell-folio-sync/README.md | 18 +++++++++++++++++- .../090-axiell-folio-sync/design-rationale.md | 19 +------------------ 2 files changed, 18 insertions(+), 19 deletions(-) diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index b553788c..9e654b0f 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -34,6 +34,7 @@ This RFC proposes an automated pipeline to synchronize data from **Axiell Collec - [2. Step Function State Machine](#2-step-function-state-machine) - [3. Lambda Function: axiell-folio-sync](#3-lambda-function-axiell-folio-sync) - [4. S3 Manifest Storage](#4-s3-manifest-storage) +- [Cost Analysis](#cost-analysis) - [Design Decisions](#design-decisions) - [Open Questions](#open-questions) - [Field Mapping from Axc to Folio Instance, Holdings and Items needs to be defined](#field-mapping-from-axc-to-folio-instance-holdings-and-items-needs-to-be-defined) @@ -587,6 +588,21 @@ manifests/ --- +## Cost Analysis + +At ~80 syncs/day the pipeline runs ~2,400 times/month, so the monthly quantities below derive from that: + +| Service | Operation | Qty/mo | Rate | Cost/mo | +|---------|-----------|--------|------|---------| +| Lambda | ~2,400 invocations × ~60s × 512 MB | ~2,400 invocations | $0.0000167/GB-s | ~$1.20 | +| Step Functions | ~2,400 state transitions | ~2,400 transitions | $0.000025/transition | ~$0.06 | +| EventBridge | ~2,400 events | ~2,400 events | $1/M events | ~$0.00 | +| S3 (manifests) | ~2,400 objects written, 90-day retention | ~2,400 objects + storage | $0.005/K PUTs + $0.023/GB/mo | ~$1.50 | +| CloudWatch Logs | ~2,400 × 5 KB ≈ 12 MB/month | 12 MB ingested | $0.50/GB ingested | ~$0.20 | +| **Total** | | | | **~$3–5** | + +--- + ## Design Decisions The decisions below are summarized in [Key Design Considerations](#key-design-considerations); the full trade-off analysis for each (alternatives weighed, why this option won) lives in **[design-rationale.md](design-rationale.md)**. @@ -605,7 +621,7 @@ The **ordering** and **error-handling** mechanisms referenced above are load-bea > **Instance storage type (decision).** This sync creates Inventory-native instances (`Instance.source = "FOLIO"`, no linked SRS record), which keeps the bibliographic fields editable through mod-inventory and keeps the PUT-based update path valid. SRS-backed instances cannot be updated through the mod-inventory API, so a mixed estate is avoided. Confirmed via the prototype: these records are updatable through mod-inventory and are received on the FOLIO adapter in the catalogue pipeline; one gap (item notes not appearing on the OAI-PMH feed) remains open, see [Item notes not visible on the FOLIO OAI-PMH feed](#item-notes-not-visible-on-the-folio-oai-pmh-feed). Full reasoning: [design-rationale.md](design-rationale.md#srs-backed-instances-and-the-update-path). -For the cost evidence behind the "~$3–5/month" figure, see [Cost Analysis](design-rationale.md#cost-analysis). +For the cost evidence behind the "~$3–5/month" figure, see [Cost Analysis](#cost-analysis). --- diff --git a/rfcs/090-axiell-folio-sync/design-rationale.md b/rfcs/090-axiell-folio-sync/design-rationale.md index f418fc98..1fe527f9 100644 --- a/rfcs/090-axiell-folio-sync/design-rationale.md +++ b/rfcs/090-axiell-folio-sync/design-rationale.md @@ -2,8 +2,7 @@ This document records *why* the [RFC 090 CMS→LMS sync](README.md) makes the choices it does. The README states the decisions; this document holds the -trade-off analysis behind them, plus the cost evidence that backs the -"keep it simple" conclusions. +trade-off analysis behind them. ## Table of Contents @@ -14,7 +13,6 @@ trade-off analysis behind them, plus the cost evidence that backs the - [Invocation Pattern: Ordering & Concurrency](#invocation-pattern-ordering--concurrency) - [Error Handling: Per-Record Isolation](#error-handling-per-record-isolation) - [SRS-backed Instances and the Update Path](#srs-backed-instances-and-the-update-path) -- [Cost Analysis](#cost-analysis) --- @@ -117,18 +115,3 @@ A consideration worth making explicit for the update path. Some FOLIO instances Records created the two ways behave differently on update, so a mixed estate is harder to reason about and maintain. This sync currently creates Inventory-native instances (`Instance.source = "FOLIO"`, with no linked SRS record), which keeps the bibliographic fields editable through mod-inventory and keeps the PUT-based update path valid. **Catalogue-pipeline impact:** the catalogue pipeline harvests FOLIO over OAI-PMH using the `marc21_withholdings` prefix (see `catalogue_graph/src/adapters/extractors/oai_pmh/folio/config.py`). Under that prefix the instance bib comes from SRS when an SRS record is present, or is generated on the fly from Inventory depending on the **mod-oai-pmh record-source** setting, while holdings and items come from Inventory. So whether the records this sync creates appear in that feed, and in what form, depends on the storage type we choose together with the mod-oai-pmh configuration. This has now been confirmed via the prototype for Inventory-native records: a `source = "FOLIO"` instance created here is updatable through mod-inventory and is received on the FOLIO adapter in the catalogue pipeline. One gap remains under investigation, the item notes not appearing on the OAI-PMH feed; see the README open question. - ---- - -## Cost Analysis - -At ~80 syncs/day the pipeline runs ~2,400 times/month, so the monthly quantities below derive from that: - -| Service | Operation | Qty/mo | Rate | Cost/mo | -|---------|-----------|--------|------|---------| -| Lambda | ~2,400 invocations × ~60s × 512 MB | ~2,400 invocations | $0.0000167/GB-s | ~$1.20 | -| Step Functions | ~2,400 state transitions | ~2,400 transitions | $0.000025/transition | ~$0.06 | -| EventBridge | ~2,400 events | ~2,400 events | $1/M events | ~$0.00 | -| S3 (manifests) | ~2,400 objects written, 90-day retention | ~2,400 objects + storage | $0.005/K PUTs + $0.023/GB/mo | ~$1.50 | -| CloudWatch Logs | ~2,400 × 5 KB ≈ 12 MB/month | 12 MB ingested | $0.50/GB ingested | ~$0.20 | -| **Total** | | | | **~$3–5** | From 61433508318e71c9c3f291dcebd8a410eb44b36b Mon Sep 17 00:00:00 2001 From: anubhavijay Date: Wed, 1 Jul 2026 10:26:18 +0100 Subject: [PATCH 15/15] Removed the design rationale table --- rfcs/090-axiell-folio-sync/README.md | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/rfcs/090-axiell-folio-sync/README.md b/rfcs/090-axiell-folio-sync/README.md index 9e654b0f..cca5c499 100644 --- a/rfcs/090-axiell-folio-sync/README.md +++ b/rfcs/090-axiell-folio-sync/README.md @@ -605,19 +605,9 @@ At ~80 syncs/day the pipeline runs ~2,400 times/month, so the monthly quantities ## Design Decisions -The decisions below are summarized in [Key Design Considerations](#key-design-considerations); the full trade-off analysis for each (alternatives weighed, why this option won) lives in **[design-rationale.md](design-rationale.md)**. - -| Decision | Choice | Rationale | -|----------|--------|-----------| -| **Mapping** | Typed Pydantic models (`mapping.py`), not a YAML mapper | [Mapping](design-rationale.md#mapping-pydantic-models-vs-yaml-mapper) | -| **Orchestration** | Step Functions + Lambda, not direct Lambda or SQS | [Orchestration](design-rationale.md#orchestration-step-functions-vs-alternatives) | -| **Reference caching** | Reload-per-run; defined scale-up progression if it grows | [Caching](design-rationale.md#caching-strategy-reference-data-scaling-options) | -| **Manifest storage** | S3 NDJSON (90-day TTL), not DynamoDB or streaming | [Storage](design-rationale.md#storage-s3-ndjson-vs-alternatives) | -| **Ordering safety** | Source-timestamp stale-write guard + Step Functions concurrency = 1 | [Invocation Pattern](design-rationale.md#invocation-pattern-ordering--concurrency) | -| **Error handling** | Per-record isolation; batch-level errors retry then fail | [Error Handling](design-rationale.md#error-handling-per-record-isolation) | -| **Instance storage type** | Inventory-native (`Instance.source = "FOLIO"`), no linked SRS record | [SRS-backed Instances](design-rationale.md#srs-backed-instances-and-the-update-path) | - -The **ordering** and **error-handling** mechanisms referenced above are load-bearing for correctness, so they are stated as behaviour in [Step 4: Upsert to FOLIO](#step-4-upsert-to-folio-per-record) and the [Upsert Key Strategy](#upsert-key-strategy-idempotency); design-rationale.md explains *why* they take the form they do. +The key choices are summarized in [Key Design Considerations](#key-design-considerations); the full trade-off analysis for each (alternatives weighed, why this option won) lives in **[design-rationale.md](design-rationale.md)** — covering mapping (Pydantic vs YAML), orchestration (Step Functions), reference caching, manifest storage (S3 NDJSON), ordering safety, error handling, and instance storage type. + +The **ordering** and **error-handling** mechanisms are load-bearing for correctness, so they are stated as behaviour in [Step 4: Upsert to FOLIO](#step-4-upsert-to-folio-per-record) and the [Upsert Key Strategy](#upsert-key-strategy-idempotency); design-rationale.md explains *why* they take the form they do. > **Instance storage type (decision).** This sync creates Inventory-native instances (`Instance.source = "FOLIO"`, no linked SRS record), which keeps the bibliographic fields editable through mod-inventory and keeps the PUT-based update path valid. SRS-backed instances cannot be updated through the mod-inventory API, so a mixed estate is avoided. Confirmed via the prototype: these records are updatable through mod-inventory and are received on the FOLIO adapter in the catalogue pipeline; one gap (item notes not appearing on the OAI-PMH feed) remains open, see [Item notes not visible on the FOLIO OAI-PMH feed](#item-notes-not-visible-on-the-folio-oai-pmh-feed). Full reasoning: [design-rationale.md](design-rationale.md#srs-backed-instances-and-the-update-path).