Keep external OCaml JSON inputs open and owned contracts closed

docs/agent-guide/implemented/bugfix/2026-10-14-external-input-open-json.md

@@ -0,0 +1,49 @@
+# Open external OCaml JSON inputs and keep owned contracts closed
+
+## Problem
+
+A real Journal sync error rejects pull/ok.txs[0].tx-id before transactions are applied. Both the generated server DTO and the structural whitelist reject external response extensions. External services can add fields Journal does not consume without changing required fields.
+
+## Decision
+
+Limit this PR to OCaml JSON codecs, as explicitly requested by the user. Keep PPX. External WS and E2EE HTTP records use separate input-only open readers, including nested records; declared required fields, types, nullability, supported variants, semantics and size budgets remain validated. Unused pull tx-id is ignored. Owned client/output DTOs remain closed, and existing internal Config/Protocol/full persistence admission stays closed. Catalog and snapshot metadata retain open input behavior with decode-only derivations. Outbox metadata readers remain decode-only projections rather than full-record admission.
+
+All Swift codec behavior and its compiler/test/CI configuration are restored exactly to the PR's merge-base. The generic Swift decoder, native call-site replacements, native boundary tests and offline provider compile mode are outside the narrowed scope.
+
+The user additionally approved the smallest test-only correction for the existing CI failure in Application's retained root feedback checks. The fixture's global title lookup can select Detail's same-title node instead of the root node. Add an optional identity predicate to that test helper and restrict root-content observations to the root navigation owner; retain the original layout and node identity assertions. This does not change production UI behavior.
+
+## Alternatives considered
+
+### Declare only tx-id
+
+Would still reject the next external extension. The user requested open external input schemas.
+
+### Make shared input/output DTOs open
+
+Would spread external tolerance into owned contracts. Keep separate input-only readers and closed output DTOs.
+
+### Add strict Swift receiving infrastructure
+
+Not part of the narrowed OCaml scope. Generated Encodable/PPX writers already emit their declared fields; strict receiving is a separate defensive policy. Restore Swift and dependent wiring to their prior behavior.
+
+## Consequences
+
+Open object validation checks declared fields and duplicate safety without applying the owned-field whitelist. Output uses the original generated closed DTOs and readers with semantic/budget validation. Internal full-record codecs stay closed; projections do not expose encoders. All Dune/spec/Swift/CI files have no net changes relative to the merge-base. No UI changes, real account/graph writes, new diagnostics, merge or phone installation belong to this PR.
+
+## Validation and coverage
+
+Tests were written first. Against unchanged main953, public WS extension/declared-control tests failed, and public Runner/local TLS HTTP tests failed four extension-acceptance assertions; required/type/null/duplicate controls passed. Core metadata and internal Config controls passed at baseline. These are synthetic regression payloads, not live service compatibility claims.
+
+| Boundary | Actual evidence | Public regression boundary |
+| --- | --- | --- |
+| WS server/client | Materialized and viewed screenshot establishes pull/ok.txs[0].tx-id rejection; payload and deployed SHA unknown | Sync_protocol: all supported variants and rejection reasons, root/nested extras, ignored tx-id shapes, required/type/null/semantic/variant failures, closed client/output and limits |
+| Catalog/E2EE HTTP | Prior authorized type-only App logs show catalog/user-key HTTP 200 and graph-key behavior before/after PR72; no fresh signed-in payload collection | Runner with independent localhost TLS peer and generated readers |
+| Snapshot metadata/baseline | Upstream schema, no live signed-in response | Core raw completions preserve required/type/null/HTTPS URL checks |
+| Internal/outbound | No real account record accessed | Config rejects root/nested extras; existing Protocol and persistence suites retain closed full admission; generated client/server output round trips |
+| Transit/snapshot/assets | No new live graph/asset samples | Existing envelope/size/transaction/snapshot/asset regressions; formats unchanged |
+
+Frozen upstream dev 949a4f0e72cddb2dad34091393bfa09592171944 declares optional non-null UUID tx-id in server entries; old 015cc9f does not declare that server field. Both use open Malli maps. This source evidence does not establish the screenshot value/type or deployment revision: [frozen schema](https://github.com/logseq/logseq/blob/949a4f0e72cddb2dad34091393bfa09592171944/deps/db-sync/src/logseq/db_sync/malli_schema.cljs#L10). Journal does not consume tx-id.
+
+After scope removal, rerun the OCaml build/native embed and regression suite, review the exact new head in the existing independent Codex session using journal-code-review, and track its CI to success. Prior Swift-inclusive review/CI snapshots are superseded. spec-dev-tool check --all has three preexisting invalid historical decisions unrelated to this change.
+
+The unchanged root-feedback assertion was reproduced on both 953d4a7 and 9293bad by randomizing only the test observer table's hash order. The root body has one stack candidate; both snapshots preserve root content node 180, while the global title lookup may select Detail node 259. The candidate correction passes the two fixed public Application cases in five randomized runs per snapshot (20 case executions). All original containment and node identity comparisons remain. Diagnostic mirrors and logs stay outside the repository.

logseq_db_storage/lib/sync_outbox_store.ml

@@ -33,7 +33,9 @@ let schema db =
        | _ -> Error "unsupported sync outbox storage format")
 ;;
 
-type mutation_identity = { mutation_id : string } [@@deriving yojson { strict = false }]
+(* These decode-only projections extract indexing metadata. The full persisted
+   record is admitted by the closed domain codec at its owner. *)
+type mutation_identity = { mutation_id : string } [@@deriving of_yojson { strict = false }]
 
 type mutation_identity_wire =
   | Save_block of mutation_identity
@@ -44,15 +46,15 @@ type mutation_identity_wire =
   | Create_journal_page of mutation_identity
   | Set_task_status of mutation_identity
   | Clear_task_status of mutation_identity
-[@@deriving yojson]
+[@@deriving of_yojson]
 
 type payload_header =
   { format_version : int
   ; mutation : mutation_identity_wire
   ; sequence : int
   ; sync_revision : int
   }
-[@@deriving yojson { strict = false }]
+[@@deriving of_yojson { strict = false }]
 
 let payload_identity record =
   match

logseq_db_worker/test/test_protocol.ml

@@ -1,6 +1,56 @@
 module T = Logseq_db_worker_test_support.Test_support
 module P = Logseq_db_worker.Protocol
 
+let closed_internal_config_fields () =
+  let base =
+    `Assoc
+      [ "application_support_directory", `String "/tmp/synthetic-support"
+      ; ( "target"
+        , `List
+            [ `String "Managed_sync"
+            ; `Assoc [ "base_url", `String "https://example.invalid" ]
+            ] )
+      ; "compatibility_profile", `List [ `String "Logseq_65_33_or_newer" ]
+      ; "response_budget_bytes", `Int P.maximum_response_bytes
+      ; "default_page_size", `Int P.default_page_size
+      ]
+  in
+  T.require
+    (Result.is_ok (Logseq_db_worker.Config.of_yojson base))
+    "valid owned Config control was rejected";
+  let outer =
+    match base with
+    | `Assoc fields -> `Assoc (("future", `Bool true) :: fields)
+    | _ -> assert false
+  in
+  let nested =
+    match base with
+    | `Assoc fields ->
+      `Assoc
+        (List.map
+           (fun (key, value) ->
+              if key = "target"
+              then
+                ( key
+                , `List
+                    [ `String "Managed_sync"
+                    ; `Assoc
+                        [ "base_url", `String "https://example.invalid"
+                        ; "future", `Bool true
+                        ]
+                    ] )
+              else key, value)
+           fields)
+    | _ -> assert false
+  in
+  List.iter
+    (fun json ->
+       T.require
+         (Result.is_error (Logseq_db_worker.Config.of_yojson json))
+         "owned internal Config accepted an extra field")
+    [ outer; nested ]
+;;
+
 let page_tree_contract_rejects_retired_scope () =
   let raw =
     Yojson.Safe.from_string
@@ -934,7 +984,8 @@ let request_tree_boundaries () =
 let () =
   T.run
     "protocol"
-    [ T.case "tree depth node and root boundaries" request_tree_boundaries
+    [ T.case "closed internal Config root and nested fields" closed_internal_config_fields
+    ; T.case "tree depth node and root boundaries" request_tree_boundaries
     ; T.case "config golden and validation" config_generated_golden_and_validation
     ; T.case "generated golden and legacy rejection" generated_golden_and_legacy_rejection
     ; T.case "request domain constraints" request_domain_constraints

logseq_sync/lib/effect_runner/protocol/catalog.ml

@@ -44,17 +44,17 @@ let schema = function
 ;;
 
 type readiness = { ready : bool option [@key "graph-ready-for-use?"] [@default None] }
-[@@deriving yojson { strict = false }]
+[@@deriving of_yojson { strict = false }]
 
 type graph_dto =
   { graph_id : Graph_types.Uuid.t [@key "graph-id"]
   ; graph_name : string [@key "graph-name"]
   ; schema_version : string option [@key "schema-version"] [@default None]
   ; encrypted : bool option [@key "graph-e2ee?"] [@default None]
   }
-[@@deriving yojson { strict = false }]
+[@@deriving of_yojson { strict = false }]
 
-type response = { graphs : Yojson.Safe.t list } [@@deriving yojson { strict = false }]
+type response = { graphs : Yojson.Safe.t list } [@@deriving of_yojson { strict = false }]
 
 let decode_graph json =
   bind (readiness_of_yojson json) (fun ready ->

logseq_sync/lib/effect_runner/protocol/e2ee.ml

@@ -40,23 +40,26 @@ let bounded_string field value =
 type graph_key_response_dto =
   { encrypted_aes_key : string [@key "encrypted-aes-key"]
   ; encrypted_aes_key_base64 : string option
-      [@key "encrypted-aes-key-base64"] [@default None]
+        [@key "encrypted-aes-key-base64"] [@default None]
   }
-[@@deriving yojson]
+[@@deriving of_yojson { strict = false }]
 
 type user_keys_response_dto =
   { public_key : string [@key "public-key"]
   ; encrypted_private_key : string [@key "encrypted-private-key"]
   }
-[@@deriving yojson]
+[@@deriving of_yojson { strict = false }]
 
 let graph_key_response source =
   try
     bind
       (Logseq_db_types.Json_codec.decode
          graph_key_response_dto_of_yojson
          (Yojson.Safe.from_string source))
-      (fun dto -> bounded_string "encrypted-aes-key" dto.encrypted_aes_key)
+      (fun dto ->
+         (* The optional compatibility field is validated by PPX but not consumed. *)
+         let _ = dto.encrypted_aes_key_base64 in
+         bounded_string "encrypted-aes-key" dto.encrypted_aes_key)
   with
   | Yojson.Json_error _ -> Error "E2EE response must be valid JSON"
 ;;

logseq_sync/lib/pure_reducer/core.ml

@@ -1019,7 +1019,7 @@ type snapshot_uri_response =
   { ok : bool
   ; url : string
   }
-[@@deriving yojson { strict = false }]
+[@@deriving of_yojson { strict = false }]
 
 let decode_snapshot_uri source =
   try