fix: harden sync error handling, unwrap the gateway envelope, and report RLS denials

[marcobambini]

Release 1.1.4 hardens sync error handling and receive checkpoints, preserves error origin, fixes gateway response parsing, and bounds network and fragment-maintenance work.

Apply errors and receive checkpoints

An apply now stops at the first failed write, including constraints, raising triggers, type errors, RLS denials, missing privileges and transient failures. Later changes are not processed and the receive checkpoint stays put. PostgreSQL preserves the database SQLSTATE and rolls back the failing SQL statement. SQLite may retain earlier successful groups, which re-merge on redelivery once the cause is fixed. Network results retain the applied prefix in receive.rows and report receive.error.

There is no internal denial retry loop and no receive.denied or receive.failed counter. A permanent denial remains an error until its cause is resolved. This applies to ordinary columns, block/GOS paths and fragmented values.

Pending ordinary columns and their metadata share a group savepoint; rejected flushes roll back sentinel/reset/winner clocks. Block materialization flushes ordinary columns first, so this is not a promise of whole-payload or whole-row atomicity on SQLite.

Receive streams hold one durable checkpoint until the final watermark, reject incomplete fragmented values and restart paging after apply failure. Direct v3 fragment calls do not advance the receive checkpoint, and a fragmented final chunk without a watermark fails explicitly.

Fragment concurrency and cleanup

Staging, reconstruction, application and removal of each completed value happen under a savepoint. READ COMMITTED fragment applies serialize per value; SERIALIZABLE can require a retry; REPEATABLE READ fragment applies are refused.

Cleanup only removes incomplete groups whose newest piece is over 24 hours old, and skips values being applied by another transaction. Each PostgreSQL cleanup now materializes at most 64 candidate groups before acquiring advisory locks. The previous query could acquire a lock for every stale group, exhaust shared lock memory and roll back all maintenance. A 30,000-group backlog reproduced this indefinitely. Subsequent bounded passes now make progress; the existing once-per-minute, per-connection throttle remains.

Test 61 covers that backlog, a concurrently locked group, bounded lock retention, progress on the next pass and completion of a genuine fragmented value. It fails on the previous query and passes with the fix (9d0abb3).

Other changes

• Gateway success envelopes are explicitly unwrapped before scoped lookups for URLs, sync state and failures.
• PostgreSQL owns SPI tuple tables per statement and restores savepoint resource owners/memory contexts; errors retain SQLSTATE.
• Block failures report their stage, table, column and underlying cause. Existing block/GOS rows use UPDATE with upsert fallback when no row changes.
• PK doubles preserve deployed little-endian IEEE754 bytes on every architecture; clocks above UINT32_MAX are retained.
• Decompression is bounded by LZ4's maximum expansion ratio and INT_MAX, not a fixed 256 MiB cap.
• Curl uses asynchronous DNS, API deadlines and progress-based artifact deadlines. Settings can tune deadlines at runtime, and SQLite 3.41+ can cancel an in-flight call with sqlite3_interrupt.
• Node rejects ia32. make unittest-s390x replaces the ineffective forced-endian test.

Verification

Local verification of the cleanup fix:

• PostgreSQL 15.19 / 17.11 / 18.6: 521 checks per version, zero failures, with ON_ERROR_STOP enabled and expected errors handled by the suite.
• Negative control: test 61 fails with out of shared memory and no cleanup progress on the previous query; all five assertions pass with the bounded query.
• Migration check: 1.1.3 → 1.1.4, unchanged SQL surface, no migration required.
• Git whitespace check passed.

Independent review of preceding commit 1820a15 also passed 150 SQLite checks plus focused regressions, extension-core ASan/UBSan builds, 30,000 randomized writes across three replicas (10,800 payload deliveries and 600 convergence comparisons), 100 randomized fragment/retry trials, 14 Linux network groups, 340 fractional-indexing tests, and 14 Node tests/typecheck/build.

Real-cloud verification and limits

The repository already has the seven integration secrets. Linux CI on 1820a15 reports OK, not SKIPPED, for Init+Sync, Token Auth, all seven chunked cases, Offline Error and Failure Path. That overall workflow failed in Android x86_64; do not interpret it as fully green.

The new push starts CI for the cleanup-fix commit. Building the client does not deploy the PostgreSQL extension to remote nodes: validating the cleanup fix on a real server requires a dedicated staging node running this build. Cloud test guide documents the existing CI route, tenant fixtures and server-side staging procedure. Audit details describe the current contract and coverage.

Known pre-existing limits from the independent review: SQLite deferred-constraint commit failures can leave a transaction open; very deep PostgreSQL savepoints exceed the fixed owner-storage depth; allocation-fault cleanup still has gaps. The local review did not run Windows/Android/WASM/iOS or s390x and did not complete PostgreSQL ASan runtime testing. These are not claimed as validated by the local test results above.

[marcobambini]

Review follow-ups

Issues found while reviewing this PR, all fixed in the 7 commits just pushed. Each commit builds and passes SQLite unit/regression, network and PostgreSQL 17 suites (502 PASS, 0 FAIL); make unittest-s390x also passes on big-endian.

Apply path — 9cc60d2

• Sticky errors stalled sync forever: a row failing on its data (constraint, raising trigger) blocked the cursor with no escape; on PG the whole payload was rejected. Such writes are now skipped, logged and reported (receive.failed / failedError); transient and configuration failures (lock, deadlock, serialization, cancel, OOM, I/O, missing privilege, read-only) still fail the apply and keep the cursor.
• receive.denied was always 0: denials only happen on PG, which has no receive drain. Field removed; PG emits one summary WARNING.
• RLS denials outside the batched path (block columns, GOS tables) failed the whole apply. Each row now runs in its own savepoint and the trigger re-raises denials as 42501.
• Order-dependent denials lost data (e.g. a membership row later in the payload). Denied rows are retried once the rest of the payload is applied.
• Block columns and GOS tables never wrote under RLS / NOT NULL: the per-column upsert proposed a partial row. Existing rows are now updated in place (falling back to the upsert when no row changes), and pending columns are flushed before blocks.
• A failed flush left partial metadata (sentinel, zeroed clocks), so a resurrected row was silently never created on re-delivery; skipped counts also missed the sentinel. Each PK group now runs under one savepoint and counts payload rows.
• PostgreSQL crashes/errors from savepoints (pre-existing): SELECT cloudsync_payload_apply(data) FROM t failed with a foreign buffer pin, and nested savepoints could trip EnsurePortalSnapshotExists. Resource owner and memory context are restored; only the outermost savepoint swaps the snapshot.
• Error origin lost: trigger and apply errors surfaced as XX000 (breaking 40001/40P01 retry logic), SQLite triggers reported SQLITE_ERROR without context, and some block failures were blank. SQLSTATE and result codes are preserved, and messages name stage, column and table.

Other fixes

• af8e5b0 — block migration failed on metadata whose base row is gone, making set_column(..., 'block') impossible; those rows are skipped again.
• 268011b — the fixed 256 MiB cap rejected payloads the library itself produces; the declared size is now bounded by LZ4's 255:1 ratio, and INT_MAX is still enforced.
• 1ade3e3 — new make unittest-s390x, running the SQLite suites on a real big-endian host (QEMU); it catches the pre-1.1.4 double encoding.
• c4a2b07 — removed code left unused by the block refactor.
• 31c5d06 — removed make endian-unittest, which compiled to an identical object and could not fail, plus the unused host-order helpers.
• 3e8d95d — test temp-directory cleanup no longer depends on file names and reports what is left.
• Submodule: ddaf414 was only on a side branch of fractional-indexing; it has been fast-forwarded to its main (same SHA) and the branch deleted.

New/updated tests: review_regressions.c, network_unit.c, PostgreSQL 39, 57 and new 59. Each fix was checked against a negative control.

🤖 Generated with Claude Code

[andinux]

New commits: fail fast on apply errors, one durable receive window per stream

These five commits change how failures in received changes are handled:

5c980a1 fix(apply): stop at the first failed write instead of skipping it
05d42de fix(receive): keep one checkpoint per stream and check its fragments
684aae3 fix(fragments): keep resumed groups and make each fragment call one unit
ce0557f fix(postgres): serialize concurrent applies of one fragmented value
8ebc951 fix(receive): fail a fragmented stream whose final chunk has no watermark

Why

Skipping errors can permanently acknowledge missing data. Until now, a write that failed on its data (a constraint, a trigger, a type error) or was denied by an RLS policy was skipped, and the receive checkpoint still moved past it. Once the checkpoint has passed a change, nothing delivers that change again. Fixing the data or the policy afterwards doesn't bring it back, so the row is lost for good, reported only as a warning. Retrying denied rows later in the same payload makes this rarer but doesn't remove it.

These commits deliberately choose explicit failure and retry instead:

• cloudsync_payload_apply stops at the first error.
  • It returns the original error, with its SQLSTATE on PostgreSQL.
  • The failed primary-key group is rolled back, data and sync metadata alike, and no later change in the payload is applied.
  • On SQLite, the changes applied before the failure are kept where the enclosing transaction allows it. On PostgreSQL the failed statement rolls back as usual.
  • Once the cause is fixed (data, policy or grant), delivering the same payload again applies it. The changes already applied merge again as no-ops.
  • A policy that depends on rows later in the same payload is not reordered or retried inside the apply: the apply fails until those rows are in.
• The receive response reports the successful prefix. receive.rows and receive.tables count the changes applied before the failure, and receive.error gives the first error with receive.complete: false. receive.failed / receive.failedError are removed.
• Updates that write nothing are not recorded as applied. A batched multi-column update that changes no row (the row is gone, or hidden by an UPDATE policy) now falls back to the upsert. Before, it recorded the change as applied without storing it.

One durable receive window until final success. check_dbversion/check_seq also identify the server's prepared pages. So they stay fixed for the whole chunk stream: across non-final chunks, calls capped by max_chunks, and failures. They move once, to the final watermark, only after the final chunk has applied. Moving them between pages would reset the server's paging and replay the backlog. Specifically:

• A failed chunk makes the next call replay the window from page 0. Changes already applied re-merge as no-ops.
• A checkpoint write that fails is reported instead of ignored.
• A fragmented value that the stream delivered must be complete before the stream checkpoints; otherwise the call fails.
  • Only values delivered by the current stream are tracked, so pieces left behind by other streams, interrupted attempts or direct SQL calls never block progress.
  • Pieces of a value the stream already applied are ignored, so a replay doesn't leave them behind again.
• A direct SQL call applying a fragment never moves the receive checkpoint.
• A stream from a server that sends no watermark and ends in a fragment now fails loudly instead of replaying the same window forever. The current server always sends a watermark, so this is purely defensive.

Also included

• Fragment staging.
  • The age cleanup removes a group only when all its pieces are old, and it runs after the new piece is staged. Before, it could delete the old pieces of a value whose delivery had just resumed.
  • A failing cleanup no longer fails the apply.
  • Applying a completed value and removing its pieces now succeed or fail together.
• PostgreSQL concurrency. The server applies each uploaded chunk as its own job, so pieces of one value can be applied by concurrent transactions. Under READ COMMITTED, two transactions could each see only their own piece, both succeed, and the value was never applied (reproduced by the new 60_fragment_concurrency.sql before the fix).
  • A per-value advisory lock serializes those transactions.
  • SERIALIZABLE relies on PostgreSQL's serialization failure instead, which the client can retry.
  • REPEATABLE READ is refused for fragments.

No SQL signatures, payload formats or table definitions change (check-postgres-migration.sh: SQL surface identical). API.md, the RLS reference and the unreleased 1.1.4 CHANGELOG entries describe the new behaviour.

Tested: SQLite unit and review regressions, network unit tests (including new receive-stream scenarios run against a test /check responder), PostgreSQL 17 (516 PASS) and Supabase (514 PASS).

🤖 Generated with Claude Code

[andinux]

Network deadlines: how the libcurl timeouts work now

Three more commits complete the request deadlines that this PR introduces:

4e2d36e fix(network): resolve names on a thread so DNS lookups honor the deadlines
443eae7 feat(network): cancel a transfer in flight with sqlite3_interrupt()
1820a15 feat(network): tune the deadlines at runtime with cloudsync_set

Two policies, by kind of request

	API requests (check, upload URL, apply, status)	Payload transfers (upload / download of a payload)
Connect, name resolution included	30 s	30 s
Bound	whole request: 300 s	lack of progress: aborted when slower than 1024 B/s for 60 s, with a 3600 s absolute limit as a backstop

API calls carry small JSON, so an elapsed-time cap fits them. Payload transfers are bulk: an elapsed-time cap would kill a healthy large transfer on a slow link, so they are bounded by lack of progress instead.

DNS is bounded too (4e2d36e)

libcurl was built with --disable-threaded-resolver. Its synchronous resolver can only time out a lookup with SIGALRM, and CURLOPT_NOSIGNAL turns that off, so a hung DNS server blocked a request past every deadline. libcurl is now built with --enable-threaded-resolver (c-ares stays disabled). --disable-pthreads was dropped: curl 8.12 doesn't recognize it, so it did nothing. A unit test asserts that the linked libcurl resolves names asynchronously (CURL_VERSION_ASYNCHDNS).

Cancellation (443eae7)

• sqlite3_interrupt() on the connection stops a network call in flight. libcurl checks the connection on its progress callback, about once a second even while idle, and aborts the transfer.
• The call then fails with SQLITE_INTERRUPT instead of a generic error, so the caller can tell a deliberate stop from a failure worth retrying.
• The check needs SQLite 3.41 or later (sqlite3_is_interrupted). With an older host library a call still ends on its deadline.

Runtime tuning with cloudsync_set (1820a15)

The compiled defaults (the CLOUDSYNC_*_TIMEOUT_SECONDS / CLOUDSYNC_ARTIFACT_LOW_SPEED_* macros) can now be overridden per database. These are cloudsync_set keys stored in cloudsync_settings, like payload_max_chunk_size; no SQL function or signature changes.

Key	Unit	Default	Max
network_connect_timeout	seconds	30	86400
network_request_timeout	seconds	300	86400
network_artifact_timeout	seconds	3600	86400
network_artifact_low_speed_limit	bytes/s	1024	1073741824
network_artifact_low_speed_time	seconds	60	86400

• When it applies: values are read for every request, so a change applies to the next request.
• Invalid or missing values: a missing, zero or negative value keeps the default. A value can raise or lower the default, and anything above the maximum is clamped.
• Connect vs total: the connect timeout never exceeds the request's total.
• Removing an override: settings persist in the database; cloudsync_set(key, NULL) removes one.
• Call it after init: cloudsync_set is only stored once cloudsync_init / cloudsync_network_init has run. Before that it is silently ignored.
• Hosts that configure through environment variables: read them at startup and call cloudsync_set after init, with NULL when a variable is unset so a stale value doesn't linger. Values are whole seconds, so round milliseconds up.

Tests

• Unit tests against a local server that never answers check that:
  • the API elapsed cap and the payload stall limit each abort, on pooled and non-pooled handles;
  • a transfer on an interrupted connection aborts at once;
  • a raised network_request_timeout takes effect;
  • a zero value keeps the default;
  • sqlite3_interrupt() from another thread ends a call with a 30 s deadline well before it, with SQLITE_INTERRUPT.
• SQLite unit, review and network suites pass locally. check-postgres-migration.sh reports the SQL surface unchanged.

Not covered

The Mac Catalyst build uses NSURLSession (network.m) instead of libcurl. That path still runs on the system default timeouts (60 s idle, 7 days per resource) and has no cancellation; it's left for a follow-up.

🤖 Generated with Claude Code

[marcobambini]

Ho completato la revisione approfondita e pubblicato la correzione nel commit 9d0abb306e44d9a92dbe2a12c735b93d1bf0ee41.

Regressione trovata

La pulizia PostgreSQL dei frammenti scaduti acquisiva un advisory lock transazionale per ogni gruppo candidato, senza limitarne il numero. Con 30.000 gruppi incompleti vecchi di due giorni, la query esauriva la memoria condivisa dei lock:

WARNING: cloudsync: stale fragment cleanup failed: out of shared memory
stale_before = 30000
stale_after  = 30000

L'errore annullava tutta la pulizia. Una nuova sessione riproduceva lo stesso risultato, impedendo al backlog di ridursi. La query precedente su main, sugli stessi dati, eliminava invece tutti i 30.000 gruppi. Il payload di prova poteva comunque completarsi: il difetto osservato riguardava la mancata pulizia, non la perdita di quel valore.

Correzione

La query ora materializza al massimo 64 gruppi candidati prima di acquisire gli advisory lock. Ogni passaggio mantiene quindi limitato il lavoro sui lock; le chiamate successive proseguono sul backlog. Restano invariati la protezione dei gruppi usati da altre transazioni, il criterio di scadenza basato sul frammento più recente e il throttling di un minuto per connessione.

Non viene eseguito un ciclo di batch nella stessa transazione: rilasciare un savepoint non libera i lock transazionali già acquisiti. Transazioni applicative molto lunghe possono comunque accumulare lock da più apply.

Il nuovo 61_fragment_cleanup_backlog.sql verifica:

• progresso su 30.000 gruppi scaduti, con un candidato bloccato da un'altra transazione;
• numero limitato di lock trattenuti;
• conservazione del gruppo bloccato e dei frammenti appena ricevuti;
• ulteriore progresso nella chiamata successiva, dopo il rilascio del lock;
• completamento di un vero valore suddiviso in due frammenti.

Controllo negativo: con la query precedente il test riproduce out of shared memory e fallisce le verifiche di progresso; con la correzione passano tutte e cinque le verifiche.

Validazione

Dopo la correzione, suite completa con ON_ERROR_STOP=1:

PostgreSQL	Risultato
15.19	521 controlli superati, zero fallimenti
17.11	521 controlli superati, zero fallimenti
18.6	521 controlli superati, zero fallimenti

Superati anche il controllo whitespace e la verifica migrazione: 1.1.3 → 1.1.4, superficie SQL invariata, nessuna migrazione richiesta.

La revisione del commit precedente 1820a15 aveva inoltre superato gli stress test indipendenti: 30.000 scritture casuali fra tre repliche SQLite, 10.800 applicazioni di payload e 600 confronti senza divergenze, più 100 prove sui frammenti con ordine casuale, duplicati, errore forzato, rollback e retry. Questi risultati sono distinti dalla verifica della nuova correzione.

Documentazione e cloud reale

Ho aggiornato la descrizione della PR, docs/internal/audit-regressions.md e il changelog. Le descrizioni precedenti parlavano ancora di skip dei dinieghi RLS e contatori rimossi: il codice attuale interrompe l'apply al primo errore e mantiene fermo il checkpoint.

La nuova guida docs/internal/cloud-e2e.md documenta il percorso di test cloud già disponibile. Nei log Linux del commit 1820a15 risultano effettivamente OK, non SKIPPED, autenticazione, tutti i sette casi chunked e i casi offline/failure. Il push della correzione ha avviato una nuova CI; questo commento non ne dichiara l'esito finale.

La correzione del garbage collector non è ancora stata verificata su un server cloud aggiornato: la CI del client non installa l'estensione PostgreSQL sui nodi remoti. Per quella verifica serve un nodo staging con questa build e il test del backlog attivato da un upload reale.

I limiti preesistenti emersi durante la revisione — transazione SQLite lasciata aperta dopo alcuni commit falliti, savepoint PostgreSQL molto profondi e cleanup incompleto in alcuni percorsi OOM — sono documentati separatamente e non sono presentati come risolti da questo commit.