feat: getLogs since resume hint - #367
Open
andrevmatos wants to merge 34 commits into
Open
Conversation
Lets a getLogs caller resume from the last emitted log instead of a fresh startBlock/startTime: since.blockNumber/blockTimestamp merge as raising start floors via withSinceStart, so the hint alone satisfies the start requirement; cursor-capable chains additionally resume exclusively past the hinted log. getMessageById and getExecutionReceipts accept it too.
Queries floor at the hint's block (blocks are always fetched whole), but within that block logs with index <= since.index are not re-emitted, so a partial hint is honored exactly while same-block followers still stream. Foreign-address hints are ignored wholesale.
The hint signature seeds the native `until` cursor (exclusive); its blockNumber is also an absolute slot floor, since `until` only has an effect when the queried node actually knows the signature. Watch polls now keep a resume cursor even when the initial scan collects nothing, instead of re-emitting the unbounded head page.
Single-handle streams resume exactly from the hint's event sequence number; multi-handle streams floor at blockNumber + 1. The +1 is safe because versions are now emitted atomically: the round ceiling is exclusive and a handle counts as caught up only past the tip seq — previously >= dropped the event exactly at a full batch boundary.
The composite transactionHash carries an exact per-account lt cursor; the resume floor is exclusive (sinceLt + 1) and only ever raises the requested start. When the hint's block covers startBlock, the shard-header floor lookup is skipped entirely. Hints from a foreign account — by address field or by the account embedded in the hash — are ignored, as are malformed ones.
The since resume hint now bounds rescans directly, making the per-address cursor-page cache redundant. Long-lived watch streams retained the account's parsed transaction history in memory (production memory hazard); the walk also releases its initial batch once yielded.
andrevmatos
requested review from
a team,
PabloMansanet and
aelmanaa
as code owners
August 21, 2026 16:18
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
aelmanaa
previously approved these changes
Aug 21, 2026
There was a problem hiding this comment.
Pull request overview
Adds cross-chain getLogs resume hints to avoid rescanning previously processed logs.
Changes:
- Adds shared
sincefloor handling and chain-specific exclusive cursors. - Improves Aptos batch atomicity and boundary handling.
- Removes TON’s transaction cache and adds resume-focused tests.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
CHANGELOG.md |
Documents resume behavior and fixes. |
ccip-sdk/src/types.ts |
Makes generic log fields writable. |
ccip-sdk/src/ton/txcache.test.ts |
Removes obsolete cache tests. |
ccip-sdk/src/ton/logs.ts |
Releases retained transaction arrays. |
ccip-sdk/src/ton/index.ts |
Removes caching and adds LT resume logic. |
ccip-sdk/src/ton/index.test.ts |
Adds TON resume tests. |
ccip-sdk/src/sui/index.test.ts |
Tests Sui resume floors. |
ccip-sdk/src/sui/events.ts |
Applies shared resume floors. |
ccip-sdk/src/solana/logs.ts |
Adds signature-based resume behavior. |
ccip-sdk/src/solana/logs.test.ts |
Tests Solana resume semantics. |
ccip-sdk/src/requests.ts |
Accepts hints when finding messages. |
ccip-sdk/src/logs.test.ts |
Tests shared floor merging. |
ccip-sdk/src/index.ts |
Exports the floor helper. |
ccip-sdk/src/evm/logs.ts |
Adds block/index resume filtering. |
ccip-sdk/src/evm/logs.test.ts |
Tests EVM resume behavior. |
ccip-sdk/src/chain.ts |
Defines the hint API and receipt plumbing. |
ccip-sdk/src/aptos/logs.ts |
Adds cursors and version-atomic batching. |
ccip-sdk/src/aptos/logs.test.ts |
Tests Aptos resume and batching behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Coverage Report |
The override scans the indexer directly (queryTransactionBlocks) and derived its checkpoint/time bounds only from hints.startBlock/startTime, so a since-only call scanned from genesis. Merge with withSinceStart first.
The floor merge ran before the composite-hash account check, so a hint with a matching address field but a foreign embedded account still raised startBlock/startTime and could skip this account's logs. A provably foreign hint is now dropped wholesale — cursor and floors alike; the regression test uses a hint block above startBlock so a leaked floor would be observable.
Collaborator
Author
|
Both Copilot review comments addressed in the two fix commits just pushed:
Full unit suite re-run after both fixes: 171/171 across the touched suites, |
Comment on lines
+688
to
+690
| } catch { | ||
| delete opts.since | ||
| } |
Comment on lines
+296
to
+300
| const startUtime = Math.max(0, Math.floor(Number(opts.startTime ?? 0))) | ||
| let firstPage: TonV3Message[] | ||
| try { | ||
| const [page, indexedTip] = await Promise.all([ | ||
| fetchV3Messages(ctxV3, { source: acct.toRawString(), startUtime, limit }), |
Comment on lines
+203
to
+207
| export function prettyFormat( | ||
| value: unknown, | ||
| sameFamily: ChainFamily, | ||
| familyFor: Readonly<Record<string, ChainFamily>> = {}, | ||
| ): unknown { |
Comment on lines
+1471
to
1473
| if (/^-?\d+:[0-9a-fA-F]{64}$/.test(bytes)) { | ||
| return bytes | ||
| } |
Comment on lines
+303
to
+307
| if (indexedTip < cutoff - V3_MAX_INDEX_LAG) { | ||
| throw new CCIPHttpError( | ||
| 0, | ||
| `TON v3 index lagging: tip ${indexedTip} vs requested cutoff ${cutoff}`, | ||
| ) |
andrevmatos
force-pushed
the
feat/sinceLog
branch
from
August 22, 2026 21:45
f86778a to
11470ec
Compare
CI (saw 97 on the public index) exceeded the 96 per-tx seqno RPC cap: txs newer than the index's tip fall back to per-tx seqno resolution by design, and the public index lags a few blocks, so a healthy walk legitimately falls back for its most recent txs. Oracle engagement is already enforced by the retry loop (v3transactions >= 1); keep the cap as a loose guard against the pre-oracle ~2-3 RPCs per walked tx.
andrevmatos
force-pushed
the
feat/sinceLog
branch
from
August 22, 2026 22:03
11470ec to
3551f22
Compare
Still keep v2 for >=-4h
…STENT The coverage test requires every error code to carry a recovery hint; the new stream-inconsistency code (index/RPC disagreement mid-scan) was missing both the hint and the transient-code registration.
…aded CI runners' shared egress gets 429-stormed by the keyless public toncenter index, making the live scans flap. Add a preflight probe of the v3 tip (twice before skipping) and skip individual tests whose index-dependent assertions can't hold under a degraded index (every attempt truncated, or the meta path never engaged). Healthy networks still run the full suite; the mocked unit tests cover the logic hermetically.
The hint's signature used to seed the node's transaction-granular `until` cursor, which drops the WHOLE hinted tx — including logs after the hint's index (batch executions, multi-topic streams). The walk now pages to the hint's slot floor and SolanaChain.getLogs applies the per-log (transactionHash, index) exclusion, mirroring EVM's per-block behavior.
…fast path The v2 walk's `to_lt` cursor is transaction-granular, so a tx-exclusive floor at the hint's lt silently skipped the hinted tx's own followers. With the composite tx lt AND the hint's per-log index (a log's created_lt), the walk now floors inclusively at the hinted tx and drops only its logs at/before the hint's index; index-carrying hints also seed the v3 /messages fast path at index + 1 (start_lt inclusive, message-granular), so same-tx and same-block logs always flow and the hinted log is never re-fetched. A composite-only hint stays tx-exclusive, an index-only one floors at the message lt. startTime is applied as an emit-time filter when the scan floor is block-derived.
andrevmatos
force-pushed
the
feat/sinceLog
branch
from
August 24, 2026 16:06
470a9c7 to
f44e28b
Compare
…harden withSinceStart typing An addressless (topic-/typeAndVersions-only) sweep's since hint is its own last emitted log — every log carries an address, and (blockNumber, index) is the previously-emitted cursor — so it is honored wholesale: the hinted block is fetched whole and its logs at or before the hinted index are skipped, exactly as on address-scoped filters. Only an address-scoped filter whose hint addresses another contract drops the hint wholesale (no floors, no exclusion). withSinceStart is retyped to accept LeanNumbers<LogFilter> callers, keeps its kind-preserving floor writes and drops the hint's tx backref; the merged startBlock remains the scan floor with startTime filtered at emission (contract wording updated). getMessagesInBatch destructures a leaked `since` out of its backward retry legs.
andrevmatos
force-pushed
the
feat/sinceLog
branch
from
August 24, 2026 18:59
f44e28b to
5a4a139
Compare
…ation The v3 fast path assumed /messages rows are created_lt-ascending; the index actually orders them by created_at (lts regress across rows of the same second), so an lt-based page cursor could silently skip rows and the forward-only meta prune turned out-of-order joins into endless truncation. Pages now turn by created_at with a per-page created_lt re-sort, meta joins by tx hash without pruning, a boundary second overflowing one page is drained via offset (never cut), a row missing its tx hash truncates loudly, and the turn cursor is the page's max created_at so it never regresses. The scan is clamped to the index's ingested tip (blocks past it aren't indexed yet — emitting to the cutoff silently under-delivered, N2), and any truncation now surfaces as CCIPLogsStreamInconsistentError instead of ending silently (N3).
An all-empty fast-path scan (dormant address, nothing new) is almost always clamped: the index tip always trails a near-tip cutoff (latest - 1) by a few blocks, and busy streams only avoid the marker via the block > cutoff break. Throwing CCIPLogsStreamInconsistentError on every such poll was a permanent transient-error storm with zero silent-loss protection — nothing emitted means no poller watermark can have moved. The N2 truncation signal now fires only once a scan has EMITTED logs (whose watermark could be trusted past the ingested tip); pre-emission truncation (clamp or mid-stream corruption) ends quietly, as the tail is re-scanned on the next poll anyway. Unit tests cover both sides (loud after emissions, quiet before); the live integration suite treats the mid-stream throw as a retryable attempt instead of a hard failure.
The previous 'clamp' was a boolean keyed on the probe-time tip (cached ~30s)
vs a fresh cutoff (latest - 1): stale-tip drift could mark a scan truncated
even when it missed nothing, throwing CCIPLogsStreamInconsistentError
intermittently on healthy near-tip scans (looked like flake in CI; the
quiet-empty fix removed the storm but not the spurious signal).
openV3EventStream now really clamps: it returns { stream, cutoff:
min(cutoff, indexedTip) } and the consumer seals there, so every fast-path
scan ends as a COMPLETE prefix by construction — a watermark poller resumes
from its own last emitted log, so the tail is simply fetched next poll. A
PINNED endBlock above the ingested tip cannot be clamped (a definite-window
request would be silently under-delivered), so the fast path declines and
the complete v2 walk serves it. The truncated marker is now reserved for
genuine index self-contradiction (a message row missing its tx hash, or an
unrecoverable page turn), surfacing loudly only once logs were emitted (no
watermark can have moved before that; quiet-empty semantics kept).
Unit coverage: sealed-prefix clamp (was: throws), pinned-endBlock decline
(v2 walk delivers the whole window), plus the existing loud/quiet truncation
pair. Integration comments aligned; CHANGELOG updated.
Turn queries carry start_utime only (a created_lt cursor would silently skip regressing rows), so the boundary second re-serves rows BELOW a since hint's seed floor: the hinted log's own row and earlier rows of the same second (parallel shards). They were not in the seen set, so fresh() accepted them, they were hydrated, and the consumer read the earlier block as a mid-stream rewind (CCIPLogsStreamInconsistentError) on every multi-page hint scan. Rows below the seed are now counted for the offset drain (they are part of the server-side result set) but never hydrated or emitted, and fresh() excludes them so a floor-excluded prefix cannot loop the turn forever. They are deliberately NOT added to seen (keyed by tx hash): the hinted tx's own followers share that hash and must still flow. Repro test: page=2 hint scan whose boundary second carries the hinted tx's own row -> threw 'block rewind 11 -> 10' before, emits ['11001','11002'] after. Also: regression test for the rewind diagnostic naming both blocks (was interpolating undefined after the cursor reset), the deliberate-vs-silent asymmetry with the v2 walk documented at its catch site, and a stale '30 seconds' cache comment in solana fixed to the actual 5s TTL.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a partial
sinceresume hint toLogFilter: pass the last log from a previousgetLogscall, and the stream resumes just past it instead of re-scanning fromstartBlock/startTime. Aimed at watermark-driven pollers/watchers, where every poll currently re-walks pages (and, on TON, re-resolves shard floors) it has already seen.Behavior
withSinceStart):since.blockNumber/blockTimestampstand in forstartBlock/startTime— each effective floor is the larger of the explicit bound and the hint's, sosincealone satisfies the start requirement.getMessageByIdandgetExecutionReceiptsacceptsincetoo.since.blockNumber(blocks always fetched whole), excluding that block's logs withindex <= since.index; foreign-address hints ignored.untilcursor;since.blockNumberis an absolute slot floor for when the queried node doesn't know the signature (pruned/lagging). Watch polls also keep a resume cursor when the initial scan collects nothing.ltembedded in the compositetransactionHash(exclusive); skips theaccountShardEndLt(startBlock - 1)shard-header lookup when the hint coversstartBlock(saves 2 RPCs/scan on the steady-state poll path). Hints from a foreign account — by address field or the account embedded in the hash — are ignored.blockNumber + 1.getLogsnot implemented yet.Bundled fixes (required by / uncovered by the above)
blockNumber + 1floor sound). Also fixescatchedUpdropping the event exactly at a full page boundary (start > end, was>=).getTransactionswindow cache — redundant oncesincebounds rescans, and it retained parsed account history for the life of watch streams (production memory hazard).Testing
withSinceStartmatrix), exclusive-resume semantics, stale/foreign/malformed hints, watch-mode empty-scan resume, Aptos batch-boundary atomicity, EVM index exclusion; changelog entry included.ccip-sdkunit suite passes (1120 tests,tscclean). Commits are bisect-clean: core plumbing first, then one commit per chain, TON cache removal split from the TON feature.