Repository navigation
feat(rest): add scan plan endpoint support to REST catalog client - #783
gsandeep1241 wants to merge 36 commits into
Conversation
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
Hey @gsandeep1241, do you want to revive this? |
|
@wgtmac Thanks for re-opening it. Apologies for the long delay, I'll get back on it this week - next set of changes should be out for review in the next couple of days! |
fb8da85 to
d8a7d6c
Compare
2bab861 to
bcc9da4
Compare
Thanks @wgtmac for reviving this! This is now ready for review. Please take a look when you can :) |
a9cbb7d to
5de0344
Compare
|
@wgtmac Thank you for the detailed review. I've addressed your comments. Can you please take a look once more when you have the chance? |
| if (metadata_->current_snapshot_id == kInvalidSnapshotId) return {}; | ||
| ICEBERG_ASSIGN_OR_RAISE(auto snapshot, metadata_->Snapshot()); | ||
| if (!snapshot) return {}; | ||
| request.end_snapshot_id = snapshot->snapshot_id; |
There was a problem hiding this comment.
When no explicit to_snapshot_id is set, this always uses the table's current snapshot and ignores context_.branch. A branch whose head differs from main will send the wrong end-snapshot-id; please resolve the branch head here and validate explicit endpoints against it.
There was a problem hiding this comment.
Addressed by overriding the other PlanFiles method here and letting the base handle it.
| RestScanContext rest_context); | ||
|
|
||
| RestScanContext rest_context_; | ||
| mutable std::shared_ptr<FileIO> scan_io_; |
There was a problem hiding this comment.
This stores the vended scan IO, but the incremental scan does not override TableScan::io(). Normal callers therefore keep using the table IO after PlanFiles(), so credentials returned for incremental planning are never used to read the tasks. Please expose the effective IO here as well.
| Result<std::unique_ptr<DataTableScan>> RestTableScanBuilder::Build() { | ||
| ICEBERG_RETURN_UNEXPECTED(CheckErrors()); | ||
| ICEBERG_RETURN_UNEXPECTED(context_.Validate()); | ||
| ICEBERG_ASSIGN_OR_RAISE(auto schema, ResolveSnapshotSchema()); |
There was a problem hiding this comment.
ResolveSnapshotSchema() always returns the snapshot schema when snapshot_id is set, even when UseRef() marked a branch scan with use_snapshot_schema=false. Java uses the current table schema for branches and the snapshot schema for tags/time travel; the schema passed to this REST scan should follow that flag.
There was a problem hiding this comment.
Done. Fixed in table_scan.cc
| .catalog_config = config_.configs(), | ||
| .table_config = table_config, | ||
| }; | ||
| return RestTable::Make(identifier, std::move(result.metadata), |
There was a problem hiding this comment.
This creates a RestTable only when the table is loaded. MakeTableFromCommitResponse() still returns a plain Table, so committing through a server-planning table silently switches later scans back to local manifest planning. Please preserve the REST table type and scan context after commit.
| std::shared_ptr<FileIO>& scan_io) { | ||
| if (credentials.empty()) return {}; | ||
| ICEBERG_ASSIGN_OR_RAISE( | ||
| auto io, MakeTableFileIO(ctx.catalog_config, ctx.table_config, credentials)); |
There was a problem hiding this comment.
This builds the scan IO from static credentials only. It does not install a StorageCredentialProvider or pass the scan plan-id; Java adds rest-scan-plan-id and refreshes vended credentials through the credentials endpoint. Temporary credentials can otherwise expire during a scan.
| ICEBERG_UNWRAP_OR_FAIL(auto tasks1, scan->PlanFiles()); | ||
| EXPECT_TRUE(tasks1.empty()); | ||
|
|
||
| // Second call: no credentials. Verifies scan_io_ was cleared so stale |
There was a problem hiding this comment.
These assertions only check that planning returns tasks. Please also call scan->io() after vended credentials are returned; the public incremental-scan path currently exposes the original table IO.
| // UseSnapshot(): the POST body sent to the server must contain both | ||
| // "snapshot-id" and "use-snapshot-schema": true. | ||
| // -------------------------------------------------------------------------- | ||
| TEST_F(RestTableScanTest, UseSnapshotPropagatesUseSnapshotSchemaInContext) { |
There was a problem hiding this comment.
Please add request-body tests for Project() and IncludeColumnStats({"..."}). The current suite checks snapshot flags, but it does not catch the missing select projection or stats-fields described above.
| // -------------------------------------------------------------------------- | ||
| // PlanFilesStream: SUBMITTED → poll until COMPLETED, stream yields all tasks. | ||
| // -------------------------------------------------------------------------- | ||
| TEST_F(RestTableScanTest, PlanFilesStreamSubmittedThenCompleted) { |
There was a problem hiding this comment.
This submitted case has no plan tasks, so it cannot verify lazy fetching. Please return multiple plan-task tokens from the poll response and assert that /tasks is not called until Next() consumes the stream.
| EXPECT_FALSE(scan->context().use_snapshot_schema); | ||
| } | ||
|
|
||
| INSTANTIATE_TEST_SUITE_P(TableScanVersions, TableScanTest, testing::Values(1, 2, 3)); |
There was a problem hiding this comment.
These use_snapshot_schema checks do not depend on format version, but the parameterization runs each one for versions 1, 2, and 3. Please make these non-parameterized or avoid repeating the same assertions three times.
| // PlanFilesStream: stream destroyed with no Next() calls at all → | ||
| // DELETE /plan called by the destructor. | ||
| // -------------------------------------------------------------------------- | ||
| TEST_F(RestTableScanTest, PlanFilesStreamCancelOnPartialConsumption) { |
There was a problem hiding this comment.
The test name says partial consumption, but the stream is destroyed without any Next() call. Please rename it to make the scenario clear, such as CancelOnNoConsumption.
6dcfb2e to
5ff7165
Compare
gsandeep1241
left a comment
There was a problem hiding this comment.
Addressed all comments.
| if (metadata_->current_snapshot_id == kInvalidSnapshotId) return {}; | ||
| ICEBERG_ASSIGN_OR_RAISE(auto snapshot, metadata_->Snapshot()); | ||
| if (!snapshot) return {}; | ||
| request.end_snapshot_id = snapshot->snapshot_id; |
There was a problem hiding this comment.
Addressed by overriding the other PlanFiles method here and letting the base handle it.
| RestScanContext rest_context); | ||
|
|
||
| RestScanContext rest_context_; | ||
| mutable std::shared_ptr<FileIO> scan_io_; |
| Result<std::unique_ptr<DataTableScan>> RestTableScanBuilder::Build() { | ||
| ICEBERG_RETURN_UNEXPECTED(CheckErrors()); | ||
| ICEBERG_RETURN_UNEXPECTED(context_.Validate()); | ||
| ICEBERG_ASSIGN_OR_RAISE(auto schema, ResolveSnapshotSchema()); |
There was a problem hiding this comment.
Done. Fixed in table_scan.cc
| .catalog_config = config_.configs(), | ||
| .table_config = table_config, | ||
| }; | ||
| return RestTable::Make(identifier, std::move(result.metadata), |
|
|
||
| Result<std::unique_ptr<DataTableScan>> RestTableScanBuilder::Build() { | ||
| ICEBERG_RETURN_UNEXPECTED(CheckErrors()); | ||
| ICEBERG_RETURN_UNEXPECTED(context_.Validate()); |
|
|
||
| constexpr int64_t kMinSleepMs = 1'000; | ||
| constexpr int64_t kMaxSleepMs = 60'000; | ||
| constexpr int kMaxRetries = 10; |
There was a problem hiding this comment.
Done. Kept the sleeps here to match java.
| ICEBERG_ASSIGN_OR_RAISE(auto json, FromJsonString(response.body())); | ||
| ICEBERG_ASSIGN_OR_RAISE(auto result, | ||
| PlanTableScanResponseFromJson(json, specs, schema)); | ||
| ICEBERG_RETURN_UNEXPECTED(result.Validate()); |
| std::shared_ptr<FileIO>& scan_io) { | ||
| if (credentials.empty()) return {}; | ||
| ICEBERG_ASSIGN_OR_RAISE( | ||
| auto io, MakeTableFileIO(ctx.catalog_config, ctx.table_config, credentials)); |
| TableScanBuilder<ScanType>::ResolveSnapshotSchema() { | ||
| if (snapshot_schema_ == nullptr) { | ||
| if (context_.snapshot_id.has_value()) { | ||
| if (context_.snapshot_id.has_value() && context_.use_snapshot_schema) { |
There was a problem hiding this comment.
This changes existing behavior for locally planned scans too.
When a table is loaded from a REST catalog that advertises the PlanTableScan
endpoint, NewScan() now returns a RestTableScanBuilder whose Build() produces
a RestTableScan. PlanFiles() on that scan delegates manifest resolution to
the server via POST /plan, GET /plan/{id} (with exponential backoff),
POST /tasks/{id}, and DELETE /plan/{id} (best-effort cancel), instead of
reading manifests locally.
- Add RestTable, RestTableScanBuilder, RestTableScan and RestScanContext
- Promote DataTableScan::PlanFiles and TableScanBuilder::Build to virtual
- Convert RestCatalog::client_ and paths_ to shared_ptr so RestScanContext
can share ownership with live scans
- Cancel server-side plan when ResolveScanTasks fails partway through - Propagate use_snapshot_schema from scan context to PlanTableScanRequest: true for UseSnapshot/AsOfTime/tag refs and incremental scans, false for branch refs and default scans - Gate RestTable creation on effective scan-planning-mode config (table config overrides client config, default is client); error if server mode is requested but endpoint is not advertised - Add ScanPlanningMode enum and ScanPlanningModeFrom() parser to RestCatalogProperties - Make HttpClient methods virtual and add HttpResponse::MakeForTesting() to support unit test mocking - Add tests: use_snapshot_schema in table_scan_test, ScanPlanningModeFrom parsing in catalog_properties_test, and RestTableScan HTTP flow tests in rest_table_scan_test
PlanFiles() was made virtual in this PR to support a RestTableScan override. That override has since been replaced by RestTableScan::PlanFilesStream(), so PlanFiles() no longer needs to be virtual — callers dispatch through the virtual PlanFilesStream() hook instead.
… and virtual io() - PlanFilesStream: COMPLETED, lazy token fetching, cancel-on-partial-consume, SUBMITTED→poll→COMPLETED, FAILED - RestIncrementalAppendScan: COMPLETED, plan-tasks, SUBMITTED→poll→COMPLETED, FAILED, endpoint-not-supported, no-current-snapshot returns empty, explicit to_snapshot_id, exclusive from_snapshot_id, inclusive from_snapshot_id with and without a parent snapshot - RestTable::NewIncrementalAppendScan returns RestIncrementalAppendScanBuilder
FetchPlanningResult returned early on GET failure, JSON parse error, response deserialization error, Validate() failure, and ApplyStorageCredentials failure without cancelling, leaving the server holding plan resources. ExecuteScanPlanStream and ExecuteScanPlan had the same gap for the kFailed status and ApplyStorageCredentials failure in the kCompleted branch. Replace ICEBERG_ASSIGN_OR_RAISE / ICEBERG_RETURN_UNEXPECTED with explicit checks in FetchPlanningResult's poll loop, and add CancelPlanning before every error return in ExecuteScanPlanStream and ExecuteScanPlan where plan_id is known.
If PlanFilesStream()/PlanFiles() is called more than once and only the first response vends storage credentials, the second plan would inherit the stale credential-scoped FileIO from the first. Reset the slot to null before each planning call so that io() falls back to the table IO when the server does not vend credentials in the new response.
…ial fixes - Add JsonBodyHas/JsonBodyLacks GMock matchers that parse the POST body so request field propagation is verified against actual server JSON, not just internal context state - Update UseSnapshot and DefaultScan tests to call PlanFiles() and match "snapshot-id"/"use-snapshot-schema" in the POST body - Update incremental scan snapshot routing tests to match "start-snapshot-id"/"end-snapshot-id" values in the POST body - Add PlanFilesStreamFailedWithPlanIdCancels: verifies DELETE is called when ExecuteScanPlanStream receives FAILED with a plan-id - Add CancelCalledOnFetchPlanningResultGetError: verifies DELETE is called when FetchPlanningResult GET fails after SUBMITTED - Add SecondPlanFilesStreamCallClearsStaleCredentials: verifies scan_io_slot_ is reset so second plan response with no credentials reverts io() to table IO - Add PlanFilesFailedWithPlanIdCancels for RestIncrementalAppendScan - Add SecondPlanFilesCallClearsStaleCredentials for RestIncrementalAppendScan
…partial-consume tests - Rename PlanFilesStreamLazilyFetchesPlanTasks to PlanFilesStreamFetchesEachTokenSeparately; verify task content and count rather than empty responses, since the old comment implied laziness was being tested when only call-count was verified - Add PlanFilesStreamFetchesTokenOnlyWhenBufferExhausted: uses a counter incremented inside WillOnce lambdas to assert the second FetchScanTasks POST is not made until the first token buffer is fully drained - Add PlanFilesStreamNextReturnsErrorOnFetchFailure: calls Next() directly and asserts it returns IOError when FetchScanTasks fails - Add PlanFilesStreamCancelAfterPartialConsumption: consumes one task then destroys the stream, verifying DELETE called with a token still pending - Retain PlanFilesStreamCancelOnPartialConsumption with updated comment clarifying it tests destroy-before-any-Next
…ests
- Add MakeForTesting() factory to RestCatalog that bypasses FetchServerConfig,
enabling unit tests to inject pre-built client/paths/endpoints.
- Add rest_catalog_unit_test.cc with four tests covering:
table config server scan -> RestTable, client config server scan -> RestTable,
server scan with missing PlanTableScan endpoint -> NotSupported,
and default (no config) -> plain Table.
- Fix RestIncrementalAppendScan::PlanFiles() to check kInvalidSnapshotId
before calling Snapshot(), so a table with no current snapshot returns
empty results rather than a NotFound error.
- Fix make_unique<RestFileScanTaskStream> deduction failure by replacing
brace-init {} with std::vector<std::string>{} for plan_task_tokens arg.
- Relax PlanTableScanResponse::Validate() to allow plan-id in failed
responses, enabling the cancel-on-fail code path to be reached.
- Update rest_json_serde_test.cc: remove FailedWithPlanId invalid case,
add FailedWithPlanId roundtrip test.
- Add iceberg/manifest/manifest_entry.h and iceberg/constants.h includes
to rest_table_scan_test.cc to fix DataFile and kInvalidSnapshotId access.
Override the two-arg PlanFiles so the snapshot range is resolved by ResolvePlanFiles, which uses the branch head when UseBranch() is set and validates explicit start/end snapshots against it. Return NotSupported when there is no start snapshot, since the REST API requires both start-snapshot-id and end-snapshot-id.
Override io() so callers read incremental scan tasks with the FileIO built from storage credentials returned during planning, falling back to the table's FileIO when none are returned.
… commit ResolveSnapshotSchema() now honors use_snapshot_schema, so branch scans use the current table schema while tags and time travel use the snapshot schema, matching Java's SnapshotUtil.schemaFor(table, ref). MakeTableFromCommitResponse() now goes through the same scan planning mode resolution as LoadTable, so committing through a server-planning table returns a RestTable instead of a plain Table.
REST scan builders now resolve IncludeColumnStats() columns, so
stats-fields is sent and unknown columns are rejected.
The plan request's select is derived from the scan's projected schema,
matching Java's RESTTableScan, so Project() is honored, filter columns
are included, and Select("*") expands to all columns.
FetchPlanningResult now returns the completed response instead of resolving its plan tasks, so a SUBMITTED plan builds the same lazy stream as an immediate COMPLETED one. The stream also queues nested plan tasks returned by fetchScanTasks instead of resolving them eagerly, matching Java's ScanTaskIterable.
REST scans fetch plan tasks serially and do not use an executor set via PlanWith(). Document this, add a TODO for concurrent fetching, and test that the executor is not used.
PlanTableScanResponseFromJson validates the response, so a failure dropped the plan id before it could be cancelled. Read the plan id from the raw JSON first and cancel the plan if parsing or validation fails.
Add rest-scan-planning.poll-timeout-ms (default 300000, must be positive) and rest-scan-planning.poll-num-retries (default 10, must be non-negative), matching Java. They are validated before the plan is submitted, and polling no longer sleeps after the final attempt.
Main now aliases Result to ::iceberg::expected so the library builds in C++20 compat mode; std::unexpected no longer converts to it.
Vended credentials from scan planning responses now get a StorageCredentialProvider that refreshes through the LoadCredentials endpoint with the planId query parameter, matching the table FileIO provider and Java's plan-scoped credential refresh.
Restore the PlanTableScanResponse validation that allows plan-id only for submitted or completed plans. The plan is still cancelled because the plan-id is read from the raw response before parsing.
A completed plan response may omit plan-id. Refresh credentials table-scoped in that case instead of sending an empty planId query parameter.
The spec only defines storage-credentials on completed planning results, and Java ignores them on fetchScanTasks. Drop the field from FetchScanTasksResponse so later responses can no longer replace the plan's scan FileIO, and keep the scan FileIO as a plain member now that the stream no longer writes it.
Preserve residual filters, keep scan sessions alive, and centralize plan cleanup. Share planning context and consolidate REST tests.
e848b44 to
b40f915
Compare
wgtmac
left a comment
There was a problem hiding this comment.
Thanks for working on this! I rebased onto main and fixed residual handling, plan cleanup, and auth-session lifetime. I also simplified the APIs and tests to reduce duplication.
When a table is loaded from a REST catalog with scan-planning-mode=server (table config takes precedence over catalog config), NewScan() now returns a RestTableScanBuilder whose Build() produces a RestTableScan. PlanFiles() and PlanFilesStream() on that scan delegate manifest resolution to the server via POST /plan, GET /plan/{id} (with exponential backoff), POST /tasks, and DELETE /plan/{id} (best-effort cancel), instead of reading manifests locally.