feat: support OPTIMIZE INDEX SQL

[majin1102]

Summary

• add ALTER TABLE ... OPTIMIZE INDEX SQL support
• maintain only the named index through Dataset.optimizeIndices() on the driver
• optionally pass through num_indices_to_merge; when omitted, leave merge policy to Lance Core
• enforce table writeability, reject system indexes, and preserve catalog-provided storage options
• derive output metrics from Lance Core index descriptions and statistics
• normalize and validate SQL options consistently across supported Spark versions
• add parser, end-to-end, read-only branch, error-path tests, and documentation

Size-tiered parameters are tracked separately in #744.

Closes #743

Testing

• ./mvnw spotless:apply
• Spark 3.4: OptimizeTest, LanceSqlExtensionsAstBuilderTest, and BranchDDLTest (63 tests)
• Spark 3.5: OptimizeTest, LanceSqlExtensionsAstBuilderTest, and BranchDDLTest (65 tests)
• Spark 4.2 / Scala 2.13 compile

[puchengy]

@majin1102 Hi, I didn't noticed we have a PR already, and I created my own version #755, do you plan to continue to work on this? Thanks

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-02T10:20:14.192408Z	c654ac5	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[majin1102]

@majin1102 Hi, I didn't noticed we have a PR already, and I created my own version #755, do you plan to continue to work on this? Thanks

Sorry for the late reply. This PR is now ready for review, and your feedback would be very welcome. There are also a few related issues we can look into together：#744, #750

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The transaction-derived metrics resolve the earlier attribution risk, but two state-transition failures remain: a valid MemWAL catch-up commit is reported as a command failure, and a concurrent DROP INDEX can be undone by the optimize commit.

Accept an empty target delta as a successful zero-work commit when segment state is unchanged. For concurrent drop, the safe path is to fix Lance Core's CreateIndex/drop conflict handling and consume that fix here before exposing this operation through SQL.

[lance-gatekeeper]

This call can rebase past a concurrent same-name DROP INDEX and publish the index again. Lance Core 11's drop is a CreateIndex transaction with no new_indices, so its current conflict check does not reject the stale optimizer's CreateIndex. That violates the durable index-lifecycle contract exposed by this SQL command. Please fix the conflict in Lance Core and bump that dependency here (or provide an equally authoritative exclusion); a post-commit check cannot prevent the resurrection.

Reproducer

I ran a temporary Spark 3.5 / Scala 2.13 JUnit test on this head: create a zonemap idx, append an unindexed fragment, open two Dataset handles, call dropIndex("idx") on one, then call targeted optimizeIndices on the stale handle. Reopening the dataset made assertFalse(fresh.listIndexes().contains("idx")) fail with expected: <false> but was: <true>.

[majin1102]

Agreed that this is a Lance Core transaction-conflict bug rather than something Spark SQL can safely repair after the fact. It affects the existing Dataset.optimizeIndices API independently of this SQL wrapper. I filed lance-format/lance#8982 with the reproducer and expected lifecycle invariant: lance-format/lance#8982. I am intentionally not adding Spark-local locking or an unrelated dependency/workflow change in this PR; once Core ships the conflict fix, lance-spark should consume it through the normal Core dependency update.

[lance-gatekeeper]

The Core fix has now merged in lance#8984, but this PR still uses lance-core 11.0.0-beta.21 and therefore remains vulnerable. The SQL command should wait for and consume a Core release containing that fix.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The MemWAL empty-delta failure is fixed. One prerequisite remains: this new SQL surface still uses Lance Core 11.0.0-beta.21, where a concurrent DROP INDEX can be undone by an optimize commit. lance#8984 has merged the authoritative conflict fix; consume a Core release containing it here before merging this SQL command.