[Epic] Production-ready Lance Namespace support across daft-lance and Daft

[FANNG1]

Goal

Deliver Lance Namespace support from daft-lance through Daft's user-facing APIs, with Gravitino Lance REST as the primary integration target.

Delivery plan

• Foundation namespace support — Support Lance Namespace table addressing (catalog-managed Lance tables, e.g. Gravitino Lance REST server) #50 / feat(namespace): support Lance Namespace across table operations #35
  • Address tables by namespace implementation, properties, and table ID.
  • Support read, create, append, overwrite, index, compaction, and existing merge APIs.
  • Use catalog-vended storage credentials and propagate managed versioning correctly.
• write_lance(mode="merge") namespace semantics — Support Lance Namespace in write_lance merge mode #52
• Release a new daft-lance version
  • Document supported namespace operations and publish the package.
  • Verify the published package against Gravitino without ambient AWS credentials.
• Update Daft's write_lance entry point — Complete the daft.io.lance -> daft_lance migration: delegate DataFrame.write_lance (follow-up to #6957) Eventual-Inc/Daft#7282
  • Delegate to the released daft-lance implementation and expose namespace parameters without monkeypatching.

Follow-up optimization

• Refresh temporary catalog credentials in distributed workers — Refresh temporary namespace credentials in distributed workers #53

Credential refresh improves reliability for long-running jobs but does not block the initial namespace feature, daft-lance release, or Daft entry-point integration.

Done when

• Namespace-addressed tables work through daft-lance with catalog-vended credentials.
• A released daft-lance package contains the feature.
• Daft exposes the namespace-capable write_lance entry point.

[FANNG1]

@rchowell @universalmind303, could you take a look at this proposed breakdown and let us know whether it seems reasonable?

The current plan is:

1. Keep feat(namespace): support Lance Namespace across table operations #35 focused on the foundation: namespace addressing, catalog-vended credentials, managed versioning, and the existing read/write/index/compaction/merge APIs.
2. Track write_lance(mode="merge") semantics separately in Support Lance Namespace in write_lance merge mode #52.
3. Release a new daft-lance version, then update Daft's write_lance entry point in Complete the daft.io.lance -> daft_lance migration: delegate DataFrame.write_lance (follow-up to #6957) Eventual-Inc/Daft#7282.
4. Treat distributed temporary-credential refresh (Refresh temporary namespace credentials in distributed workers #53) as a follow-up reliability improvement rather than a blocker for the initial feature and release.

In particular, feedback on the package boundary, delivery order, and whether any of these items should move into or out of #35 would be very helpful.