Add support for Custom Credential Loader in Opendal GCS storage interface - #8
Conversation
| /// succeeds using whatever else the chain can reach, including the very static | ||
| /// `gcs.oauth2.token` the loader exists to replace, or ambient credentials on a GCE | ||
| /// instance. | ||
| fn suppress_default_credential_sources(cfg: &mut GcsConfig) { |
There was a problem hiding this comment.
Is there a reason this gets done as part of building the operator instead of during config initialization (e.g. crate::gcs::gcs_config_parse)?
There was a problem hiding this comment.
so this is because config initialization is not actually a required step, it happens in the OpenDalStorageFactory::build function, and one could technically skip that and the whole OpenDalStorageFactory step, go straight to building an OpenDalStorage::Gcs{ ... } themselves, meaning this step gets skipped.
Arguably that's fine, since we're realistically the only ones using this interface and this is the only location where it happens, but this way just ensures that the suppression actually happens. Maybe even we should make the default credential suppression optional so that customers can decide to use it as a fallback, but I'm okay with saying that if they specify vended-credentials in their catalog connection, then we WILL use vended creds, and error if the creds don't work for some reason.
Install Hive directly Signed-off-by: Jannik Steinmann <jannik.steinmann@datadoghq.com> (cherry picked from commit 4687d26)
…che#3187) fix(hms): use get_table_req for Hive 4 metastore compatibility Hive 4.0.1 removed the `get_table` thrift method, so the HMS catalog fails with `Invalid method name: 'get_table'` against any Hive 4 metastore. Switch `load_table`, `table_exists`, and `rename_table` to `get_table_req`, which exists in every Hive release since 2.3 (the IDL the `hive_metastore` crate is generated from), so older metastores keep working. Move the integration test metastore from `apache/hive:3.1.3` (Debian Bullseye, EOL) to `apache/hive:4.2.1` so CI exercises a Hive 4 server, mirroring apache/iceberg-python#3924. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 4aa55ba)
|
cherry picked a dockerfile fix for the Hive install from the upstream that was causing the CI failure here, probably just going to leave it in this PR since it'll be promptly merged into |
5 participants
Updates
OpenDalStorage::Gcsto follow the pattern thatOpenDalStorage::Awsto allow vended credentials from the materialize side, by adding an option to provide a custom credential loader to the storage factory that is used to fetch credentials when writing to storage. On the materialize side, this will be implemented by a refreshing vended credential fetcher and passed in.