Rename Dataproc Spark Connect to Managed Spark Connect - #188
Open
ajma wants to merge 7 commits into
Open
Conversation
Renames the package, modules, and public classes from Dataproc-branded names to Managed Spark Connect (DataprocSparkSession -> ManagedSparkSession, dataproc_spark_connect -> managed_spark_connect, dataproc_magics -> managed_spark_magics, PyPI package dataproc-spark-connect -> managed-spark-connect). The sessionTemplate() builder method is renamed to runtimeProfile(), and the DATAPROC_SPARK_CONNECT_* env vars gain MANAGED_SPARK_CONNECT_* equivalents. Old import paths, class names, sessionTemplate(), and env var names are kept as deprecated aliases that emit a DeprecationWarning, so existing integrations keep working. References to the actual underlying GCP Dataproc API (google-cloud-dataproc, dataproc_v1, dataprocSessionConfig/ dataprocSessionId, dataproc.googleapis.com) are left unchanged since they name real external resources, not this library's branding.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
requirements-dev.txt pins pyink~=24.0, but the pyink CI workflow installs the latest unpinned version (26.5.1), which enforces a blank line after module docstrings.
Pins all action references to commit SHAs, adds an explicit contents: read permission, sets persist-credentials: false on checkout, and stops extracting secret fields via fromJson(secrets.*) in expressions (which bypasses GitHub's log redaction) in favor of a dedicated step that masks the derived values explicitly.
Contributor
Author
|
I removed the Gemini comments since this PR is just file renames but that seems to trigger Gemini |
medb
requested changes
Aug 27, 2026
Aligns with Google's naming convention for other google-cloud-* client libraries rather than the shorter managed-spark-connect.
…Spark rename Removes the google.cloud.dataproc_spark_connect / google.cloud.dataproc_magics shim packages, the DataprocSparkSession/DataprocMagics aliases, the sessionTemplate() deprecated alias for runtimeProfile(), and the DATAPROC_SPARK_CONNECT_* env var fallback. Anyone moving to the new google-cloud-managed-spark-connect package needs to update their imports, builder calls, and env vars directly rather than relying on a deprecation period.
Spells out each breaking change individually (package name, imports, sessionTemplate() -> runtimeProfile(), env var prefix) with code examples instead of a flat bullet list, since sessionTemplate -> runtimeProfile in particular isn't obvious from a name-only diff.
Undo the runtimeProfile() rename; the method still sets Session.session_template but keeps its original name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
google.cloud.dataproc_spark_connect→google.cloud.managed_spark_connect,google.cloud.dataproc_magics→google.cloud.managed_spark_magics,DataprocSparkSession→ManagedSparkSession,DataprocMagics→ManagedSparkMagics, and the PyPI distributiondataproc-spark-connect→google-cloud-managed-spark-connect.sessionTemplate()builder method toruntimeProfile(), and the library's ownDATAPROC_SPARK_CONNECT_*env vars toMANAGED_SPARK_CONNECT_*.google-cloud-dataprocdependency,dataproc_v1imports,dataprocSessionConfig/dataprocSessionIdmethods,dataproc.googleapis.comendpoints) since those name real external resources, not this library's branding..env.example, CI workflow env vars, and the Cloud Build image tags accordingly.integration-tests.yamlto commit SHAs and stops extracting secret fields viafromJson(secrets.*)in expressions (fixes zizmor security findings).Test plan
pytest tests/unit— 161 passed (1 pre-existing, environment-dependent test deselected: it assumesgoogle.auth.default()fails with no ADC configured, which doesn't hold when running on GCP-hosted infra with metadata-server credentials)pyink --checkpasses on all touched filespython -m build+twine check dist/*passes under the newgoogle-cloud-managed-spark-connectdistribution namezizmorreports zero findings onintegration-tests.yaml