Repository navigation
workspaces: add Spark Connect operator and user guides for notebooks and workspaces - #4479
Open
vikas-saxena02 wants to merge 14 commits into
Open
vikas-saxena02 wants to merge 14 commits into
vikas-saxena02 wants to merge 14 commits into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
Adds guides for running Spark from a Kubeflow notebook pod with the Kubeflow SDK, so
users can start a Spark Connect session without hand-writing
SparkApplicationmanifests. Covers both Notebooks and Workspaces, since Workspaces is still beta and
Notebooks is what most users are running today.
All pages are deliberately short and defer the detail to the integration tests in
kubeflow/community-distribution, which CI keeps green on every pull request. Anything
that would date quickly — the per-session pod settings, the client/server version
pairing — lives there rather than here.
notebooks/spark-connect.md— the permission check, installing the client, a minimalSparkClient.connect()example, and releasing the session. No administrator setup isdocumented because none is needed: a notebook pod runs as
default-editor, which aProfile already binds to
kubeflow-edit, and the aggregatedkubeflow-spark-editClusterRole carries
sparkconnects.workspaces/user-guides/spark-connect.md— the same flow for a Workspace. The onlypod-level setting shown is the driver ServiceAccount, via
Driver(service_account=...).workspaces/operator-guides/spark-connect.md— the one change an administrator mustmake for Workspaces, binding
kubeflow-spark-editto the WorkspaceKind's ServiceAccount,and why Spark pods have to stay out of the service mesh. Workspaces v2.0.0-beta.0 stopped
granting the Kubeflow ClusterRoles to
ws-{WORKSPACE_NAME}, which is why this step existsfor Workspaces and not for Notebooks.
workspaces/user-guides/_index.md— new section landing page.The mesh exclusion is presented as a platform concern in both guides rather than something
a data scientist configures. Its root cause is upstream:
SPARK-42411.
Changes since the first revision, from review:
(@juliusvonkohout).
uses the connect API's own field, and the Istio content moved out of the user-facing
flow (@tariq-hasan).
in the community-distribution test scripts, and chore(spark): replace pyspark-connect with pyspark-client (#781) sdk#795 is changing the
underlying package anyway.
Both tests the guides link to are merged in kubeflow/community-distribution and share one
session script, so the Notebooks and Workspaces examples are the same code exercised from
two pod types.
Related Issues
Related: kubeflow/notebooks#1362
Checklist