Skip to content

workspaces: add Spark Connect operator and user guides for notebooks and workspaces - #4479

Open
vikas-saxena02 wants to merge 14 commits into
kubeflow:masterfrom
vikas-saxena02:docs-workspaces-spark-connect
Open

vikas-saxena02 wants to merge 14 commits into
kubeflow:masterfrom
vikas-saxena02:docs-workspaces-spark-connect

Conversation

@vikas-saxena02

@vikas-saxena02 vikas-saxena02 commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Drafted with the help of Claude.

Description of Changes

Adds guides for running Spark from a Kubeflow notebook pod with the Kubeflow SDK, so
users can start a Spark Connect session without hand-writing SparkApplication
manifests. Covers both Notebooks and Workspaces, since Workspaces is still beta and
Notebooks is what most users are running today.

All pages are deliberately short and defer the detail to the integration tests in
kubeflow/community-distribution, which CI keeps green on every pull request. Anything
that would date quickly — the per-session pod settings, the client/server version
pairing — lives there rather than here.

notebooks/spark-connect.md — the permission check, installing the client, a minimal
SparkClient.connect() example, and releasing the session. No administrator setup is
documented because none is needed: a notebook pod runs as default-editor, which a
Profile already binds to kubeflow-edit, and the aggregated kubeflow-spark-edit
ClusterRole carries sparkconnects.

workspaces/user-guides/spark-connect.md — the same flow for a Workspace. The only
pod-level setting shown is the driver ServiceAccount, via Driver(service_account=...).

workspaces/operator-guides/spark-connect.md — the one change an administrator must
make for Workspaces, binding kubeflow-spark-edit to the WorkspaceKind's ServiceAccount,
and why Spark pods have to stay out of the service mesh. Workspaces v2.0.0-beta.0 stopped
granting the Kubeflow ClusterRoles to ws-{WORKSPACE_NAME}, which is why this step exists
for Workspaces and not for Notebooks.

workspaces/user-guides/_index.md — new section landing page.

The mesh exclusion is presented as a platform concern in both guides rather than something
a data scientist configures. Its root cause is upstream:
SPARK-42411.

Changes since the first revision, from review:

  • Pages cut by roughly half, with the detail replaced by links to the tested examples
    (@juliusvonkohout).
  • The pod template overrides are gone from the user guides. The driver ServiceAccount now
    uses the connect API's own field, and the Istio content moved out of the user-facing
    flow (@tariq-hasan).
  • The "Image requirements" section is removed. The version pairing is handled, and tested,
    in the community-distribution test scripts, and chore(spark): replace pyspark-connect with pyspark-client (#781) sdk#795 is changing the
    underlying package anyway.
  • Notebooks coverage added alongside Workspaces.

Both tests the guides link to are merged in kubeflow/community-distribution and share one
session script, so the Notebooks and Workspaces examples are the same code exercised from
two pod types.

Related Issues

Related: kubeflow/notebooks#1362

Checklist

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/notebooks AREA: Kubeflow Notebooks size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants