[Neha-Nimsatkar]

What this PR does / why we need it:
Replaces the heavy pyspark-connect dependency with pyspark-client to remove unnecessary JVM/JAR runtime dependencies from the client environment while maintaining remote Spark Connect functionality (sc://).

Audit & Investigation Findings:

• JAR Count Reduction: Reduced .jar count in .venv from 324 down to 0 (find .venv -name '*.jar' | wc -l returned 0).
• Dependencies Removed: Stripped transitive JVM runtime dependencies pyspark==4.2.0 and py4j==0.10.9.9.
• Spark Version Alignment (4.2.0 → 4.0.4): Aligned pyspark-client to version 4.0.4 (matching DEFAULT_SPARK_VERSION and apache/spark:4.0.4), eliminating client/server protocol skew and preventing unintended runtime drift from floating 4.2.0.
• Import Check: Verified from pyspark.sql import SparkSession imports cleanly without JVM dependencies.

Local Verification:

• make verify (linting, formatting, and type checks passed)
• make test-python (695/695 unit tests passed)
• test_spark_connect_crd_smoke passed
• Added entry to CHANGELOG/CHANGELOG-0.5.md covering pyspark-client migration, sc:// remote requirement, and JAR/Py4J removal.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

Fixes #781

Checklist:

• Docs included if any changes are user facing

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[tariq-hasan]

/ok-to-test
/retest

[RobuRishabh]

Hi @Neha-Nimsatkar thanks for this PR. Can you add a short CHANGELOG entry for the next release covering: [spark] now installs pyspark-client, remote-only via sc://, and full PySpark/Py4J/JARs no longer installed.

Optionally mention the 4.2.0 → 4.0.4 alignment in the PR description, it’s a meaningful improvement worth calling out.

Overall this looks good to me from a functionality and safety perspective. Nice work on #781!
cc: @tariq-hasan @Shekharrajak

[Neha-Nimsatkar]

@RobuRishabh Thanks for the review!

• Added the CHANGELOG entry under CHANGELOG/CHANGELOG-0.5.md covering the pyspark-client migration, remote-only sc:// requirements, and removal of bundled JARs/Py4J.
• Updated the PR description to call out the 4.2.0 → 4.0.4 version alignment with the Spark operator image.

All pre-commit checks and DCO sign-offs are in place. Ready for your final look!

[Shekharrajak]

/lgtm
/approve

[Shekharrajak]

/assign @astefanutti

[tariq-hasan]

We should remove the changelog. It’s auto-generated as part of cliff.toml.

[Neha-Nimsatkar]

We should remove the changelog. It’s auto-generated as part of cliff.toml.

Thanks @tariq-hasan , I have Reverted the changelog file and squashed down to a single clean commit .

[RobuRishabh]

/lgtm

Hi @andreyvelich @astefanutti @kramaranya @Electronic-Waste this PR has a direct security impact worth prioritizing.

The current pyspark-connect dependency pulls in the full pyspark package (324 bundled JARs) including CVE-prone Java libraries (Jackson, Netty, Log4j components, Hadoop, etc.) that the SDK never uses, it only talks to Spark Connect servers over gRPC. Vulnerability scanners flag these JARs, creating noise and blocking deployments for teams with compliance gates. Switching to pyspark-client drops these JARs, removes py4j and zstandard, fixes the 4.2.0→4.0.4 version skew, and all CI checks (including E2E) are green. It's a drop-in replacement with no API changes.

Could we get /approve when you get a chance? Thanks!

[Shekharrajak]

/lgtm
/approve

[Shekharrajak]

@astefanutti for merge.

[andreyvelich]

Thanks for this work everyone!
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, Shekharrajak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment