[SPARK-59181][NETWORK] Use system default trust managers in SSLFactory when truststore is missing

[RinZ27]

(Replacing stale PR #56089 because GitHub blocked reopening it after branch updates)

What changes were proposed in this pull request?

Refactored SSLFactory.java to replace the insecure credulousTrustStoreManagers fallback with the JVM's default TrustManagerFactory. When a truststore is not explicitly provided, the system now correctly utilizes the standard system trust managers instead of blindly trusting all certificates.

Why are the changes needed?

The previous implementation defaulted to a "trust-all" behavior if a truststore was missing or misconfigured. This silent security failure left internal Spark networking (RPC, Shuffle) vulnerable to Man-in-the-Middle (MITM) attacks. Aligning with standard Java security practices ensures robust certificate validation is maintained by default.

Does this PR introduce any user-facing change?

Yes. Operators relying on self-signed or internal-CA certificates without explicitly configuring a truststore will now see RPC TLS connections fail. They must either explicitly configure a truststore (spark.ssl.trustStore) or add their internal CA to the JVM's default trust store.

How was this patch tested?

Added a new unit test case testBuildWithoutTrustStoreUsesSystemDefault in SSLFactorySuite.java to verify successful engine initialization without a specific truststore. Verified all existing tests in SSLFactorySuite pass as expected.

Was this patch authored or co-authored using generative AI tooling?

No.

[tdcmeehan]

This is a user-facing change: operators relying on self-signed or internal-CA certificates without an explicit truststore may see RPC TLS connections fail, including in pre-production environments. Please update docs/security.md with the new behavior and migration guidance.

Given that behavioral impact, this should not be tagged [MINOR]. Let’s file a Spark JIRA under NETWORK and update the PR title accordingly.

[RinZ27]

Thanks for the review @tdcmeehan! I have updated docs/security.md to explicitly document the behavior change and provide migration guidance. I also updated the PR description to mark this as a user-facing change.

Could you (or another committer) please help file a JIRA issue under the NETWORK component? I don't have Jira permissions. Once the JIRA is created and the ID is shared, I will immediately update this PR's title to include it.

[tdcmeehan]

@RinZ27 I created https://issues.apache.org/jira/browse/SPARK-59181

[RinZ27]

@tdcmeehan Appreciated — just updated the PR title to reference SPARK-59181.