[SPARK-59229][CORE] Identify Apache Spark in the User-Agent for Ivy dependency resolution

[brianf]

What changes were proposed in this pull request?

Set the http.agent system property in MavenUtils so that Ivy identifies itself as Spark when resolving --packages / spark.jars.packages coordinates from remote repositories.

Requests now carry:

Apache-Spark/4.0.0 (Apache-Ivy/2.5.3)

instead of Ivy's default Apache Ivy/2.5.3.

The format follows RFC 9110 Section 10.1.5 (product *( RWS ( product / comment ) )) — a product token carrying Spark's identity and version, plus a comment carrying the underlying resolution engine, so operator tooling that already matches on the Ivy version keeps working.

Details:

• http.agent is Ivy's own designed override point — AbstractURLHandler.getUserAgent() returns System.getProperty("http.agent", "Apache Ivy/" + Ivy.getIvyVersion()), and both BasicURLHandler and HttpClientHandler inherit it. No new Ivy API or custom handler is required.
• The property is set once, if absent, rather than set-and-restored around the download. URLHandlerRegistry.getHttp() returns HttpClientHandler.DELETE_ON_EXIT_INSTANCE when httpclient is on the classpath (it is — httpclient 4.5.14 ships in the distribution), and that singleton bakes the User-Agent in at class-initialization time via HttpClients.custom().setUserAgent(...). A save/set/restore wrapper would therefore have no effect on requests and would race between concurrent resolutions.
• buildIvySettings() / loadIvySettings() are the insertion points: all three call paths (Artifact.scala, IsolatedClientLoader.scala, DependencyUtils.scala) go through one of them before resolveMavenCoordinates(), and neither touches HTTP itself. In loadIvySettings() it is the first statement, ahead of ivySettings.load(file), because XmlSettingsParser is the one reachable path that can trigger HttpClientHandler's static init (when the settings XML carries httpRequestMethod).
• SparkBuildInfo.spark_version is used rather than org.apache.spark.SPARK_VERSION, which lives in core and is not available from common/utils.

Why are the changes needed?

Repository operators rely on the User-Agent to attribute traffic and to reach the responsible party when a client misbehaves. A bare Apache Ivy/2.5.3 is un-attributable: it could be Spark, Ant, sbt, or a hand-rolled Ivy client. Operators cannot tell which tool to talk to, cannot distinguish an old Spark with poor caching behavior from a current one, and have no signal to route a conversation about excessive consumption.

This matters in practice because Spark clusters without a repository-manager proxy generate a burst of resolution traffic at every job launch, so Spark is a meaningful share of this traffic on public repositories.

Does this PR introduce any user-facing change?

Yes, in two respects, both additive:

1. Requests Spark makes to remote artifact repositories during dependency resolution now carry User-Agent: Apache-Spark/<version> (Apache-Ivy/<version>) instead of Apache Ivy/<version>. No resolution behavior changes.
2. Setting -Dhttp.agent=... explicitly continues to win — the property is only set when unset — and this is now documented under spark.jars.packages in docs/configuration.md.

Note that http.agent is a JVM-global property affecting any HttpURLConnection, not only Ivy. In practice the Spark-adjacent HTTP clients (Hadoop, AWS SDK, Kubernetes client) use their own stacks and set their own User-Agent, so the blast radius is small, and labelling stray JDK-level requests from a Spark driver as Spark is accurate anyway. The set-only-if-absent guard is the mitigation.

How was this patch tested?

New tests in MavenUtilsSuite:

• the User-Agent matches the RFC 9110 product RWS comment form, and is composed from the real SparkBuildInfo.spark_version and Ivy.getIvyVersion rather than a hardcoded literal;
• buildIvySettings sets the property when it is unset;
• loadIvySettings sets it too;
• an explicitly configured value is not overwritten.

The suite's beforeEach/afterEach were extended to save, clear, and restore http.agent, so the new global side effect cannot leak between tests or into other suites in the same JVM.

./build/mvn -pl common/utils -am test

→ 101 tests succeeded, 0 failed (MavenUtilsSuite: tests="17" failures="0" errors="0").

./dev/scalastyle → passed.

Additionally, verified on the wire rather than only by inspection, since the HttpClientHandler class-init timing is the part worth confirming empirically. A local HTTP server was pointed at as the repository and the request headers logged:

CONFIGURED_AGENT=Apache-Spark/5.0.0-SNAPSHOT (Apache-Ivy/2.5.3)
OBSERVED_USER_AGENT=Apache-Spark/5.0.0-SNAPSHOT (Apache-Ivy/2.5.3)

and with the override in place:

CONFIGURED_AGENT=AcmeCorp/1.0
OBSERVED_USER_AGENT=AcmeCorp/1.0

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)