HADOOP-19848: AzureNativeFileSystemStore fails with NoSuchMethodError on Jetty 9.4.57+

[surendralilhore]

Description of PR

When using hadoop-azure with Apache Spark 4.0, AzureNativeFileSystemStore fails to load with:

 
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoSuchMethodError: 'java.util.Properties org.eclipse.jetty.util.log.Log.getProperties()' [in thread "Thread-31"]

 Root cause:

 AzureNativeFileSystemStore.createPermissionJsonSerializer() calls org.eclipse.jetty.util.log.Log.getProperties() to suppress Jetty's log announcement. While this method exists in jetty-util 9.4.57 (as shipped with Hadoop 3.4.x), Spark 4.0 bundles a different version of Jetty on the runtime classpath where this method has been removed. This causes a compile-vs-runtime classpath mismatch — the code compiles fine against Hadoop's Jetty version but fails at runtime under Spark's classloader.

  This is a fragile dependency on a Jetty internal API (org.eclipse.jetty.util.log.Log) that is unnecessary. Jetty reads the org.eclipse.jetty.util.log.announce property from system properties, so System.setProperty() achieves the same effect without coupling to Jetty internals.

How was this patch tested?

Tested with Spark 4.0 Job. After this change Azure filesytem initialization is successful

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? -- Yes
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? --Yes
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? -- No
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files? -- No

AI Tooling

If an AI tool was used: -- No

• The PR includes the phrase "Contains content generated by "
  where is the name of the AI tool used.
• My use of AI contributions follows the ASF legal policy
  https://www.apache.org/legal/generative-tooling.html

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	20m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ branch-3.4 Compile Tests _
+1 💚	mvninstall	52m 11s		branch-3.4 passed
+1 💚	compile	0m 57s		branch-3.4 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 51s		branch-3.4 passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	checkstyle	0m 45s		branch-3.4 passed
+1 💚	mvnsite	0m 54s		branch-3.4 passed
+1 💚	javadoc	0m 53s		branch-3.4 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 46s		branch-3.4 passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	spotbugs	1m 22s		branch-3.4 passed
+1 💚	shadedclient	38m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 35s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 34s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	javac	0m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	mvnsite	0m 32s		the patch passed
+1 💚	javadoc	0m 29s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
+1 💚	spotbugs	1m 6s		the patch passed
+1 💚	shadedclient	37m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 26s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		163m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8365/1/artifact/out/Dockerfile
GITHUB PR	#8365
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 71e6483a3ab8 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / fe6d300
Default Java	Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-gaus1-0ubuntu120.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8365/1/testReport/
Max. process+thread count	534 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8365/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[ayushtkn]

I was under impression that Spark uses the shaded jars: https://github.com/apache/spark/blob/a5a758846c97c564b4d58c40b013ad8dbed5f9ff/pom.xml#L1475-L1479
which has jetty-util shaded.

Maybe not for cloud connectors, but not shading such dependencies looks like a bigger problem, like most of these thirparty-libs aren't backward compatible, it would break some other day for some other lib.

@ajfabbri / @pan3793 any pointers here?

I don't have any better suggestion around the code change

[pan3793]

Sorry, I don't have experience with hadoop-azure, and AFAIK, Spark does not have integration test coverage for it.

I suppose hadoop cloud storage connectors should only depend on hadoop-client-api, maybe with a few 3rd party common libs, e.g., apache-http-client, jackson, which usually provides quite stable APIs, I don't understand why hadoop-azure uses Jetty's JSON and Log, could it be replaced with Jackson?

[anujmodi2021]

+1 for the change here.
But this won't make it to the release as 3.4.x releases are over and 3.5.0 is going to be the next official release.
3.5.0 does not have WASB code anymore so this change is irrelevant there.

[surendralilhore]

@anujmodi2021, @ayushtkn

I think we should still commit this change, even if we are not planning a release for 3.4, since this code is no longer part of the active codebase in 3.5 and trunk.

Spark master is still using 3.4.3, so some users may encounter this issue. Keeping this change will serve as a useful reference for others.

https://github.com/apache/spark/blob/0419e68b7e7cdda443a44efa3abe251595af3e0a/pom.xml#L130

[pan3793]

Spark master is still using 3.4.3

it's true as of today, but Spark is likely to move to hadoop 3.5.0 once it's available.

[pan3793]

Additional info: Spark 4.0 and 4.1 use Jetty 11, and master uses Jetty 12. Since it still calls Jetty's JSON even after this change, have you checked it thoroughly?

[surendralilhore]

@pan3793 , Yes, I've verified this. The org.eclipse.jetty.util.ajax.JSON class and the APIs used here (new JSON(), addConvertor(), JSON.Convertor interface, toJSON(), fromJSON()) are fully stable across Jetty 9, 11, and 12 — confirmed via Jetty 12 javadoc.