HADOOP-19447. Add Caching Mechanism to HostResolver to Avoid Redundant Hostname Resolutions by yangjiandan · Pull Request #7527 · apache/hadoop

yangjiandan · 2025-03-21T08:28:31Z

Description of PR

Currently, the two implementations of org.apache.hadoop.security.SecurityUtil.HostResolver, StandardHostResolver and QualifiedHostResolver in Hadoop performs hostname resolution each time it is called. Each heartbeat between the AM and RM causes the RM to invoke the HostResolver#getByName method once. In large-scale clusters running numerous applications, this results in a high frequency of redundant hostname resolutions.

see HADOOP-19447

How was this patch tested?

current UT and newly developed UT

…t Hostname Resolutions

hadoop-yetus · 2025-03-21T11:55:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 20s		trunk passed
+1 💚	compile	16m 51s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	14m 39s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 22s		trunk passed
+1 💚	mvnsite	1m 42s		trunk passed
+1 💚	javadoc	1m 15s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 48s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 36s		trunk passed
+1 💚	shadedclient	37m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 56s		the patch passed
+1 💚	compile	14m 49s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	14m 49s		the patch passed
+1 💚	compile	13m 30s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	13m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 14s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 9 new + 104 unchanged - 0 fixed = 113 total (was 104)
+1 💚	mvnsite	1m 37s		the patch passed
+1 💚	javadoc	1m 11s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 50s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
-1 ❌	spotbugs	2m 42s	/new-spotbugs-hadoop-common-project_hadoop-common.html	hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	36m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	14m 50s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 3s		The patch does not generate ASF License warnings.
		205m 39s

Reason	Tests
SpotBugs	module:hadoop-common-project/hadoop-common
	Should org.apache.hadoop.security.SecurityUtil$StandardHostResolver$StandardLoader be a static inner class? At SecurityUtil.java:inner class? At SecurityUtil.java:[lines 636-639]

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/1/artifact/out/Dockerfile
GITHUB PR	#7527
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 7f4731c98efc 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `7e921a1`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/1/testReport/
Max. process+thread count	1261 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-03-22T03:14:08Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 10s		trunk passed
+1 💚	compile	17m 28s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	15m 15s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 18s		trunk passed
+1 💚	mvnsite	1m 37s		trunk passed
+1 💚	javadoc	1m 16s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 50s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 36s		trunk passed
+1 💚	shadedclient	41m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 55s		the patch passed
+1 💚	compile	16m 48s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	16m 48s		the patch passed
+1 💚	compile	14m 56s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	14m 56s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 15s		the patch passed
+1 💚	mvnsite	1m 35s		the patch passed
+1 💚	javadoc	1m 11s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 51s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 39s		the patch passed
+1 💚	shadedclient	41m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	14m 57s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 1s		The patch does not generate ASF License warnings.
		223m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/2/artifact/out/Dockerfile
GITHUB PR	#7527
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux d21588684b10 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f5a5c73`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/2/testReport/
Max. process+thread count	2851 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

yangjiandan · 2025-03-23T23:34:01Z

@slfan1989 @TaoYang526 Could you help review this PR?

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExpiringCache.java

hadoop-yetus · 2025-03-24T10:41:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 49s		trunk passed
+1 💚	compile	17m 36s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	14m 58s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 19s		trunk passed
+1 💚	mvnsite	1m 38s		trunk passed
+1 💚	javadoc	1m 14s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 51s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 34s		trunk passed
+1 💚	shadedclient	42m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 24s	/patch-mvninstall-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	compile	0m 42s	/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt	root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌	javac	0m 42s	/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt	root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌	compile	0m 52s	/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.txt	root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.
-1 ❌	javac	0m 52s	/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.txt	root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 59s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 3 new + 104 unchanged - 0 fixed = 107 total (was 104)
-1 ❌	mvnsite	0m 28s	/patch-mvnsite-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	javadoc	0m 29s	/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt	hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌	javadoc	0m 25s	/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.txt	hadoop-common in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06.
-1 ❌	spotbugs	0m 25s	/patch-spotbugs-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	shadedclient	6m 34s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 24s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		132m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/3/artifact/out/Dockerfile
GITHUB PR	#7527
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 1d3f7cd3544c 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `acbe616`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/3/testReport/
Max. process+thread count	524 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-03-24T14:55:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 34s		trunk passed
+1 💚	compile	17m 20s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	14m 52s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 15s		trunk passed
+1 💚	mvnsite	1m 37s		trunk passed
+1 💚	javadoc	1m 15s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 50s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 32s		trunk passed
+1 💚	shadedclient	40m 41s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 55s		the patch passed
+1 💚	compile	16m 48s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	16m 48s		the patch passed
+1 💚	compile	14m 59s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	14m 59s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 17s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 3 new + 104 unchanged - 0 fixed = 107 total (was 104)
+1 💚	mvnsite	1m 36s		the patch passed
+1 💚	javadoc	1m 11s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 51s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 40s		the patch passed
+1 💚	shadedclient	41m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	14m 53s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 3s		The patch does not generate ASF License warnings.
		220m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/4/artifact/out/Dockerfile
GITHUB PR	#7527
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux edd5341887f0 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `bfda60d`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/4/testReport/
Max. process+thread count	3134 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-03-25T12:34:53Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 29s		trunk passed
+1 💚	compile	17m 48s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	15m 17s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 19s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 15s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 51s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 35s		trunk passed
+1 💚	shadedclient	42m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 54s		the patch passed
+1 💚	compile	16m 54s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	16m 54s		the patch passed
+1 💚	compile	14m 54s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	14m 54s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 12s		the patch passed
+1 💚	mvnsite	1m 34s		the patch passed
+1 💚	javadoc	1m 11s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 51s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 42s		the patch passed
+1 💚	shadedclient	41m 33s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	14m 56s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		222m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/5/artifact/out/Dockerfile
GITHUB PR	#7527
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 473799cde8f3 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `7b1732e`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/5/testReport/
Max. process+thread count	1433 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

yangjiandan · 2025-03-26T00:49:24Z

@slfan1989 I have updated PR based on your suggestion, and using guava‘s LoadingCache significantly simplifys the code and improve readability.
Please take another look when you have time. Let me know if there is anything else that needs to be addressed.

...common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java

hadoop-yetus · 2025-03-26T09:05:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 52s		trunk passed
+1 💚	compile	18m 29s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	15m 23s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 14s		trunk passed
+1 💚	mvnsite	1m 37s		trunk passed
+1 💚	javadoc	1m 13s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 47s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 45s		trunk passed
+1 💚	shadedclient	41m 35s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 0s		the patch passed
+1 💚	compile	17m 56s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	17m 56s		the patch passed
+1 💚	compile	15m 45s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	15m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 10s		the patch passed
+1 💚	mvnsite	1m 34s		the patch passed
+1 💚	javadoc	1m 9s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 47s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 40s		the patch passed
+1 💚	shadedclient	42m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 21s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 3s		The patch does not generate ASF License warnings.
		226m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/6/artifact/out/Dockerfile
GITHUB PR	#7527
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 2c0c37577b74 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2c30103`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/6/testReport/
Max. process+thread count	1253 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2025-03-26T15:36:34Z

@yangjiandan Thank you for your contribution. I will review this PR tomorrow.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java

hadoop-yetus · 2025-03-27T09:58:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 58s		trunk passed
+1 💚	compile	17m 18s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	15m 0s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 17s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 15s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 51s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 35s		trunk passed
+1 💚	shadedclient	40m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 55s		the patch passed
+1 💚	compile	16m 39s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	16m 39s		the patch passed
+1 💚	compile	14m 51s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	14m 51s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 14s		the patch passed
+1 💚	mvnsite	1m 36s		the patch passed
+1 💚	javadoc	1m 11s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 51s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 44s		the patch passed
+1 💚	shadedclient	41m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	14m 53s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		223m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/7/artifact/out/Dockerfile
GITHUB PR	#7527
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 43cfbce791a3 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2da7127`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/7/testReport/
Max. process+thread count	1247 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

yangjiandan · 2025-03-27T11:10:49Z

@slfan1989 @TaoYang526 Thanks again for your feedback！
I've update PR based on all the comment and suggestions provided.
When you have a chance, could you help take another look.
Appreciate your time!

TaoYang526 · 2025-03-27T13:58:24Z

+1 LGTM
@yangjiandan Thanks for your contribution!

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java

...common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java

…fault.xml

hadoop-yetus · 2025-03-31T10:48:45Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	19m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 17s		trunk passed
+1 💚	compile	17m 35s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	15m 26s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 19s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 13s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 52s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 34s		trunk passed
+1 💚	shadedclient	40m 55s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 55s		the patch passed
+1 💚	compile	16m 54s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	16m 54s		the patch passed
+1 💚	compile	15m 32s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	15m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 15s		the patch passed
+1 💚	mvnsite	1m 37s		the patch passed
+1 💚	javadoc	1m 10s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 50s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	2m 42s		the patch passed
+1 💚	shadedclient	41m 25s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	14m 56s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		241m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/8/artifact/out/Dockerfile
GITHUB PR	#7527
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 5558fb978661 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5c91a22`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/8/testReport/
Max. process+thread count	2933 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7527/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

yangjiandan · 2025-03-31T10:55:46Z

Hi, @slfan1989, just checking in.
What do you think about my update based on your code comment?
Do you have any further suggestion or feedback?

slfan1989 · 2025-04-01T01:37:53Z

@yangjiandan Thanks for the contribution! LGTM.

yangjiandan · 2025-04-01T05:52:47Z

Hi, @slfan1989 and @TaoYang526 Thanks a lot for your review and the +1!

slfan1989 · 2025-04-02T01:43:51Z

Hi, @slfan1989 and @TaoYang526 Thanks a lot for your review and the +1!

@TaoYang526 Do you have any other suggestions?

TaoYang526 · 2025-04-02T06:56:19Z

@slfan1989 No more suggestions from my side.
+1 for the latest PR. Nice work!

yangjiandan · 2025-04-03T09:21:47Z

Hi, @slfan1989 @TaoYang526,
Thanks for your approvals!
Since both of you have approved the PR, could you help merge it into master branch when convenient?

slfan1989 · 2025-04-07T02:24:04Z

@yangjiandan Thanks for the contribution! @TaoYang526 Thanks for the review!

HADOOP-19447. Add Caching Mechanism to HostResolver to Avoid Redundan…

7e921a1

…t Hostname Resolutions

github-actions bot added trunk Common labels Mar 21, 2025

fix findbug and checkstyle warn

f5a5c73

slfan1989 reviewed Mar 24, 2025

View reviewed changes

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExpiringCache.java Outdated Show resolved Hide resolved

use Guava's LoadingCache instead of adding custom ExpiringCache

acbe616

use guava lib in org.apache.hadoop.thirdparty

bfda60d

fix checkstyle warning

7b1732e

TaoYang526 reviewed Mar 26, 2025

View reviewed changes

...common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java Outdated Show resolved Hide resolved

TaoYang526 reviewed Mar 26, 2025

View reviewed changes

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java Outdated Show resolved Hide resolved

1. rename key of cache expire interval; 2. fix possible npe

2c30103

slfan1989 reviewed Mar 27, 2025

View reviewed changes

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java Outdated Show resolved Hide resolved

replace getInt with getTimeDuration

2da7127

slfan1989 reviewed Mar 31, 2025

View reviewed changes

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java Show resolved Hide resolved

slfan1989 reviewed Mar 31, 2025

View reviewed changes

...common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java Show resolved Hide resolved

add hadoop.security.hostname.cache.expire-interval.seconds in core-de…

5c91a22

…fault.xml

slfan1989 approved these changes Apr 1, 2025

View reviewed changes

TaoYang526 approved these changes Apr 3, 2025

View reviewed changes

slfan1989 merged commit d19b35a into apache:trunk Apr 7, 2025
4 checks passed

Comments

Conversation

yangjiandan commented Mar 21, 2025

Description of PR

How was this patch tested?

Uh oh!

hadoop-yetus commented Mar 21, 2025

Uh oh!

hadoop-yetus commented Mar 22, 2025

Uh oh!

yangjiandan commented Mar 23, 2025

Uh oh!

Uh oh!

hadoop-yetus commented Mar 24, 2025

Uh oh!

hadoop-yetus commented Mar 24, 2025

Uh oh!

hadoop-yetus commented Mar 25, 2025

Uh oh!

yangjiandan commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Mar 26, 2025

Uh oh!

slfan1989 commented Mar 26, 2025

Uh oh!

Uh oh!

hadoop-yetus commented Mar 27, 2025

Uh oh!

yangjiandan commented Mar 27, 2025

Uh oh!

TaoYang526 commented Mar 27, 2025

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Mar 31, 2025

Uh oh!

yangjiandan commented Mar 31, 2025

Uh oh!

slfan1989 commented Apr 1, 2025

Uh oh!

yangjiandan commented Apr 1, 2025

Uh oh!

slfan1989 commented Apr 2, 2025

Uh oh!

TaoYang526 commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yangjiandan commented Apr 3, 2025

Uh oh!

Uh oh!

slfan1989 commented Apr 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yangjiandan commented Mar 26, 2025 •

edited

Loading

TaoYang526 commented Apr 2, 2025 •

edited

Loading