Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YARN-11796. Accept extra JDK 17 options in tests #7512

Merged
merged 2 commits into from
Mar 26, 2025

Conversation

stoty
Copy link
Contributor

@stoty stoty commented Mar 16, 2025

Description of PR

JIRA: YARN-11796. Accept extra JDK 17 options in tests

Accept options automatically added when run on JDK17+ in relevant tests

How was this patch tested?

Ran the modified tests with JDK8 and JDK17

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 11m 48s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 35m 48s trunk passed
+1 💚 compile 1m 25s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 1m 25s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 0m 38s trunk passed
+1 💚 mvnsite 0m 46s trunk passed
+1 💚 javadoc 0m 47s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 40s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 1m 25s trunk passed
+1 💚 shadedclient 34m 20s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 34s the patch passed
+1 💚 compile 1m 16s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 1m 16s the patch passed
+1 💚 compile 1m 13s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 1m 13s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 27s the patch passed
+1 💚 mvnsite 0m 36s the patch passed
+1 💚 javadoc 0m 34s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 31s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 1m 23s the patch passed
+1 💚 shadedclient 34m 16s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 27m 33s hadoop-yarn-server-nodemanager in the patch passed.
+1 💚 asflicense 0m 38s The patch does not generate ASF License warnings.
158m 52s
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7512/1/artifact/out/Dockerfile
GITHUB PR #7512
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 2abca4cb1401 5.15.0-134-generic #145-Ubuntu SMP Wed Feb 12 20:08:39 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e42fef9
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7512/1/testReport/
Max. process+thread count 737 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7512/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@slfan1989
Copy link
Contributor

@pan3793 Could you review this PR? Thank you very much! I remember you made similar changes, but I'm not sure if the changes in this PR are necessary.


if (Shell.isJavaVersionAtLeast(17)) {
assertTrue(result.remove("--add-exports=java.base/sun.net.dns=ALL-UNNAMED"));
assertTrue(result.remove("--add-exports=java.base/sun.net.util=ALL-UNNAMED"));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAPREDUCE-7456 defines

  private static final String ADDITIONAL_JDK17_PLUS_OPTIONS =
       "--add-opens=java.base/java.lang=ALL-UNNAMED " +
       "--add-exports=java.base/sun.net.dns=ALL-UNNAMED " +
       "--add-exports=java.base/sun.net.util=ALL-UNNAMED";

Why should only the last two be removed?

BTW, could you please leave a few comments that at least mention MAPREDUCE-7456 ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @pan3793 .

JDK17 options are defined at several locations.
Other than the one you mentioned, they are also defined in org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer which only adds
these two, and apparently these tests trigger that one.

If I added all three, the tests would fail, as --add-opens=java.base/java.lang=ALL-UNNAMED is never added for those test cases.

This also bothers me, I have opened HADOOP-19505 earlier to track this. (Though I have no immediate plans to work on that)

Copy link
Contributor Author

@stoty stoty Mar 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As for comments, most tests in TestContainerLocalizer already have similar ifs for JDK17 (I guess these were just added later, and never tested with JDK17).
Would you like me to add comments to those as well ?

Copy link
Member

@pan3793 pan3793 Mar 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you like me to add comments to those as well ?

Yes, please do that, I think the brief comments should help future explorers easy to understand the test code for this case

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 14m 12s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 35m 45s trunk passed
+1 💚 compile 1m 26s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 1m 22s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 0m 39s trunk passed
+1 💚 mvnsite 0m 45s trunk passed
+1 💚 javadoc 0m 48s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 40s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 1m 24s trunk passed
+1 💚 shadedclient 34m 22s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 34s the patch passed
+1 💚 compile 1m 15s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 1m 15s the patch passed
+1 💚 compile 1m 13s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 1m 13s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 27s the patch passed
+1 💚 mvnsite 0m 35s the patch passed
+1 💚 javadoc 0m 33s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 0m 31s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 1m 23s the patch passed
+1 💚 shadedclient 34m 15s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 27m 38s hadoop-yarn-server-nodemanager in the patch passed.
+1 💚 asflicense 0m 36s The patch does not generate ASF License warnings.
161m 12s
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7512/2/artifact/out/Dockerfile
GITHUB PR #7512
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux af7af42cb2cf 5.15.0-134-generic #145-Ubuntu SMP Wed Feb 12 20:08:39 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 697d5eb
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7512/2/testReport/
Max. process+thread count 554 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7512/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@stoty
Copy link
Contributor Author

stoty commented Mar 26, 2025

@slfan1989 Could you take another look, and commit if you are satisfied ?

@slfan1989
Copy link
Contributor

@slfan1989 Could you take another look, and commit if you are satisfied ?

@stoty Thanks for the contribution! LGTM.

@slfan1989 slfan1989 merged commit 2694071 into apache:trunk Mar 26, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants