HBASE-29226 Migrate to jetty 12 with EE8 and bump java servlet to 4.0.1 #6783

NihalJain · 2025-03-11T20:31:01Z

Please find a detailed summary of all changes as below:

Dependencies

Replaced hbase-shaded-jetty i.e. Jetty 9 with hbase-shaded-jetty-12-plus-core and hbase-shaded-jetty-12-plus-ee8 i.e. Jetty 12 EE8
Upgrade servlet.api.version to 4.0.1
Upgrade tomcat.jasper.version to 9.0.102

Code Adjustments

Modified import statements to use org.eclipse.jetty.ee8 etc. as per Jetty 12
Replace HandlerCollection with Handler.Sequence
- Refer https://jetty.org/docs/jetty/12/programming-guide/migration/11-to-12.html#api-changes-handler-sequence
Adjust request log handler.
- Refer https://jetty.org/docs/jetty/12/programming-guide/migration/11-to-12.html#api-changes-handler-requestlog
Adjust error handlers' set logic based on available methods
Remove usage of deprecated method setResourceBase() with setBaseResourceAsPath(). Also, passing logDir path in canonical format as otherwise, we get 404 in case we have \..\ in static path.
- Refer Discrepancy between Jetty 9 and Jetty 12 when setting the base resource to a path containing .. jetty/jetty.project#12958
Remove MultiException with ExceptionUtil.MultiException in HttpServer.java.
Modify assignment: webServer.getHandlers() with new return type: Array to List
Replaced Constraint with ServletConstraint
Added LogLevelExceptionUtils.java to handle HTML error response for log levels as connection.getResponseMessage() now returns error code as string in message and hence causes test failure otherwise
Allow javax as xsd files at javax/servlet/resources come via org.apache.hbase.thirdparty:hbase-shaded-jetty-12-ee8

Test Adjustments

Update checkBindAddress() to call server.start() else testBindAddress() fails with NPE. Verified doing same without Jetty 12 works as well.
Replace JSON.parse() with new JSON().fromJSON() and JSON.toString() with new JSON().toJSON()
Update app name from ".." to "" as it fails with IllegalArgumentException otherwise
Update MockHttpApiRule where we replace AbstractHandler with Handler.Abstract and adjust code based on new interface.
- Refer https://jetty.org/docs/jetty/12/programming-guide/migration/11-to-12.html#handler-request-content-apis
Deleted webdefaults.xml. Need to see if still needed.
Added necessary compliance rules for URI validation, required by tests and possibly users as well. Add note for what fails if these rules are not there in tests.
- Refer Jetty 12: 400: Ambiguous URI path encoding for path <%=FOO%>~1 (encoded: %3C%25%3DFOO%25%3E%7E1) jetty/jetty.project#11890 (comment)

References

Umbrealla at HBASE-29224
Discussion at https://lists.apache.org/thread/bkrfm705kqd3bqzyvo7jv46t6p64x2n5
Dependent thirdparty change at HBASE-29225 Add module for Jetty 12 with EE8 to hbase-thirdparty hbase-thirdparty#131

hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java

hbase-it/src/test/java/org/apache/hadoop/hbase/MockHttpApiRule.java

hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/NamespacesResource.java

...shaded/hbase-shaded-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh

NihalJain · 2025-03-11T20:42:54Z

pom.xml

@@ -863,9 +863,9 @@
    <jackson.version>2.17.2</jackson.version>
    <jackson.databind.version>2.17.2</jackson.databind.version>
    <jaxb-api.version>2.3.1</jaxb-api.version>
-    <servlet.api.version>3.1.0</servlet.api.version>
+    <servlet.api.version>4.0.1</servlet.api.version>


just a note: hbase worked with 3.1.0 as well but as per https://stackoverflow.com/questions/77007560/missing-jetty-servlet-12-0-0-dependency we should be on 4.x

pom.xml

hbase-it/src/test/java/org/apache/hadoop/hbase/MockHttpApiRule.java

NihalJain · 2025-03-27T12:36:37Z

hbase-http/src/main/java/org/apache/hadoop/hbase/http/log/LogLevel.java

@@ -267,7 +266,11 @@ private void process(String urlString) throws Exception {

      HttpURLConnection connection = connect(url);

-      HttpExceptionUtils.validateResponse(connection, 200);
+      // We implement the validateResponse method inside hbase to handle for HTML response.


added note for reviewers' ease, let me know if should drop before merge

also previous method was not useful, the error is HTML but it parses JSON

hbase-http/src/test/java/org/apache/hadoop/hbase/http/log/TestLogLevel.java

hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java

hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestGetAndPutResource.java

hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestSecureRESTServer.java

NihalJain · 2025-03-27T13:29:46Z

...il/src/main/resources/org/apache/hadoop/hbase/shaded/org/eclipse/jetty/webapp/webdefault.xml

-<!--
-Copyright (C) Jetty Authors
-
-Licensed under the Apache License, Version 2.0 (the "License");


Not sure why we even had this file in our codebase. Tried deleting to find regression, all UTs ran fine locally. Anyone could help me with how to validate if this is even needed?

This should be coming from the ee support in jetty.
Though I believe we're not using the web.xml server config at all in HBase.

stoty · 2025-05-09T11:29:29Z

What's the status here ?
Looking at the latest CI run, I only see a single unrealted test failure.

NihalJain · 2025-05-09T11:40:28Z

What's the status here ?

Looking at the latest CI run, I only see a single unrealted test failure.

I have verified all major flows in a separate jira. Could not get any more reviews here. We need to merge thirdparty change, release that, consume here and merge this.

NihalJain · 2025-05-09T11:40:54Z

Verified the changes and updated at https://issues.apache.org/jira/browse/HBASE-29227

Copy pasting here..

Summary of Jetty 9 to Jetty 12 with EE8 Upgrade Testing for HBase

Environment Setup

Task: Set up a distributed HBase environment with Kerberos and SSL enabled (built with PR #6783 and commit 634c6820754ecb3d976c5360e9afcf3c0b42873d)

Status: Completed

Components Tested: Master, RegionServer, REST, Thrift

Tests Conducted

1. Basic Sanity Check

Task: Verify basic HBase CRUD and admin operations via shell.

Status: Passed

2. User Interfaces (UIs)

Master UI with SSL: Passed

RegionServer UI with SSL: Passed

REST Server UI with SSL: Passed

Thrift Server UI with SSL: Passed

Note: Verified almost all web pages except for profiler servlet.

3. REST Server

Task: Perform basic CRUD operations with SSL.

Status: Passed

4. Thrift Server

Task: Run Thrift demo client.

Status: Passed

5. LDAP Setup in Web UI

Task: Test LDAP-based UI login.

Status: Passed

6. Misc UI Security Settings

Task: Verify the following security configurations:

hbase.master.ui.readonly=false

hbase.http.filter.no-store.enable=true

hbase.security.authentication.ui.config.protected=true

hadoop.security.instrumentation.requires.admin=true

etc.

Status: Passed

Please let me know if any further testing is needed.

CC: @stoty @Apache9 @ndimiduk @apurtell

FYR

NihalJain · 2025-05-09T13:12:44Z

With jetty 12.0.20 we have fixed jetty/jetty.project#12958 i.e. "Discrepancy between Jetty 9 and Jetty 12 when setting the base resource to a path containing .." Hence changes for handling log directory has been reverted. Lets see if tests go well with latest hbase-thirdparty

NihalJain · 2025-05-15T19:47:34Z

Retriggered to consume latest hbase-thirdparty snapshot at apache/hbase-thirdparty#137

… add details on why we need violations

…r LEGACY disuss

… - Discrepancy between Jetty 9 and Jetty 12 when setting the base resource to a path containing ..

Apache-HBase · 2025-05-15T22:41:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 28s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 16s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 10s		master passed
+1 💚	compile	7m 51s		master passed
+1 💚	checkstyle	1m 10s		master passed
+1 💚	spotbugs	11m 11s		master passed
+1 💚	spotless	0m 44s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 40s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 16s		the patch passed
+1 💚	compile	10m 26s		the patch passed
+1 💚	javac	10m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 21s	/results-checkstyle-root.txt	root: The patch generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10)
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	xmllint	0m 1s		No new issues.
+1 💚	spotbugs	15m 45s		the patch passed
+1 💚	hadoopcheck	18m 13s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	1m 4s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	1m 39s		The patch does not generate ASF License warnings.
		89m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6783/20/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6783
Optional Tests	dupname asflicense javac codespell detsecrets xmllint hadoopcheck spotless compile spotbugs checkstyle hbaseanti shellcheck shelldocs
uname	Linux caa62588d20b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `27d2c17`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	188 (vs. ulimit of 30000)
modules	C: hbase-http hbase-server hbase-thrift hbase-it hbase-rest hbase-shaded/hbase-shaded-testing-util hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6783/20/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3 shellcheck=0.8.0 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-05-16T02:34:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 28s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 15s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 12s		master passed
+1 💚	compile	2m 9s		master passed
+1 💚	javadoc	3m 52s		master passed
+1 💚	shadedjars	5m 55s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 41s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 5s		the patch passed
+1 💚	compile	2m 12s		the patch passed
+1 💚	javac	2m 12s		the patch passed
+1 💚	javadoc	4m 49s		the patch passed
+1 💚	shadedjars	6m 6s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	282m 32s		root in the patch passed.
		322m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6783/20/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6783
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 0cf3a875fad1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `27d2c17`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6783/20/testReport/
Max. process+thread count	8314 (vs. ulimit of 30000)
modules	C: hbase-http hbase-server hbase-thrift hbase-it hbase-rest hbase-shaded/hbase-shaded-testing-util hbase-shaded/hbase-shaded-check-invariants hbase-shaded/hbase-shaded-with-hadoop-check-invariants . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6783/20/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

NihalJain marked this pull request as draft March 11, 2025 20:31

This comment was marked as duplicate.

Sign in to view

NihalJain commented Mar 11, 2025

View reviewed changes

hbase-http/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java Outdated Show resolved Hide resolved

NihalJain commented Mar 11, 2025

View reviewed changes

hbase-it/src/test/java/org/apache/hadoop/hbase/MockHttpApiRule.java Outdated Show resolved Hide resolved

NihalJain commented Mar 11, 2025

View reviewed changes

hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/NamespacesResource.java Outdated Show resolved Hide resolved

NihalJain commented Mar 11, 2025

View reviewed changes

...shaded/hbase-shaded-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh Show resolved Hide resolved

NihalJain commented Mar 11, 2025

View reviewed changes

pom.xml Show resolved Hide resolved

NihalJain commented Mar 11, 2025

View reviewed changes

hbase-it/src/test/java/org/apache/hadoop/hbase/MockHttpApiRule.java Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

NihalJain mentioned this pull request Mar 11, 2025

Issue #12828 support relative uris for Response.encodeURI jetty/jetty.project#12833

Merged

NihalJain changed the title ~~[WIP] Migrate to jetty 12 with EE8 and bump java servlet to 4.0.1~~ HBASE-29226 Migrate to jetty 12 with EE8 and bump java servlet to 4.0.1 Mar 27, 2025

NihalJain commented Mar 27, 2025

View reviewed changes

hbase-http/src/test/java/org/apache/hadoop/hbase/http/log/TestLogLevel.java Show resolved Hide resolved

NihalJain commented Mar 27, 2025

View reviewed changes

hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java Outdated Show resolved Hide resolved

NihalJain commented Mar 27, 2025

View reviewed changes

hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestGetAndPutResource.java Show resolved Hide resolved

NihalJain commented Mar 27, 2025

View reviewed changes

hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestSecureRESTServer.java Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

NihalJain marked this pull request as ready for review March 27, 2025 12:56

This comment has been minimized.

Sign in to view

NihalJain commented Mar 27, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

NihalJain mentioned this pull request Mar 28, 2025

HBASE-29225 Add module for Jetty 12 with EE8 to hbase-thirdparty apache/hbase-thirdparty#131

Merged

NihalJain requested review from apurtell, ndimiduk and stoty March 28, 2025 16:38

This comment has been minimized.

Sign in to view

NihalJain force-pushed the jetty12 branch from 634c682 to 2c2fc35 Compare May 9, 2025 13:06

This comment has been minimized.

Sign in to view

NihalJain added 14 commits May 16, 2025 02:32

Rebase

2c3e97e

Make changes to fix UTs, fix /logs page not loading issue etc.

08c7a3e

Fix spotless

ba3f9eb

Make changes to test EE8 + EE9 in single module

edaa461

Update as per thirdparty changes post review

4c2bac9

Fix type and use ExceptionUtil.MultiException and revert to old code

c2de9e2

ee9 -> ee9+

8acad71

Fix TODOs, update log dir handling based on jetty project's feedback,…

1e0ee0e

… add details on why we need violations

Update comments, run spotless

572f945

Add snapshot repo

a5efe9b

Set update policy to always to force update

606fbb5

Fix spotbugs and spotless issues

62ccdc9

Remove check to optionally enable full compliance, and remove todo fo…

691275e

…r LEGACY disuss

Bump to jetty 12.0.20 which brings in fix for jetty/jetty.project#12958…

27d2c17

… - Discrepancy between Jetty 9 and Jetty 12 when setting the base resource to a path containing ..

NihalJain force-pushed the jetty12 branch from 52d9f49 to 27d2c17 Compare May 15, 2025 21:04

HBASE-29226 Migrate to jetty 12 with EE8 and bump java servlet to 4.0.1 #6783

Are you sure you want to change the base?

HBASE-29226 Migrate to jetty 12 with EE8 and bump java servlet to 4.0.1 #6783

Conversation

NihalJain commented Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as duplicate.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NihalJain Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

NihalJain Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

NihalJain Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

NihalJain Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

stoty Mar 29, 2025

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

stoty commented May 9, 2025

Uh oh!

NihalJain commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NihalJain commented May 9, 2025

Summary of Jetty 9 to Jetty 12 with EE8 Upgrade Testing for HBase

Environment Setup

Tests Conducted

1. Basic Sanity Check

2. User Interfaces (UIs)

3. REST Server

4. Thrift Server

5. LDAP Setup in Web UI

6. Misc UI Security Settings

Uh oh!

NihalJain commented May 9, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

NihalJain commented May 15, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented May 15, 2025

Uh oh!

Apache-HBase commented May 16, 2025

Uh oh!

Uh oh!

NihalJain commented Mar 11, 2025 •

edited

Loading

NihalJain Mar 11, 2025 •

edited

Loading

NihalJain commented May 9, 2025 •

edited

Loading