Tomcat request count only considers requests handeled by the default Servlet

[toKrause]

General information

CheckMK provides out-of-the-box-support for monitoring a Tomcat Application Server. This includes a metric for a request count.

• Tomcat is a Java web application server, aka. Java Servlet Container.
• As such, Tomcat implements the Servlet API and can host multiple web applications and multiple Servlets per web application. Each Servlet is a component that responds to incoming HTTP requests.
• Tomcat also provides a "default" Servlet that responds to incoming HTTP requests that aren't handled by any other Servlet. This "default" Servlet delivers the static content of a Java web application, such as image or CSS files.
• Tomcat keeps an internal request count per Servlet.
• Tomcat exposes these internal counts via JMX (Java Management Extensions), a management API common in the Java world, but not used anywhere else.
• Jolokia is a 3rd-party component that exposes parts of JMX as a REST-API.
• CheckMK requires Jolokia to be installed in the monitored Tomcat instance in order to query, among other information, the request counts.
• CheckMK only supports a single request count per web application, where Tomcat keeps track of multiple such counts per web application; one for each Servlet.

Bug report

What is the expected behavior?

For each web application, the request count monitored by CheckMK should equal the total number of incoming HTTP requests, regardless of which Servlet responded to them.

What is the observed behavior?

For each web application, the request count monitored by CheckMK considers only HTTP request responded to by the "default" servlet, i.e. requests for static content such as images of CSS files.

The request count monitored by CheckMK is always significantly smaller than the actual request count for any given web application.

An extreme example would be a web application that only provides a REST-API and has no static content. In such a case, the request count would currently always be zero.

Proposed changes

Using Jolokia, query not just the request count of the "default" Servlet (which is a named Servlet with name default). Query the request counts of all Servlets and aggregate the returned values.

The agent plugin mk_jolokia.py contains two selectors for the request count of Servlets.

# Check not yet working
("*:j2eeType=Servlet,name=default,*", "requestCount", None, ["WebModule"], False),
# too wide location for addressing the right info
# ( "*:j2eeType=Servlet,*", "requestCount", None, [ "WebModule" ] , False),

The first one is currently active and queries only the named Servlet default. If this selector is used, the response from Jolokia contains (per web application) only a single entry for a value named requestCount. This is the reqest count for the "default" Servlet. This causes the issue described above.

The second one is currently commented out and queries all Servlets. If this selector is used, the response from Jolokia contains (per web application) multiple entries for values named requestCount - one for each Servlet of the corresponding web application.

When the response from Jolokia is processed, each of these entries is processes separately. For each entry, the requestCount is extracted, but overwrites any previously extracted value. Therefore, only the last value would be seen by CheckMK. This would not be okay.

It is therefore necessary to aggregate all entries containing a requestCount into a single such entry, before entries are further processed.

This can be done in fetch_metric.

The approach described here and the change proposed in this PR are certainly not the most elegant solution to the problem, as they introduce hard-coded behavior for specific selectors. However, the method fetch_metric already includes hard-coded behavior for another specific selector (threadStatus, threadParam), so this should be acceptable.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[toKrause]

I have read the CLA Document and I hereby sign the CLA or my organization already has a signed CLA.