AbstractTcpTransport connection reset by peer when using PanOS 9 input as a source

[ed-graylog]

Expected Behavior

As a Graylog user when I create a PanOS 8 or 9 input, I expect it to work out of the box without any issues.

Current Behavior

Currently when looking through the Graylog master logs we see the following message over and over
(cause java.io.IOException: Connection reset by peer)

Possible Solution

I'm unclear exactly as to why this is occurring. We know the customer is behind an F5 load balancer however we have attempted using keep alive and not keep alive settings on the inputs. We have additionally attempted using both PanUS 8 and 9 and still got the same errors. Strangely when using raw tcp input we did not get this error however the fields were not created and the time was incorrect when using raw input. We also checked the logs for Graylog and also for F5 and found nothing of note.

Steps to Reproduce (for bugs)

1. Please see HS-635292232 for additional notes and context.

[user29835461]

From my own research I'd say most likely this is due Netty logging when for performance optimizations some network device or server doesn't fully close the TCP connections. It is harmless, although it is irritating to see error spam in the logs.

It is possible to configure log4j logging level for Netty to FATAL, although this might eat also other ERRORs.

[bernd]

@edward-marrufo Thank you! I moved this over to the integrations repository where the code lives.

[waab76]

This appears to be an issue with how we are handling the logging for these connection reset events. In our Netty exception logging handler, we would attempt to log these events at the trace level if trace logging is enabled. However, since trace logging is presumably not enabled in this case, the events are getting logged at the error level: https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog2/inputs/transports/netty/ExceptionLoggingChannelHandler.java#L42-L47

I think we should take another look at the exception logging handler and consider logging these events at trace if trace is enabled or not logging them at all if trace is disabled.

[ed-graylog]

From a support perspective, does this mean there is no legitimate issue, it is merely a harmless logging issue at this point?

[dio99]

same is if the raw syslog input too.

2022-05-12 13:13:36,801 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Raw/Plaintext TCP/618241d06c7ad24c5315a989] (channel [id: 0xd360c1af, L:/172.16.2.116:514 ! R:/172.16.2.14:32894]) (cause io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer)
2022-05-12 13:13:39,786 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Raw/Plaintext TCP/618241d06c7ad24c5315a989] (channel [id: 0x864ddd0f, L:/172.16.2.116:514 ! R:/172.16.2.14:32946]) (cause io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer)