Syslog forwarding instructions not working

[GRRLjay]

I am attempting to collect syslog data from Extreme switches. I've followed the docs below. I know that the logs are getting to the rsyslog service because I can see them in the syslog. However, they never show up in Kibana.

What do I need to do to troubleshoot?

https://cisagov.github.io/lme-docs/docs/markdown/logging-guidance/syslog-forwarding/

[cbaxley]

We are going to have to look into this. We haven't been following this path since the upgrade to 2.0 but are planning to.
I will take a look at it.

[cbaxley]

So they are getting to the rsyslog service on what machine? One of the clients with an agent installed, or the LME servers rsyslog?

[cbaxley]

The markdown parsing is hiding how the configuration should look in the syslog config. It should read *.* @@lme-server-ip:5140

[GRRLjay]

So they are getting to the rsyslog service on what machine? One of the clients with an agent installed, or the LME servers rsyslog?

Messages are getting to the /var/log/syslog file of the machine that is running the LME containers.

[GRRLjay]

The markdown parsing is hiding how the configuration should look in the syslog config. It should read *.* @@lme-server-ip:5140

I do have this correct in the config file.

[cbaxley]

Ok. Try this and see if we can get something of value.
Make sure you have restarted the lme services

service lme restart

Enable Debug Mode in syslog
Edit /etc/rsyslog.conf and add at the top:

$DebugFile /var/log/rsyslog-debug.log
$DebugLevel 2

Then restart rsyslog:

sudo systemctl restart rsyslog
sudo tail -f /var/log/rsyslog-debug.log

[GRRLjay]

It turned out to be a certificate issue. I previously had #519 where we worked through signing the certificates with our corporate root.

However, it seems the process is incomplete, because the root certificate is not installed in the LME container which means our cert is invalid and doesn't work. I was able to determine there was a cert issue by using the diagnostics tab for the lme-fleet-server agent.

The short term answer was to set "ssl.verification_mode: none" in the fleet-server-policy. I would like it all to work properly though. Any suggestions?

[GRRLjay]

Now it's not working again so maybe that wasn't it. The ryslog-debug.log is not helpful to me, I don't see any obvious issues. I'm attaching the output for one message.

syslog-debug.log

Please note that the "ssl.verification_mode: none" setting DID in fact improve the situation because it turns out we were getting nothing at all from lme-fleet-server. Now were are at least getting something, just not the syslog data.

[GRRLjay]

I have it working again. I found an issue with the permissions on /var/log/syslog

So to summarize the components of the issue:

1. The missing root cert did not allow communication.
2. The permissions on /var/log/syslog has to be 644.
3. I had to change the timezone in the integration to get the syslog entries to register right.

Outstanding: how do we get the root cert installed in the container?