SensuGO Agent and Backend Logging

Sensu Go
1

Hello Team - I have been trying to redirect the agent and backend logs to the service specific files, however seeing some challenges. I did walk through the Sensu-agent not running - Solved , however still having some questions that I thought I will post here for clarity.

I am using systemd service for managing the agent as well as the backend.

[Unit]
Description=The Sensu Agent process.
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
LimitNOFILE=65535
ExecStart=/usr/share/sensugo/sensu-agent start
Restart=on-failure
RestartSec=1min
WorkingDirectory=/

[Install]
WantedBy=multi-user.target

And it looks like the startup is using chroot.

cat /usr/share/sensugo/sensu-agent

name=sensu-agent
program=/usr/sbin/sensu-agent
args=“start”
pidfile="/var/run/sensugo/$name.pid"
user=“sensu”
group=“sensu”
chroot="/"
chdir="/"

chroot --userspec “$user”:"$group" “$chroot” sh -c "
cd "chroot\" ulimit -n {MAX_OPEN_FILES}
exec “$program” $args
" >> /var/log/sensu/sensu-agent.log 2>> /var/log/sensu/sensu-agent.log &

If I don’t use chroot and call the binary directly, the logs are not getting redirected to the service specific log files. Is it possible to direct the log files with a simple bash -c “sensu-agent binary start” >> log_file 2>>log_file while still able to write to the log files ?

I also followed this guide this https://docs.sensu.io/sensu-go/latest/guides/systemd-logs/ , but no luck.

Wish we had a parameter to pass the log file location to keep it simple :slight_smile: , similar to sensu-core

Any other recommendations please ?

2

Hey,
It sounds like you are not familiar with systemd’s journal and the journalctl facility. But this isn’t a linux forum, so I’ll refrain from diving into a lesson on using journalctl and try to show you how you can modify the systemd managed service to produce log files like you’ve stated.

If you want to customize the systemd service unit for sensu-agent, first create an overrides directory for it:

/etc/systemd/system/sensu-agent.service.d

Any configuration you put into that directory will override provided settings in the vendor package provided service unit. If you edit the vendor package provided service unit, you risk losing your edits on package upgrade. The overrides directory concept in systemd is so much better than some sysvinit implementations, it really helps to keep local mods separate from vendor tested/support scripting for troubleshooting later. Essentially every part of the systemd service configuration is overridable.

Now create a file like:

/etc/systemd/system/sensu-agent.service.d/redirect-logs.conf

Name doesn’t matter so much, name it something that helps you remember why its there.
This is the file were we are going to add systemd service configuration options or override the running executable script as needed to run sensu-agent however we want.

Okay, first option if your systemd is new enough to support it (systemd 240 I believe), you can specify
exactly what files you want to redirect stdout and stderr to, and set them up to append.
Here’s the doc reference:
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardOutput=
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardError=

Here’s the content of my redirect-logs.conf

[Service]

StandardOutput=append:/var/log/sensu/sensu-agent.log
StandardError=append:/var/log/sensu/sensu-agent.log

Once you add the file you’ll need to do a systemctl daemon-reload to have systemd pick up the new config, and then restart the sensu-agent service.
After that you should have the sensu-agent.log getting populated and the journal should be pretty bare.

If that doesn’t work you can use the same override config file to replace the ExecStart= with a bash script wrapper lilke…

ExecStart=/bin/sh -c '/usr/sbin/sensu-agent start 2>&1 >> /var/log/sensu/sensu-agent.log'

Again don’t edit the vendor provided service unit, use the override directory to make sure package upgrades don’t overwrite your overrides.