sensu not really sudoing for checks?


#1

Hi guys,

I have a check definition that is this:

{

“checks”: {

“jetty_core_processes”: {

“command”: “/usr/bin/sudo /etc/sensu/plugins/check-sv-processes.sh jetty-core”,

“subscribers”: [

“coreserver”

],

“interval”: 60

}

}

}

The sensu admin dashboard is showing this alert as critical. However, when running the command on the server as the sensu user account, it comes back okay (note that this shows passwordless sudo works):

sensu@s-core:~$ /usr/bin/sudo /etc/sensu/plugins/check-sv-processes.sh jetty-core

OK: All jetty-core services running | valid=1, invalid=0

I am able to replicate the error by not using sudo:

sensu@s-core:~$ /etc/sensu/plugins/check-sv-processes.sh jetty-core

CRITICAL: 1 jetty-core services down: jetty-core | valid=0, invalid=1

The sensu-client log claims it is running with sudo: {“timestamp”:“2013-06-05T23:37:57.322549+0000”,“level”:“info”,“message”:“publishing check result”,“payload”:{“client”:"",“check”:{“name”:“jetty_core_processes”,“issued”:1370475477,“command”:"/usr/bin/sudo /etc/sensu/plugins/check-sv-processes.sh jetty-core",“subscribers”:[“coreserver”],“interval”:60,“executed”:1370475477,“command_executed”:"/usr/bin/sudo /etc/sensu/plugins/check-sv-processes.sh jetty-core",“output”:“CRITICAL: 1 jetty-core services down: jetty-core | valid=0, invalid=1\n”,“status”:2,“duration”:0.021}}}

So, I am let to believe that somehow sensu isn’t sudoing properly. However, I can’t figure out why. I have other checks which also use sudo this way, but they seem to be fine. Does anyone have any ideas? The check file itself seems pretty standard:

carmstrong@s-core:~$ cat /etc/sensu/plugins/check-sv-processes.sh

#!/bin/bash

let valid=0

let invalid=0

failed_services=()

cd /etc/service

for service in $1*; do

status=$(sv status $service | sed -e ‘s/:.*//’)

if [[ $status == ‘run’ ]]; then

let valid=($valid + 1)

else

let invalid=($invalid + 1)

failed_services+=($service)

fi

done

if [ ! -z $1 ]; then

friendly_name="$1 "

fi

if [ $invalid != 0 ]; then

echo “CRITICAL: invalid {friendly_name}services down: ${failed_services} | valid=$valid, invalid=$invalid”

exit 2

else

echo “OK: All ${friendly_name}services running | valid=$valid, invalid=$invalid”

exit 0

fi

Thanks,

Chris


#2

Hate to bother the group again, but getting this check working is a priority. Any ideas?

···

On Wed, Jun 5, 2013 at 4:41 PM, Christopher Armstrong chris@chrisarmstrong.me wrote:

Hi guys,

I have a check definition that is this:

{

“checks”: {

“jetty_core_processes”: {

“command”: “/usr/bin/sudo /etc/sensu/plugins/check-sv-processes.sh jetty-core”,

“subscribers”: [

“coreserver”

],

“interval”: 60

}

}

}

The sensu admin dashboard is showing this alert as critical. However, when running the command on the server as the sensu user account, it comes back okay (note that this shows passwordless sudo works):

sensu@s-core:~$ /usr/bin/sudo /etc/sensu/plugins/check-sv-processes.sh jetty-core

OK: All jetty-core services running | valid=1, invalid=0

I am able to replicate the error by not using sudo:

sensu@s-core:~$ /etc/sensu/plugins/check-sv-processes.sh jetty-core

CRITICAL: 1 jetty-core services down: jetty-core | valid=0, invalid=1

The sensu-client log claims it is running with sudo: {“timestamp”:“2013-06-05T23:37:57.322549+0000”,“level”:“info”,“message”:“publishing check result”,“payload”:{“client”:"",“check”:{“name”:“jetty_core_processes”,“issued”:1370475477,“command”:"/usr/bin/sudo /etc/sensu/plugins/check-sv-processes.sh jetty-core",“subscribers”:[“coreserver”],“interval”:60,“executed”:1370475477,“command_executed”:"/usr/bin/sudo /etc/sensu/plugins/check-sv-processes.sh jetty-core",“output”:“CRITICAL: 1 jetty-core services down: jetty-core | valid=0, invalid=1\n”,“status”:2,“duration”:0.021}}}

So, I am let to believe that somehow sensu isn’t sudoing properly. However, I can’t figure out why. I have other checks which also use sudo this way, but they seem to be fine. Does anyone have any ideas? The check file itself seems pretty standard:

carmstrong@s-core:~$ cat /etc/sensu/plugins/check-sv-processes.sh

#!/bin/bash

let valid=0

let invalid=0

failed_services=()

cd /etc/service

for service in $1*; do

status=$(sv status $service | sed -e ‘s/:.*//’)

if [[ $status == ‘run’ ]]; then

let valid=($valid + 1)

else

let invalid=($invalid + 1)

failed_services+=($service)

fi

done

if [ ! -z $1 ]; then

friendly_name="$1 "

fi

if [ $invalid != 0 ]; then

echo “CRITICAL: invalid {friendly_name}services down: ${failed_services} | valid=$valid, invalid=$invalid”

exit 2

else

echo “OK: All ${friendly_name}services running | valid=$valid, invalid=$invalid”

exit 0

fi

Thanks,

Chris


#3

No worries! Also remember to use the mailing list in combo with IRC. Different folks have their different workflows.

I can’t see anything wrong from here, my only advice would be to print a lot of debugging output to stdout, and look at it in the logs. Sometimes sudoers can also have issues with sudo ‘requiretty’ and ‘secure_path’ settings. Mine (from /etc/sudoers.d/sensu) are:

Defaults:sensu !requiretty

Defaults:sensu secure_path = /usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin


#4

Thanks Nick. I didn’t think of using IRC. I thought about it being an issue with sudo permissions, but I have other sudo checks which (I think) are working fine. I’ll have to do an audit of those and make sure they’re actually sudoing (although I would be getting an error if they tried and couldn’t).

···

On Fri, Jun 7, 2013 at 3:13 PM, Nick Stielau nick.stielau@gmail.com wrote:

No worries! Also remember to use the mailing list in combo with IRC. Different folks have their different workflows.

I can’t see anything wrong from here, my only advice would be to print a lot of debugging output to stdout, and look at it in the logs. Sometimes sudoers can also have issues with sudo ‘requiretty’ and ‘secure_path’ settings. Mine (from /etc/sudoers.d/sensu) are:

Defaults:sensu !requiretty

Defaults:sensu secure_path = /usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin


#5