sensu-community remediation handler causes multiple server instances to spawn on restart + funkyness


#1

Hi,

I am trying to get this plugin ( https://github.com/sensu/sensu-community-plugins/blob/master/handlers/remediation/sensu.rb [latest commit ~6days old ) up and running however I seem to have run into a issue.

When I place that file for loading along with this json file:

{

“handlers”: {

“remediator”: {

“type”: “pipe”,

“command”: “/etc/sensu/extensions/handlers/sensu.rb”

}

}

}

and issue a restart of sensu-server, the server stops, appears to start (ie the logs are indicating startup) BUT the console is unresponsive and I see this happen:

BEFORE RESTART:

root@aws-ap-se-2-sgelk:/etc/fluentd# ps -ef | grep sensu

sensu 1626 1 0 Jan10 ? 00:02:06 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-client -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-client.pid -l /var/log/sensu/sensu-client.log -L info

root 1704 1 0 Jan10 ? 00:00:00 su -s /bin/sh uchiwa -c cd /opt/uchiwa/src && exec “/opt/uchiwa/bin/uchiwa” -c /etc/sensu/uchiwa.json -p /opt/uchiwa/src/public

uchiwa 1708 1704 0 Jan10 ? 00:01:30 /opt/uchiwa/bin/uchiwa -c /etc/sensu/uchiwa.json -p /opt/uchiwa/src/public

sensu 31391 1 0 08:39 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-server -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-server.pid -l /var/log/sensu/sensu-server.log -L info

sensu 31433 1 0 08:39 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-api -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-api.pid -l /var/log/sensu/sensu-api.log -L info

AFTER/DURING RESTART:

root@aws-ap-se-2-sgelk:/etc/fluentd# ps -ef | grep sensu

sensu 1626 1 0 Jan10 ? 00:02:06 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-client -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-client.pid -l /var/log/sensu/sensu-client.log -L info

root 1704 1 0 Jan10 ? 00:00:00 su -s /bin/sh uchiwa -c cd /opt/uchiwa/src && exec “/opt/uchiwa/bin/uchiwa” -c /etc/sensu/uchiwa.json -p /opt/uchiwa/src/public

uchiwa 1708 1704 0 Jan10 ? 00:01:30 /opt/uchiwa/bin/uchiwa -c /etc/sensu/uchiwa.json -p /opt/uchiwa/src/public

sensu 31433 1 0 08:39 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-api -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-api.pid -l /var/log/sensu/sensu-api.log -L info

root 31882 11209 0 08:48 pts/3 00:00:00 /bin/bash /etc/init.d/sensu-server restart

root 31883 31882 0 08:48 pts/3 00:00:00 /bin/bash /etc/init.d/sensu-service server restart

sensu 31910 31883 2 08:48 pts/3 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-server -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-server.pid -l /var/log/sensu/sensu-server.log -L info

sensu 31912 31910 0 08:48 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-server -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-server.pid -l /var/log/sensu/sensu-server.log -L info

sensu 31915 31912 0 08:48 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-server -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-server.pid -l /var/log/sensu/sensu-server.log -L info

so 3x instances of sensu-server.

I have tested it with the embedded ruby and a standard ruby:

root@aws-ap-se-2-sgelk:/etc/sensu/extensions/handlers# ruby --version

ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

root@aws-ap-se-2-sgelk:/etc/sensu/extensions/handlers# /opt/sensu/embedded/bin/ruby --version

ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-linux]

This is occuring on 2 seperate Ubuntu 14.04 instances as well:

root@aws-ap-se-2-sgelk:/etc/sensu/extensions/handlers# uname -a

Linux aws-ap-se-2-sgelk 3.13.0-36-generic #63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

and sensu-plugin (1.1.0)

I’ve also notice this pops up in the logs when terminating the sensu-server:

^C/var/lib/gems/1.9.1/gems/sensu-plugin-1.1.0/lib/sensu-plugin/utils.rb:23:in `read’: Interrupt

from /var/lib/gems/1.9.1/gems/sensu-plugin-1.1.0/lib/sensu-plugin/utils.rb:23:in `read_event’

from /var/lib/gems/1.9.1/gems/sensu-plugin-1.1.0/lib/sensu-handler.rb:53:in `block in class:Handler

Any idea? I’m a bit stumped and would love to get this working.

I’ve also noticed that the .rb file doesn’t even appear in the logs when starting up, almost like it is blocked somehow.

thanks


#2

some debugging has found that this

require ‘rubygems’ if RUBY_VERSION < ‘1.9.0’

require ‘sensu-handler’

class Show < Sensu::Handler

def handle

puts 'Settings: ’ + settings.to_hash.inspect

puts 'Event: ’ + @event.inspect

end

end

will also cause this issue and the fault seems to lie with the

require ‘sensu-handler’

···

On Wednesday, January 14, 2015 at 8:30:15 PM UTC+11, nzspambot wrote:

Hi,

I am trying to get this plugin ( https://github.com/sensu/sensu-community-plugins/blob/master/handlers/remediation/sensu.rb [latest commit ~6days old ) up and running however I seem to have run into a issue.

When I place that file for loading along with this json file:

{

“handlers”: {

“remediator”: {

“type”: “pipe”,

“command”: “/etc/sensu/extensions/handlers/sensu.rb”

}

}

}

and issue a restart of sensu-server, the server stops, appears to start (ie the logs are indicating startup) BUT the console is unresponsive and I see this happen:

BEFORE RESTART:

root@aws-ap-se-2-sgelk:/etc/fluentd# ps -ef | grep sensu

sensu 1626 1 0 Jan10 ? 00:02:06 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-client -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-client.pid -l /var/log/sensu/sensu-client.log -L info

root 1704 1 0 Jan10 ? 00:00:00 su -s /bin/sh uchiwa -c cd /opt/uchiwa/src && exec “/opt/uchiwa/bin/uchiwa” -c /etc/sensu/uchiwa.json -p /opt/uchiwa/src/public

uchiwa 1708 1704 0 Jan10 ? 00:01:30 /opt/uchiwa/bin/uchiwa -c /etc/sensu/uchiwa.json -p /opt/uchiwa/src/public

sensu 31391 1 0 08:39 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-server -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-server.pid -l /var/log/sensu/sensu-server.log -L info

sensu 31433 1 0 08:39 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-api -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-api.pid -l /var/log/sensu/sensu-api.log -L info

AFTER/DURING RESTART:

root@aws-ap-se-2-sgelk:/etc/fluentd# ps -ef | grep sensu

sensu 1626 1 0 Jan10 ? 00:02:06 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-client -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-client.pid -l /var/log/sensu/sensu-client.log -L info

root 1704 1 0 Jan10 ? 00:00:00 su -s /bin/sh uchiwa -c cd /opt/uchiwa/src && exec “/opt/uchiwa/bin/uchiwa” -c /etc/sensu/uchiwa.json -p /opt/uchiwa/src/public

uchiwa 1708 1704 0 Jan10 ? 00:01:30 /opt/uchiwa/bin/uchiwa -c /etc/sensu/uchiwa.json -p /opt/uchiwa/src/public

sensu 31433 1 0 08:39 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-api -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-api.pid -l /var/log/sensu/sensu-api.log -L info

root 31882 11209 0 08:48 pts/3 00:00:00 /bin/bash /etc/init.d/sensu-server restart

root 31883 31882 0 08:48 pts/3 00:00:00 /bin/bash /etc/init.d/sensu-service server restart

sensu 31910 31883 2 08:48 pts/3 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-server -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-server.pid -l /var/log/sensu/sensu-server.log -L info

sensu 31912 31910 0 08:48 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-server -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-server.pid -l /var/log/sensu/sensu-server.log -L info

sensu 31915 31912 0 08:48 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-server -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-server.pid -l /var/log/sensu/sensu-server.log -L info

so 3x instances of sensu-server.

I have tested it with the embedded ruby and a standard ruby:

root@aws-ap-se-2-sgelk:/etc/sensu/extensions/handlers# ruby --version

ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

root@aws-ap-se-2-sgelk:/etc/sensu/extensions/handlers# /opt/sensu/embedded/bin/ruby --version

ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-linux]

This is occuring on 2 seperate Ubuntu 14.04 instances as well:

root@aws-ap-se-2-sgelk:/etc/sensu/extensions/handlers# uname -a

Linux aws-ap-se-2-sgelk 3.13.0-36-generic #63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

and sensu-plugin (1.1.0)

I’ve also notice this pops up in the logs when terminating the sensu-server:

^C/var/lib/gems/1.9.1/gems/sensu-plugin-1.1.0/lib/sensu-plugin/utils.rb:23:in `read’: Interrupt

from /var/lib/gems/1.9.1/gems/sensu-plugin-1.1.0/lib/sensu-plugin/utils.rb:23:in `read_event’

from /var/lib/gems/1.9.1/gems/sensu-plugin-1.1.0/lib/sensu-handler.rb:53:in `block in class:Handler

Any idea? I’m a bit stumped and would love to get this working.

I’ve also noticed that the .rb file doesn’t even appear in the logs when starting up, almost like it is blocked somehow.

thanks