Initial handler setup

Hi everyone -

I’m trying to get Sensu (v 0.10.2) working, and the first thing I want to do is write a pipe handler. I’ve tried to follow the docs as well as possible, but my handler just doesn’t run. Here’s what I have:

In /etc/sensu/conf.d/check_cron.json (the handler I want to get working is “foo”):

{

“checks”: {

"cron_check": {

  "command": "/etc/sensu/plugins/check-procs.rb -p cron -C 1 ",

  "interval": 60,

  "subscribers": [ "allservers" ],

  "handlers": [ "default", "foo" ]

}

}

}

In /etc/sensu/conf.d/handler_foo.json:

{

“handlers”: {

“foo”: {

“type”: “pipe”,

“command”: “/etc/sensu/handlers/foo.rb”

}

}

}

In /etc/sensu/handlers/foo.rb:

#!/usr/bin/ruby

f = open("/tmp/crap", “w”)

f.write(“Daisy, Daisy, give me your answer do\n”)

f.close()

With this configuration, I restart sensu-server and sensu-client, and wait for the cron check to run. The server loads the handler config and publishes the check (this is sensu-server.log):

{“timestamp”:“2013-09-18T16:31:16.900729+0000”,“level”:“warn”,“message”:“config file applied changes”,"

config_file":"/etc/sensu/conf.d/handler_foo.json",“changes”:{“handlers”:{“foo”:[null,{“type”:"pip

e",“command”:"/etc/sensu/handlers/foo.rb"}]}}}

{“timestamp”:“2013-09-18T16:31:16.901039+0000”,“level”:“warn”,“message”:“config file applied changes”,"

config_file":"/etc/sensu/conf.d/check_cron.json",“changes”:{“checks”:{“cron_check”:[null,{“command”:"/e

tc/sensu/plugins/check-procs.rb -p cron -C 1 “,“interval”:60,“subscribers”:[“allservers”],“handlers”:[”

default",“foo”]}]}}}

{“timestamp”:“2013-09-18T16:31:16.901239+0000”,“level”:“warn”,“message”:“config file applied changes”,"

config_file":"/etc/sensu/conf.d/client.json",“changes”:{“client”:[null,{“name”:“sensu.exosite.com”,"add

ress":“127.0.0.1”,“subscriptions”:[“test”,“allservers”]}]}}

{“timestamp”:“2013-09-18T16:31:16.910911+0000”,“level”:“info”,“message”:“loaded extension”,“type”:"muta

tor",“name”:“only_check_output”,“description”:“returns check output”}

{“timestamp”:“2013-09-18T16:31:16.914953+0000”,“level”:“info”,“message”:“loaded extension”,“type”:"hand

ler",“name”:“debug”,“description”:“outputs json event data”}

{“timestamp”:“2013-09-18T16:31:16.949774+0000”,“level”:“info”,“message”:“i am the master”}

{“timestamp”:“2013-09-18T16:31:36.950517+0000”,“level”:“info”,“message”:“pruning aggregations”}

{“timestamp”:“2013-09-18T16:31:46.951435+0000”,“level”:“info”,“message”:“determining stale clients”}

{“timestamp”:“2013-09-18T16:31:56.950914+0000”,“level”:“info”,“message”:“pruning aggregations”}

{“timestamp”:“2013-09-18T16:32:16.952034+0000”,“level”:“info”,“message”:“determining stale clients”}

{“timestamp”:“2013-09-18T16:32:16.973885+0000”,“level”:“info”,“message”:“pruning aggregations”}

{“timestamp”:“2013-09-18T16:32:18.951999+0000”,“level”:“info”,“message”:“publishing check request”,“payload”:{“name”:“cron_check”,“issued”:1379521938,“command”:"/etc/sensu/plugins/check-procs.rb -p cron -C 1 "},“subscribers”:[“allservers”]}

The client receives the publication and processes it (sensu-client.log below):

{“timestamp”:“2013-09-18T18:47:19.139178+0000”,“level”:“info”,“message”:“received check request”,“check”:{“name”:“cron_check”,“issued”:1379530039,“command”:"/etc/sensu/plugins/check-procs.rb -p cron -C 1 "}}

{“timestamp”:“2013-09-18T18:47:19.240836+0000”,“level”:“info”,“message”:“publishing check result”,“payload”:{“client”:“sensu.exosite.com”,“check”:{“name”:“cron_check”,“issued”:1379530039,“command”:"/etc/sensu/plugins/check-procs.rb -p cron -C 1 ",“interval”:60,“subscribers”:[“allservers”],“handlers”:[“default”,“foo”],“executed”:1379530039,“output”:“CheckProcs OK: Found 1 matching processes; cmd /cron/\n”,“status”:0,“duration”:0.101}}}

No queue remains in RabbitMQ, so I guess the server must receive the response from the client. But the server never logs that it passed anything to the “foo” handler. And that handler never runs (/tmp/crap never gets created).

What am I missing? I am prepared for embarrassment; I just can’t seem to get past this step.

Handlers only handle the change of state of a check. Sensu assumes a check starts out in the passing state, and your check result shows that cron is running, so it remains in the passing state. Your handler won’t be called until it has something to do, like notify somebody that cron is dead or that it is running again. :slight_smile:

···

On Sep 18, 2013, at 3:09 PM, Dan Slimmon dan.slimmon@gmail.com wrote:

No queue remains in RabbitMQ, so I guess the server must receive the response from the client. But the server never logs that it passed anything to the “foo” handler. And that handler never runs (/tmp/crap never gets created).

What am I missing? I am prepared for embarrassment; I just can’t seem to get past this step.


Brian Lalor
blalor@bravo5.org

Also make sure that you’re configuring both your handler and your check on the server, not on the node. The handler shouldn’t be on the client, and the check configs are actually sent out from the server to all nodes*.

It’s something I wish the doc made a little clearer.

HTH,

Mat

  • Unless you’re using standalone check instead of subscriber-based ones. In this case, the check config is on the node.
···

On Thursday, September 19, 2013, Brian Lalor wrote:

On Sep 18, 2013, at 3:09 PM, Dan Slimmon dan.slimmon@gmail.com wrote:

No queue remains in RabbitMQ, so I guess the server must receive the response from the client. But the server never logs that it passed anything to the “foo” handler. And that handler never runs (/tmp/crap never gets created).

What am I missing? I am prepared for embarrassment; I just can’t seem to get past this step.

Handlers only handle the change of state of a check. Sensu assumes a check starts out in the passing state, and your check result shows that cron is running, so it remains in the passing state. Your handler won’t be called until it has something to do, like notify somebody that cron is dead or that it is running again. :slight_smile:


Brian Lalor
blalor@bravo5.org
http://github.com/blalor


Rock Solid Ops: development & operations consulting for Ruby on Rails