unknown handler "mailer"


#1

I have sensu pretty much setup and running. I used the mailer.rb handler from the sensu-community-plugins repo and put it in /etc/sensu/handlers. I also configured mailer.json and put it in /etc/sensu/conf.d/mailer.json

Here is what my mailer.json looks like.

{

“mailer”: {

"admin_gui": "http://25.149.38.106:5555/",

"mail_from": "helpdesk@mydomain.com",

"mail_to": "james.jelinek@mydomain.com",

"smtp_address": "smtp.gmail.com",

"smtp_port": "587",

"smtp_username": "helpdesk@mydomain.com",

"smtp_password": "password`",

"smtp_domain": "mydomain.com"

}

}

Then I add the handler to a check_disk.json check that I have setup as a test like so:

{

“checks”: {

"disk": {

  "command": "/etc/sensu/plugins/check-disk.sh -w 80 -c 90 -d /",

  "interval": 10,

  "handlers": ["default", "mailer"],

  "subscribers": [

    "test"

  ]

}

}

}

I tail the sensu-server.log and I get the following error when the check_disk check runs:

{“timestamp”:“2015-07-23T08:10:59.376218-0500”,“level”:“info”,“message”:“processing event”,“event”:{“id”:“c9474c1f-fbde-4605-b5d5-d40c70655fe1”,“client”:{“name”:“mfp”,“address”:“25.147.26.157”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437656971},“check”:{“command”:"/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90",“interval”:10,“handlers”:[“default”,“mailer”],“subscribers”:[“test”],“name”:“disk”,“issued”:1437657059,“executed”:1437656980,“duration”:0.011,“output”:“CRITICAL - /dev/md1 space used=94% | ‘/dev/md1 usage’=94%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:6531,“action”:“create”}}

{“timestamp”:“2015-07-23T08:10:59.378436-0500”,“level”:“error”,“message”:“unknown handler”,“handler_name”:“mailer”}

So even though I have it setup properly (I think) as a handler and a json definition, I get ,“level”:“error”,“message”:“unknown handler”,“handler_name”:“mailer”}

Any thoughts on why this is happening?

Thanks in advance guys!

-James


#2

Ok so it looks like I wasn’t specifying the handler type or command. So here’s what I have so far:

{

“handlers”: {

“mailer”: {

"type": "pipe",

"command": "/etc/sensu/handlers/mailer.rb",

"admin_gui": "http://25.149.38.106:5555/",

"mail_from": "helpdesk@mydomain.com",

"mail_to": "james.jelinek@mydomain.com",

"smtp_address": "smtp.gmail.com",

"smtp_port": "587",

"smtp_username": "helpdesk@mydomain.com",

"smtp_password": "password`",

"smtp_domain": "mydomain.com"

}

}

}

Does this look right? I have a check in check_disk.json that is failing and it’s showing on the dashboard, but I’m not receiving any emails. Here’s a sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:“helpdesk@mydomain.com”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com”,“smtp_port”:“587”,“smtp_username”:“helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:["only handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting executed, but I’m not sure if I have it setup right or not.

The docs are very sparse so I’m sort of flying blind here. Any help is greatly appreciated guys!

-James


#3

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

···

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek <jelinek@gmail.com> wrote:

Ok so it looks like I wasn't specifying the handler type or command. So
here's what I have so far:

{

"handlers": {

  "mailer": {

    "type": "pipe",

    "command": "/etc/sensu/handlers/mailer.rb",

    "admin_gui": "http://25.149.38.106:5555/",

    "mail_from": "helpdesk@mydomain.com",

    "mail_to": "james.jelinek@mydomain.com",

    "smtp_address": "smtp.gmail.com",

    "smtp_port": "587",

    "smtp_username": "helpdesk@mydomain.com",

    "smtp_password": "password`",

    "smtp_domain": "mydomain.com"

  }

}

}

Does this look right? I have a check in check_disk.json that is failing and
it's showing on the dashboard, but I'm not receiving any emails. Here's a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:“helpdesk@mydomain.com”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com”,“smtp_port”:“587”,“smtp_username”:“helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:["only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I'm not sure if I have it setup right or not.

The docs are very sparse so I'm sort of flying blind here. Any help is
greatly appreciated guys!

-James


#4

Ok here’s an outpour from my /var/sensu/sensu-server.log (stripping out the credentials). It looks like the check gets fired then on error the handler gets called. But I’m not sure exactly what I’m looking at here. It seems like it’s firing the check and then the handler. But I’m not sure what else to look at from there.

Where else in my log can I look?

{“timestamp”:“2015-07-23T09:57:29.326564-0500”,“level”:“info”,“message”:“processing event”,“event”:{“id”:“c7ddd6c8-c75b-496f-8b91-73c76f6992dc”,“client”:{“name”:“wwv”,“address”:“25.140.134.150”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437663447},“check”:{“command”:"/etc/sensu/plugins/check-disk.sh -w 80 -c 90 -d /",“interval”:10,“handlers”:[“default”,“mailer”],“subscribers”:[“test”],“name”:“disk”,“issued”:1437663449,“executed”:1437663449,“duration”:0.009,“output”:“CRITICAL - / space used=97% | ‘/ usage’=97%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:7107,“action”:“create”}}

{“timestamp”:“2015-07-23T09:57:29.335682-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“cat”,“name”:“default”},“output”:["{“id”:“c7ddd6c8-c75b-496f-8b91-73c76f6992dc”,“client”:{“name”:“wwv”,“address”:“25.140.134.150”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437663447},“check”:{“command”:"/etc/sensu/plugins/check-disk.sh -w 80 -c 90 -d /",“interval”:10,“handlers”:[“default”,“mailer”],“subscribers”:[“test”],“name”:“disk”,“issued”:1437663449,“executed”:1437663449,“duration”:0.009,“output”:“CRITICAL - / space used=97% | ‘/ usage’=97%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:7107,“action”:“create”}"]}

{“timestamp”:“2015-07-23T09:57:29.378473-0500”,“level”:“info”,“message”:“processing event”,“event”:{“id”:“3784bf61-36af-4941-ac85-6861dd2bf67b”,“client”:{“name”:“mfp”,“address”:“25.147.26.157”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437663368},“check”:{“command”:"/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90",“interval”:10,“handlers”:[“default”,“mailer”],“subscribers”:[“test”],“name”:“disk”,“issued”:1437663449,“executed”:1437663370,“duration”:0.01,“output”:“CRITICAL - /dev/md1 space used=94% | ‘/dev/md1 usage’=94%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:7102,“action”:“create”}}

{“timestamp”:“2015-07-23T09:57:29.391024-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“cat”,“name”:“default”},“output”:["{“id”:“3784bf61-36af-4941-ac85-6861dd2bf67b”,“client”:{“name”:“mfp”,“address”:“25.147.26.157”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437663368},“check”:{“command”:"/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90",“interval”:10,“handlers”:[“default”,“mailer”],“subscribers”:[“test”],“name”:“disk”,“issued”:1437663449,“executed”:1437663370,“duration”:0.01,“output”:“CRITICAL - /dev/md1 space used=94% | ‘/dev/md1 usage’=94%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:7102,“action”:“create”}"]}

{“timestamp”:“2015-07-23T09:57:30.148795-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: wwv/disk\n”]}

{“timestamp”:“2015-07-23T09:57:30.222175-0500”,“level”:“info”,“message”:“publishing check request”,“payload”:{“name”:“memory”,“issued”:1437663450,“command”:"/etc/sensu/plugins/check-mem.sh -w 1024 -c 512"},“subscribers”:[“test”]}

{“timestamp”:“2015-07-23T09:57:30.536387-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: ws/disk\n”]}

{“timestamp”:“2015-07-23T09:57:30.573765-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: mfp/disk\n”]}

···

On Jul 23, 2015, at 9:44 AM, Kyle Anderson kyle@xkyle.com wrote:

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek jelinek@gmail.com wrote:

Ok so it looks like I wasn’t specifying the handler type or command. So
here’s what I have so far:

{

“handlers”: {

“mailer”: {

"type": "pipe",

"command": "/etc/sensu/handlers/mailer.rb",

"admin_gui": "http://25.149.38.106:5555/",

"mail_from": "helpdesk@mydomain.com",

"mail_to": "james.jelinek@mydomain.com",

"smtp_address": "smtp.gmail.com",

"smtp_port": "587",

"smtp_username": "helpdesk@mydomain.com",

"smtp_password": "password`",

"smtp_domain": "mydomain.com"

}

}

}

Does this look right? I have a check in check_disk.json that is failing and
it’s showing on the dashboard, but I’m not receiving any emails. Here’s a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:“helpdesk@mydomain.com”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com”,“smtp_port”:“587”,“smtp_username”:“helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:["only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I’m not sure if I have it setup right or not.

The docs are very sparse so I’m sort of flying blind here. Any help is
greatly appreciated guys!

-James


#5

One thing I noticed with this is that the gem sensu-handler (which is a requirement of mailer.rb) is not installed on my sensu master. When I try to install via omnibus like this

GEM_PATH=/opt/sensu/embedded/lib/ruby/gems/2.0.0 /opt/sensu/embedded/bin/gem install sensu-handler

I get the following:

ERROR: Could not find a valid gem ‘sensu-handler’ (>= 0) in any repository

I’m using embedded ruby (omnibus) and so far installing other gems like mail and timeout worked just fine with the gem install command. This could be why it’s not sending out email or firing properly.

Any thoughts?

···

On Jul 23, 2015, at 9:44 AM, Kyle Anderson kyle@xkyle.com wrote:

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek jelinek@gmail.com wrote:

Ok so it looks like I wasn’t specifying the handler type or command. So
here’s what I have so far:

{

“handlers”: {

“mailer”: {

"type": "pipe",

"command": "/etc/sensu/handlers/mailer.rb",

"admin_gui": "http://25.149.38.106:5555/",

"mail_from": "helpdesk@mydomain.com",

"mail_to": "james.jelinek@mydomain.com",

"smtp_address": "smtp.gmail.com",

"smtp_port": "587",

"smtp_username": "helpdesk@mydomain.com",

"smtp_password": "password`",

"smtp_domain": "mydomain.com"

}

}

}

Does this look right? I have a check in check_disk.json that is failing and
it’s showing on the dashboard, but I’m not receiving any emails. Here’s a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:“helpdesk@mydomain.com”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com”,“smtp_port”:“587”,“smtp_username”:“helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:["only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I’m not sure if I have it setup right or not.

The docs are very sparse so I’m sort of flying blind here. Any help is
greatly appreciated guys!

-James


#6

I still don't see an actual log for the mailer handler. I see the "cat" handler?

'sensu-handler' comes from the.. sensu-plugin gem? You probably
already have it in the embedded ruby.

···

On Thu, Jul 23, 2015 at 8:05 AM, James Jelinek <jelinek@gmail.com> wrote:

One thing I noticed with this is that the gem sensu-handler (which is a
requirement of mailer.rb) is not installed on my sensu master. When I try
to install via omnibus like this

GEM_PATH=/opt/sensu/embedded/lib/ruby/gems/2.0.0 /opt/sensu/embedded/bin/gem
install sensu-handler

I get the following:

ERROR: Could not find a valid gem 'sensu-handler' (>= 0) in any repository

I’m using embedded ruby (omnibus) and so far installing other gems like mail
and timeout worked just fine with the gem install command. This could be why
it’s not sending out email or firing properly.

Any thoughts?

On Jul 23, 2015, at 9:44 AM, Kyle Anderson <kyle@xkyle.com> wrote:

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek <jelinek@gmail.com> wrote:

Ok so it looks like I wasn't specifying the handler type or command. So
here's what I have so far:

{

"handlers": {

"mailer": {

   "type": "pipe",

   "command": "/etc/sensu/handlers/mailer.rb",

   "admin_gui": "http://25.149.38.106:5555/",

   "mail_from": "helpdesk@mydomain.com",

   "mail_to": "james.jelinek@mydomain.com",

   "smtp_address": "smtp.gmail.com",

   "smtp_port": "587",

   "smtp_username": "helpdesk@mydomain.com",

   "smtp_password": "password`",

   "smtp_domain": "mydomain.com"

}

}

}

Does this look right? I have a check in check_disk.json that is failing and
it's showing on the dashboard, but I'm not receiving any emails. Here's a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:“helpdesk@mydomain.com”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com”,“smtp_port”:“587”,“smtp_username”:“helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:["only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I'm not sure if I have it setup right or not.

The docs are very sparse so I'm sort of flying blind here. Any help is
greatly appreciated guys!

-James


#7

Ok nevermind, sensu-handler is included in the gems when you install the sense-plugin gem. And it’s listed as /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb

So what else can I look at to see what’s going on here? I thank the stars for this mailing list, but a lot of this stuff should be more documented. That’s something I’m going to start doing and contribute.

Thanks for your help Kyle.

···

On Jul 23, 2015, at 10:05 AM, James Jelinek jelinek@gmail.com wrote:

One thing I noticed with this is that the gem sensu-handler (which is a requirement of mailer.rb) is not installed on my sensu master. When I try to install via omnibus like this

GEM_PATH=/opt/sensu/embedded/lib/ruby/gems/2.0.0 /opt/sensu/embedded/bin/gem install sensu-handler

I get the following:

ERROR: Could not find a valid gem ‘sensu-handler’ (>= 0) in any repository

I’m using embedded ruby (omnibus) and so far installing other gems like mail and timeout worked just fine with the gem install command. This could be why it’s not sending out email or firing properly.

Any thoughts?

On Jul 23, 2015, at 9:44 AM, Kyle Anderson kyle@xkyle.com wrote:

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek jelinek@gmail.com wrote:

Ok so it looks like I wasn’t specifying the handler type or command. So
here’s what I have so far:

{

“handlers”: {

“mailer”: {

"type": "pipe",

"command": "/etc/sensu/handlers/mailer.rb",

"admin_gui": "[http://25.149.38.106:5555/](http://25.149.38.106:5555/)",

"mail_from": "helpdesk@mydomain.com",

"mail_to": "james.jelinek@mydomain.com",

"smtp_address": "[smtp.gmail.com](http://smtp.gmail.com)",

"smtp_port": "587",

"smtp_username": "helpdesk@mydomain.com",

"smtp_password": "password`",

"smtp_domain": "[mydomain.com](http://mydomain.com)"

}

}

}

Does this look right? I have a check in check_disk.json that is failing and
it’s showing on the dashboard, but I’m not receiving any emails. Here’s a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I’m not sure if I have it setup right or not.

The docs are very sparse so I’m sort of flying blind here. Any help is
greatly appreciated guys!

-James


#8

Ok so for fun I am trying to cat the mailer.json handler definition into the handler itself to see if it’s a ruby problem and I get this on output:

[root@monitor conf.d]# cat mailer.json |/opt/sensu/embedded/bin/ruby …/handlers/mailer.rb

/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `+’: no implicit conversion of nil into String (TypeError)

from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `filter_silenced'

from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:31:in `filter'

from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:54:in `block in <class:Handler>’

I think the problem is it’s passing NIL because I am not generating an actual event since calling from the command line.

Any thoughts on where else to look into the logs to see what/where this handler is firing or not firing and how to debug?

I’m kind of hitting a wall here.

-James

···

On Jul 23, 2015, at 10:09 AM, James Jelinek jelinek@gmail.com wrote:

Ok nevermind, sensu-handler is included in the gems when you install the sense-plugin gem. And it’s listed as /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb

So what else can I look at to see what’s going on here? I thank the stars for this mailing list, but a lot of this stuff should be more documented. That’s something I’m going to start doing and contribute.

Thanks for your help Kyle.

On Jul 23, 2015, at 10:05 AM, James Jelinek jelinek@gmail.com wrote:

One thing I noticed with this is that the gem sensu-handler (which is a requirement of mailer.rb) is not installed on my sensu master. When I try to install via omnibus like this

GEM_PATH=/opt/sensu/embedded/lib/ruby/gems/2.0.0 /opt/sensu/embedded/bin/gem install sensu-handler

I get the following:

ERROR: Could not find a valid gem ‘sensu-handler’ (>= 0) in any repository

I’m using embedded ruby (omnibus) and so far installing other gems like mail and timeout worked just fine with the gem install command. This could be why it’s not sending out email or firing properly.

Any thoughts?

On Jul 23, 2015, at 9:44 AM, Kyle Anderson kyle@xkyle.com wrote:

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek jelinek@gmail.com wrote:

Ok so it looks like I wasn’t specifying the handler type or command. So
here’s what I have so far:

{

“handlers”: {

“mailer”: {

"type": "pipe",

"command": "/etc/sensu/handlers/mailer.rb",

"admin_gui": "[http://25.149.38.106:5555/](http://25.149.38.106:5555/)",

"mail_from": "helpdesk@mydomain.com",

"mail_to": "james.jelinek@mydomain.com",

"smtp_address": "[smtp.gmail.com](http://smtp.gmail.com/)",

"smtp_port": "587",

"smtp_username": "helpdesk@mydomain.com",

"smtp_password": "password`",

"smtp_domain": "[mydomain.com](http://mydomain.com/)"

}

}

}

Does this look right? I have a check in check_disk.json that is failing and
it’s showing on the dashboard, but I’m not receiving any emails. Here’s a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I’m not sure if I have it setup right or not.

The docs are very sparse so I’m sort of flying blind here. Any help is
greatly appreciated guys!

-James


#9

This shows the handler getting fired I think:

{“timestamp”:“2015-07-23T10:34:30.705274-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,"smtp_password”:“password”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: mfp/disk\n”]}

But the 180 occurrences I think is the problem. How can I override this for the handler to make sure the occurrences/refresh is lower so that it actually sends the email?

I was looking through the sensu-users list and found something you touched on last year about this but I’m not totally sure that’s what I need.

-James

···

On Jul 23, 2015, at 10:19 AM, James Jelinek jelinek@gmail.com wrote:

Ok so for fun I am trying to cat the mailer.json handler definition into the handler itself to see if it’s a ruby problem and I get this on output:

[root@monitor conf.d]# cat mailer.json |/opt/sensu/embedded/bin/ruby …/handlers/mailer.rb

/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `+’: no implicit conversion of nil into String (TypeError)

from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `filter_silenced’

from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:31:in `filter’

from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:54:in `block in class:Handler

I think the problem is it’s passing NIL because I am not generating an actual event since calling from the command line.

Any thoughts on where else to look into the logs to see what/where this handler is firing or not firing and how to debug?

I’m kind of hitting a wall here.

-James

On Jul 23, 2015, at 10:09 AM, James Jelinek jelinek@gmail.com wrote:

Ok nevermind, sensu-handler is included in the gems when you install the sense-plugin gem. And it’s listed as /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb

So what else can I look at to see what’s going on here? I thank the stars for this mailing list, but a lot of this stuff should be more documented. That’s something I’m going to start doing and contribute.

Thanks for your help Kyle.

On Jul 23, 2015, at 10:05 AM, James Jelinek jelinek@gmail.com wrote:

One thing I noticed with this is that the gem sensu-handler (which is a requirement of mailer.rb) is not installed on my sensu master. When I try to install via omnibus like this

GEM_PATH=/opt/sensu/embedded/lib/ruby/gems/2.0.0 /opt/sensu/embedded/bin/gem install sensu-handler

I get the following:

ERROR: Could not find a valid gem ‘sensu-handler’ (>= 0) in any repository

I’m using embedded ruby (omnibus) and so far installing other gems like mail and timeout worked just fine with the gem install command. This could be why it’s not sending out email or firing properly.

Any thoughts?

On Jul 23, 2015, at 9:44 AM, Kyle Anderson kyle@xkyle.com wrote:

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek jelinek@gmail.com wrote:

Ok so it looks like I wasn’t specifying the handler type or command. So
here’s what I have so far:

{

“handlers”: {

“mailer”: {

"type": "pipe",

"command": "/etc/sensu/handlers/mailer.rb",

"admin_gui": "[http://25.149.38.106:5555/](http://25.149.38.106:5555/)",

"mail_from": "helpdesk@mydomain.com",

"mail_to": "james.jelinek@mydomain.com",

"smtp_address": "[smtp.gmail.com](http://smtp.gmail.com/)",

"smtp_port": "587",

"smtp_username": "helpdesk@mydomain.com",

"smtp_password": "password`",

"smtp_domain": "[mydomain.com](http://mydomain.com/)"

}

}

}

Does this look right? I have a check in check_disk.json that is failing and
it’s showing on the dashboard, but I’m not receiving any emails. Here’s a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I’m not sure if I have it setup right or not.

The docs are very sparse so I’m sort of flying blind here. Any help is
greatly appreciated guys!

-James


#10

I’ve tried adding occurrences 3 to my check_disk.json to see if I can get the handler to fire like this:

{

“checks”: {

“disk”: {

“command”: “/etc/sensu/plugins/check-disk.sh -w 80 -c 90 -d /”,

“interval”: 10,

“handlers”: [“default”, “mailer”],

“occurrences”: 3,

“subscribers”: [

“test”

]

}

}

}

I’m still seeing the output from the handler of “only handling every 180 occurrences” even after restarting sensu-server and sense-api.

-James

···

On Jul 23, 2015, at 10:37 AM, James Jelinek jelinek@gmail.com wrote:

This shows the handler getting fired I think:

{“timestamp”:“2015-07-23T10:34:30.705274-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,"smtp_password”:“password”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: mfp/disk\n”]}

But the 180 occurrences I think is the problem. How can I override this for the handler to make sure the occurrences/refresh is lower so that it actually sends the email?

I was looking through the sensu-users list and found something you touched on last year about this but I’m not totally sure that’s what I need.

-James

On Jul 23, 2015, at 10:19 AM, James Jelinek jelinek@gmail.com wrote:

Ok so for fun I am trying to cat the mailer.json handler definition into the handler itself to see if it’s a ruby problem and I get this on output:

[root@monitor conf.d]# cat mailer.json |/opt/sensu/embedded/bin/ruby …/handlers/mailer.rb

/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `+’: no implicit conversion of nil into String (TypeError)

from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `filter_silenced'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:31:in `filter'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:54:in `block in <class:Handler>’

I think the problem is it’s passing NIL because I am not generating an actual event since calling from the command line.

Any thoughts on where else to look into the logs to see what/where this handler is firing or not firing and how to debug?

I’m kind of hitting a wall here.

-James

On Jul 23, 2015, at 10:09 AM, James Jelinek jelinek@gmail.com wrote:

Ok nevermind, sensu-handler is included in the gems when you install the sense-plugin gem. And it’s listed as /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb

So what else can I look at to see what’s going on here? I thank the stars for this mailing list, but a lot of this stuff should be more documented. That’s something I’m going to start doing and contribute.

Thanks for your help Kyle.

On Jul 23, 2015, at 10:05 AM, James Jelinek jelinek@gmail.com wrote:

One thing I noticed with this is that the gem sensu-handler (which is a requirement of mailer.rb) is not installed on my sensu master. When I try to install via omnibus like this

GEM_PATH=/opt/sensu/embedded/lib/ruby/gems/2.0.0 /opt/sensu/embedded/bin/gem install sensu-handler

I get the following:

ERROR: Could not find a valid gem ‘sensu-handler’ (>= 0) in any repository

I’m using embedded ruby (omnibus) and so far installing other gems like mail and timeout worked just fine with the gem install command. This could be why it’s not sending out email or firing properly.

Any thoughts?

On Jul 23, 2015, at 9:44 AM, Kyle Anderson kyle@xkyle.com wrote:

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek jelinek@gmail.com wrote:

Ok so it looks like I wasn’t specifying the handler type or command. So
here’s what I have so far:

{

“handlers”: {

“mailer”: {

"type": "pipe",

"command": "/etc/sensu/handlers/mailer.rb",

"admin_gui": "[http://25.149.38.106:5555/](http://25.149.38.106:5555/)",

"mail_from": "helpdesk@mydomain.com",

"mail_to": "james.jelinek@mydomain.com",

"smtp_address": "[smtp.gmail.com](http://smtp.gmail.com/)",

"smtp_port": "587",

"smtp_username": "helpdesk@mydomain.com",

"smtp_password": "password`",

"smtp_domain": "[mydomain.com](http://mydomain.com/)"

}

}

}

Does this look right? I have a check in check_disk.json that is failing and
it’s showing on the dashboard, but I’m not receiving any emails. Here’s a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I’m not sure if I have it setup right or not.

The docs are very sparse so I’m sort of flying blind here. Any help is
greatly appreciated guys!

-James


#11

For a test I changed my command in the mailer.json to send regular mail and not hit the ruby script mailer.rb

“command”: "echo $(cat) > /tmp/mail.txt; mail -s “sensu alert” james.jelinek@streetcredsoftware.com < /tmp/mail.txt”

Every time the check_disk.json fires, it sends me an email about critical disk space (which is good!). So it looks like it’s firing the mailer handler properly, now I’m thinking it’s an issue with the mailer.rb handler and the sensu-handler gem keeping it from firing until there’s 180 occurrences.

How to overwrite this in the mailer.rb?

-James

···

On Jul 23, 2015, at 10:45 AM, James Jelinek jelinek@gmail.com wrote:

I’ve tried adding occurrences 3 to my check_disk.json to see if I can get the handler to fire like this:

{

“checks”: {

“disk”: {

“command”: “/etc/sensu/plugins/check-disk.sh -w 80 -c 90 -d /”,

“interval”: 10,

“handlers”: [“default”, “mailer”],

“occurrences”: 3,

“subscribers”: [

“test”

]

}

}

}

I’m still seeing the output from the handler of “only handling every 180 occurrences” even after restarting sensu-server and sense-api.

-James

On Jul 23, 2015, at 10:37 AM, James Jelinek jelinek@gmail.com wrote:

This shows the handler getting fired I think:

{“timestamp”:“2015-07-23T10:34:30.705274-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,"smtp_password”:“password”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: mfp/disk\n”]}

But the 180 occurrences I think is the problem. How can I override this for the handler to make sure the occurrences/refresh is lower so that it actually sends the email?

I was looking through the sensu-users list and found something you touched on last year about this but I’m not totally sure that’s what I need.

-James

On Jul 23, 2015, at 10:19 AM, James Jelinek jelinek@gmail.com wrote:

Ok so for fun I am trying to cat the mailer.json handler definition into the handler itself to see if it’s a ruby problem and I get this on output:

[root@monitor conf.d]# cat mailer.json |/opt/sensu/embedded/bin/ruby …/handlers/mailer.rb

/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `+’: no implicit conversion of nil into String (TypeError)

from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `filter_silenced'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:31:in `filter'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:54:in `block in <class:Handler>’

I think the problem is it’s passing NIL because I am not generating an actual event since calling from the command line.

Any thoughts on where else to look into the logs to see what/where this handler is firing or not firing and how to debug?

I’m kind of hitting a wall here.

-James

On Jul 23, 2015, at 10:09 AM, James Jelinek jelinek@gmail.com wrote:

Ok nevermind, sensu-handler is included in the gems when you install the sense-plugin gem. And it’s listed as /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb

So what else can I look at to see what’s going on here? I thank the stars for this mailing list, but a lot of this stuff should be more documented. That’s something I’m going to start doing and contribute.

Thanks for your help Kyle.

On Jul 23, 2015, at 10:05 AM, James Jelinek jelinek@gmail.com wrote:

One thing I noticed with this is that the gem sensu-handler (which is a requirement of mailer.rb) is not installed on my sensu master. When I try to install via omnibus like this

GEM_PATH=/opt/sensu/embedded/lib/ruby/gems/2.0.0 /opt/sensu/embedded/bin/gem install sensu-handler

I get the following:

ERROR: Could not find a valid gem ‘sensu-handler’ (>= 0) in any repository

I’m using embedded ruby (omnibus) and so far installing other gems like mail and timeout worked just fine with the gem install command. This could be why it’s not sending out email or firing properly.

Any thoughts?

On Jul 23, 2015, at 9:44 AM, Kyle Anderson kyle@xkyle.com wrote:

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek jelinek@gmail.com wrote:

Ok so it looks like I wasn’t specifying the handler type or command. So
here’s what I have so far:

{

“handlers”: {

“mailer”: {

"type": "pipe",

"command": "/etc/sensu/handlers/mailer.rb",

"admin_gui": "[http://25.149.38.106:5555/](http://25.149.38.106:5555/)",

"mail_from": "helpdesk@mydomain.com",

"mail_to": "james.jelinek@mydomain.com",

"smtp_address": "[smtp.gmail.com](http://smtp.gmail.com/)",

"smtp_port": "587",

"smtp_username": "helpdesk@mydomain.com",

"smtp_password": "password`",

"smtp_domain": "[mydomain.com](http://mydomain.com/)"

}

}

}

Does this look right? I have a check in check_disk.json that is failing and
it’s showing on the dashboard, but I’m not receiving any emails. Here’s a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I’m not sure if I have it setup right or not.

The docs are very sparse so I’m sort of flying blind here. Any help is
greatly appreciated guys!

-James


#12

I found where you can override the occurrences/refresh in sense-handler.rb

def filter_repeated

defaults = {

‘occurrences’ => 1,

‘interval’ => 30,

‘refresh’ => 60

}

Set the refresh to 60 will do every 6 occurrences. But it’s still not sending email even though the handler is firing.

I think this has something to do with the sense-handler gem but I’m really not sure on how to troubleshoot it.

···

On Jul 23, 2015, at 10:59 AM, James Jelinek jelinek@gmail.com wrote:

For a test I changed my command in the mailer.json to send regular mail and not hit the ruby script mailer.rb

“command”: "echo $(cat) > /tmp/mail.txt; mail -s “sensu alert” james.jelinek@streetcredsoftware.com < /tmp/mail.txt”

Every time the check_disk.json fires, it sends me an email about critical disk space (which is good!). So it looks like it’s firing the mailer handler properly, now I’m thinking it’s an issue with the mailer.rb handler and the sensu-handler gem keeping it from firing until there’s 180 occurrences.

How to overwrite this in the mailer.rb?

-James

On Jul 23, 2015, at 10:45 AM, James Jelinek jelinek@gmail.com wrote:

I’ve tried adding occurrences 3 to my check_disk.json to see if I can get the handler to fire like this:

{

“checks”: {

“disk”: {

“command”: “/etc/sensu/plugins/check-disk.sh -w 80 -c 90 -d /”,

“interval”: 10,

“handlers”: [“default”, “mailer”],

“occurrences”: 3,

“subscribers”: [

“test”

]

}

}

}

I’m still seeing the output from the handler of “only handling every 180 occurrences” even after restarting sensu-server and sense-api.

-James

On Jul 23, 2015, at 10:37 AM, James Jelinek jelinek@gmail.com wrote:

This shows the handler getting fired I think:

{“timestamp”:“2015-07-23T10:34:30.705274-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,"smtp_password”:“password”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: mfp/disk\n”]}

But the 180 occurrences I think is the problem. How can I override this for the handler to make sure the occurrences/refresh is lower so that it actually sends the email?

I was looking through the sensu-users list and found something you touched on last year about this but I’m not totally sure that’s what I need.

-James

On Jul 23, 2015, at 10:19 AM, James Jelinek jelinek@gmail.com wrote:

Ok so for fun I am trying to cat the mailer.json handler definition into the handler itself to see if it’s a ruby problem and I get this on output:

[root@monitor conf.d]# cat mailer.json |/opt/sensu/embedded/bin/ruby …/handlers/mailer.rb

/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `+’: no implicit conversion of nil into String (TypeError)

from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `filter_silenced'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:31:in `filter'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:54:in `block in <class:Handler>’

I think the problem is it’s passing NIL because I am not generating an actual event since calling from the command line.

Any thoughts on where else to look into the logs to see what/where this handler is firing or not firing and how to debug?

I’m kind of hitting a wall here.

-James

On Jul 23, 2015, at 10:09 AM, James Jelinek jelinek@gmail.com wrote:

Ok nevermind, sensu-handler is included in the gems when you install the sense-plugin gem. And it’s listed as /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb

So what else can I look at to see what’s going on here? I thank the stars for this mailing list, but a lot of this stuff should be more documented. That’s something I’m going to start doing and contribute.

Thanks for your help Kyle.

On Jul 23, 2015, at 10:05 AM, James Jelinek jelinek@gmail.com wrote:

One thing I noticed with this is that the gem sensu-handler (which is a requirement of mailer.rb) is not installed on my sensu master. When I try to install via omnibus like this

GEM_PATH=/opt/sensu/embedded/lib/ruby/gems/2.0.0 /opt/sensu/embedded/bin/gem install sensu-handler

I get the following:

ERROR: Could not find a valid gem ‘sensu-handler’ (>= 0) in any repository

I’m using embedded ruby (omnibus) and so far installing other gems like mail and timeout worked just fine with the gem install command. This could be why it’s not sending out email or firing properly.

Any thoughts?

On Jul 23, 2015, at 9:44 AM, Kyle Anderson kyle@xkyle.com wrote:

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek jelinek@gmail.com wrote:

Ok so it looks like I wasn’t specifying the handler type or command. So
here’s what I have so far:

{

“handlers”: {

“mailer”: {

"type": "pipe",

"command": "/etc/sensu/handlers/mailer.rb",

"admin_gui": "[http://25.149.38.106:5555/](http://25.149.38.106:5555/)",

"mail_from": "helpdesk@mydomain.com",

"mail_to": "james.jelinek@mydomain.com",

"smtp_address": "[smtp.gmail.com](http://smtp.gmail.com/)",

"smtp_port": "587",

"smtp_username": "helpdesk@mydomain.com",

"smtp_password": "password`",

"smtp_domain": "[mydomain.com](http://mydomain.com/)"

}

}

}

Does this look right? I have a check in check_disk.json that is failing and
it’s showing on the dashboard, but I’m not receiving any emails. Here’s a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I’m not sure if I have it setup right or not.

The docs are very sparse so I’m sort of flying blind here. Any help is
greatly appreciated guys!

-James


#13

Ok to try and simplify this, I setup Postfix locally and set my mailer.json to the following:

{

“handlers”: {

“mailer”: {

“type”: “pipe”,

“command”: “/etc/sensu/handlers/mailer.rb”,

“admin_gui”: “http://25.149.38.106:5555/”,

“mail_from”: “root@localhost”,

“mail_to”: "james.jelinek@mydomain.com",

“smtp_address”: “localhost”,

“smtp_port”: “25”,

“smtp_domain”: “monitor.mydomain.com

}

}

}

I still see this in the logs and emails aren’t firing.

{“timestamp”:“2015-07-23T12:16:49.549053-0500”,“level”:“info”,“message”:“processing event”,“event”:{“id”:“2de1a36d-413a-4a9a-a04b-c71cffba3057”,“client”:{“name”:“mfp”,“address”:“25.147.26.157”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437671712},“check”:{“command”:"/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90",“interval”:10,“handlers”:[“default”,“mailer”],“occurrences”:3,“subscribers”:[“test”],“name”:“disk”,“issued”:1437671809,“executed”:1437671730,“duration”:0.01,“output”:“CRITICAL - /dev/md1 space used=94% | ‘/dev/md1 usage’=94%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:374,“action”:“create”}}

{“timestamp”:“2015-07-23T12:16:49.562978-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“cat”,“name”:“default”},“output”:["{“id”:“2de1a36d-413a-4a9a-a04b-c71cffba3057”,“client”:{“name”:“mfp”,“address”:“25.147.26.157”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437671712},“check”:{“command”:"/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90",“interval”:10,“handlers”:[“default”,“mailer”],“occurrences”:3,“subscribers”:[“test”],“name”:“disk”,“issued”:1437671809,“executed”:1437671730,“duration”:0.01,“output”:“CRITICAL - /dev/md1 space used=94% | ‘/dev/md1 usage’=94%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:374,“action”:“create”}"]}

{“timestamp”:“2015-07-23T12:16:50.226913-0500”,“level”:“info”,“message”:“publishing check request”,“payload”:{“name”:“memory”,“issued”:1437671810,“command”:"/etc/sensu/plugins/check-mem.sh -w 1024 -c 512"},“subscribers”:[“test”]}

{“timestamp”:“2015-07-23T12:16:50.427752-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/”,“mail_from”:“root@localhost”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“localhost”,“smtp_port”:“25”,“smtp_domain”:“monitor.mydomain.com”,“name”:“mailer”},“output”:[“only handling every 18 occurrences: wwv/disk\n”]}

{“timestamp”:“2015-07-23T12:16:50.428061-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/”,“mail_from”:“root@localhost”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“localhost”,“smtp_port”:“25”,“smtp_domain”:“monitor.mydomain.com”,“name”:“mailer”},“output”:[“only handling every 18 occurrences: ws/disk\n”]}

{“timestamp”:“2015-07-23T12:16:50.711057-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/”,“mail_from”:“root@localhost”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“localhost”,“smtp_port”:“25”,“smtp_domain”:“monitor.mydomain.com”,“name”:“mailer”},“output”:[“only handling every 18 occurrences: mfp/disk\n”]}

I checked /var/log/maillog and I don’t see any activity, so it seems even with this simple MTA/Postfix setup it’s not firing the mailer.rb handler or if it is I have no idea how to debug it.

Sorry for all the emails but I’m really lost on this piece and it’s critical to get email notifications at this point.

Cheers,

James

···

On Jul 23, 2015, at 11:55 AM, James Jelinek jelinek@gmail.com wrote:

I found where you can override the occurrences/refresh in sense-handler.rb

def filter_repeated

defaults = {

‘occurrences’ => 1,

‘interval’ => 30,

‘refresh’ => 60

}

Set the refresh to 60 will do every 6 occurrences. But it’s still not sending email even though the handler is firing.

I think this has something to do with the sense-handler gem but I’m really not sure on how to troubleshoot it.

On Jul 23, 2015, at 10:59 AM, James Jelinek jelinek@gmail.com wrote:

For a test I changed my command in the mailer.json to send regular mail and not hit the ruby script mailer.rb

“command”: "echo $(cat) > /tmp/mail.txt; mail -s “sensu alert” james.jelinek@mydomain.com < /tmp/mail.txt”

Every time the check_disk.json fires, it sends me an email about critical disk space (which is good!). So it looks like it’s firing the mailer handler properly, now I’m thinking it’s an issue with the mailer.rb handler and the sensu-handler gem keeping it from firing until there’s 180 occurrences.

How to overwrite this in the mailer.rb?

-James

On Jul 23, 2015, at 10:45 AM, James Jelinek jelinek@gmail.com wrote:

I’ve tried adding occurrences 3 to my check_disk.json to see if I can get the handler to fire like this:

{

“checks”: {

“disk”: {

“command”: “/etc/sensu/plugins/check-disk.sh -w 80 -c 90 -d /”,

“interval”: 10,

“handlers”: [“default”, “mailer”],

“occurrences”: 3,

“subscribers”: [

“test”

]

}

}

}

I’m still seeing the output from the handler of “only handling every 180 occurrences” even after restarting sensu-server and sense-api.

-James

On Jul 23, 2015, at 10:37 AM, James Jelinek jelinek@gmail.com wrote:

This shows the handler getting fired I think:

{“timestamp”:“2015-07-23T10:34:30.705274-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,"smtp_password”:“password”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: mfp/disk\n”]}

But the 180 occurrences I think is the problem. How can I override this for the handler to make sure the occurrences/refresh is lower so that it actually sends the email?

I was looking through the sensu-users list and found something you touched on last year about this but I’m not totally sure that’s what I need.

-James

On Jul 23, 2015, at 10:19 AM, James Jelinek jelinek@gmail.com wrote:

Ok so for fun I am trying to cat the mailer.json handler definition into the handler itself to see if it’s a ruby problem and I get this on output:

[root@monitor conf.d]# cat mailer.json |/opt/sensu/embedded/bin/ruby …/handlers/mailer.rb

/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `+’: no implicit conversion of nil into String (TypeError)

from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `filter_silenced'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:31:in `filter'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:54:in `block in <class:Handler>’

I think the problem is it’s passing NIL because I am not generating an actual event since calling from the command line.

Any thoughts on where else to look into the logs to see what/where this handler is firing or not firing and how to debug?

I’m kind of hitting a wall here.

-James

On Jul 23, 2015, at 10:09 AM, James Jelinek jelinek@gmail.com wrote:

Ok nevermind, sensu-handler is included in the gems when you install the sense-plugin gem. And it’s listed as /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb

So what else can I look at to see what’s going on here? I thank the stars for this mailing list, but a lot of this stuff should be more documented. That’s something I’m going to start doing and contribute.

Thanks for your help Kyle.

On Jul 23, 2015, at 10:05 AM, James Jelinek jelinek@gmail.com wrote:

One thing I noticed with this is that the gem sensu-handler (which is a requirement of mailer.rb) is not installed on my sensu master. When I try to install via omnibus like this

GEM_PATH=/opt/sensu/embedded/lib/ruby/gems/2.0.0 /opt/sensu/embedded/bin/gem install sensu-handler

I get the following:

ERROR: Could not find a valid gem ‘sensu-handler’ (>= 0) in any repository

I’m using embedded ruby (omnibus) and so far installing other gems like mail and timeout worked just fine with the gem install command. This could be why it’s not sending out email or firing properly.

Any thoughts?

On Jul 23, 2015, at 9:44 AM, Kyle Anderson kyle@xkyle.com wrote:

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek jelinek@gmail.com wrote:

Ok so it looks like I wasn’t specifying the handler type or command. So
here’s what I have so far:

{

“handlers”: {

“mailer”: {

"type": "pipe",

"command": "/etc/sensu/handlers/mailer.rb",

"admin_gui": "[http://25.149.38.106:5555/](http://25.149.38.106:5555/)",

"mail_from": "helpdesk@mydomain.com",

"mail_to": "james.jelinek@mydomain.com",

"smtp_address": "[smtp.gmail.com](http://smtp.gmail.com/)",

"smtp_port": "587",

"smtp_username": "helpdesk@mydomain.com",

"smtp_password": "password`",

"smtp_domain": "[mydomain.com](http://mydomain.com/)"

}

}

}

Does this look right? I have a check in check_disk.json that is failing and
it’s showing on the dashboard, but I’m not receiving any emails. Here’s a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I’m not sure if I have it setup right or not.

The docs are very sparse so I’m sort of flying blind here. Any help is
greatly appreciated guys!

-James


#14

To further debug I added pager duty integration into sensu which was super easy. And I added it as a handler on check_disk.json. I’m still seeing the following like I did with the mailer handler

{“id”:“a1e807d2-425f-4891-b291-ed5bda5b6105”,“client”:{“name”:“mfp”,“address”:“25.147.26.157”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437673191},“check”:{“command”:"/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90",“interval”:10,“handlers”:[“default”,“pagerduty”,“mailer”],“occurrences”:3,“subscribers”:[“test”],“name”:“disk”,“issued”:1437673279,“executed”:1437673201,“duration”:0.01,“output”:“CRITICAL - /dev/md1 space used=94% | ‘/dev/md1 usage’=94%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:513,“action”:“create”}"]}

{“timestamp”:“2015-07-23T12:41:20.873279-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/pagerduty.rb",“api_key”:“2e24c101ef754ff6add39aaaffa846c9”,“name”:“pagerduty”},“output”:["only handling every 180 occurrences: ws/disk\n”]}

It’s not firing to pagerduty either. It keeps saying only handling every 180 occurrences. I think I’m going to pull my hair out now :slight_smile:

-James

···

On Jul 23, 2015, at 12:20 PM, James Jelinek jelinek@gmail.com wrote:

Ok to try and simplify this, I setup Postfix locally and set my mailer.json to the following:

{

“handlers”: {

“mailer”: {

“type”: “pipe”,

“command”: “/etc/sensu/handlers/mailer.rb”,

“admin_gui”: “http://25.149.38.106:5555/”,

“mail_from”: “root@localhost”,

“mail_to”: "james.jelinek@mydomain.com",

“smtp_address”: “localhost”,

“smtp_port”: “25”,

“smtp_domain”: “monitor.mydomain.com

}

}

}

I still see this in the logs and emails aren’t firing.

{“timestamp”:“2015-07-23T12:16:49.549053-0500”,“level”:“info”,“message”:“processing event”,“event”:{“id”:“2de1a36d-413a-4a9a-a04b-c71cffba3057”,“client”:{“name”:“mfp”,“address”:“25.147.26.157”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437671712},“check”:{“command”:"/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90",“interval”:10,“handlers”:[“default”,“mailer”],“occurrences”:3,“subscribers”:[“test”],“name”:“disk”,“issued”:1437671809,“executed”:1437671730,“duration”:0.01,“output”:“CRITICAL - /dev/md1 space used=94% | ‘/dev/md1 usage’=94%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:374,“action”:“create”}}

{“timestamp”:“2015-07-23T12:16:49.562978-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“cat”,“name”:“default”},“output”:["{“id”:“2de1a36d-413a-4a9a-a04b-c71cffba3057”,“client”:{“name”:“mfp”,“address”:“25.147.26.157”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437671712},“check”:{“command”:"/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90",“interval”:10,“handlers”:[“default”,“mailer”],“occurrences”:3,“subscribers”:[“test”],“name”:“disk”,“issued”:1437671809,“executed”:1437671730,“duration”:0.01,“output”:“CRITICAL - /dev/md1 space used=94% | ‘/dev/md1 usage’=94%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:374,“action”:“create”}"]}

{“timestamp”:“2015-07-23T12:16:50.226913-0500”,“level”:“info”,“message”:“publishing check request”,“payload”:{“name”:“memory”,“issued”:1437671810,“command”:"/etc/sensu/plugins/check-mem.sh -w 1024 -c 512"},“subscribers”:[“test”]}

{“timestamp”:“2015-07-23T12:16:50.427752-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/”,“mail_from”:“root@localhost”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“localhost”,“smtp_port”:“25”,“smtp_domain”:“monitor.mydomain.com”,“name”:“mailer”},“output”:[“only handling every 18 occurrences: wwv/disk\n”]}

{“timestamp”:“2015-07-23T12:16:50.428061-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/”,“mail_from”:“root@localhost”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“localhost”,“smtp_port”:“25”,“smtp_domain”:“monitor.mydomain.com”,“name”:“mailer”},“output”:[“only handling every 18 occurrences: ws/disk\n”]}

{“timestamp”:“2015-07-23T12:16:50.711057-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/”,“mail_from”:“root@localhost”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“localhost”,“smtp_port”:“25”,“smtp_domain”:“monitor.mydomain.com”,“name”:“mailer”},“output”:[“only handling every 18 occurrences: mfp/disk\n”]}

I checked /var/log/maillog and I don’t see any activity, so it seems even with this simple MTA/Postfix setup it’s not firing the mailer.rb handler or if it is I have no idea how to debug it.

Sorry for all the emails but I’m really lost on this piece and it’s critical to get email notifications at this point.

Cheers,

James

On Jul 23, 2015, at 11:55 AM, James Jelinek jelinek@gmail.com wrote:

I found where you can override the occurrences/refresh in sense-handler.rb

def filter_repeated

defaults = {

‘occurrences’ => 1,

‘interval’ => 30,

‘refresh’ => 60

}

Set the refresh to 60 will do every 6 occurrences. But it’s still not sending email even though the handler is firing.

I think this has something to do with the sense-handler gem but I’m really not sure on how to troubleshoot it.

On Jul 23, 2015, at 10:59 AM, James Jelinek jelinek@gmail.com wrote:

For a test I changed my command in the mailer.json to send regular mail and not hit the ruby script mailer.rb

“command”: "echo $(cat) > /tmp/mail.txt; mail -s “sensu alert” james.jelinek@mydomain.com < /tmp/mail.txt”

Every time the check_disk.json fires, it sends me an email about critical disk space (which is good!). So it looks like it’s firing the mailer handler properly, now I’m thinking it’s an issue with the mailer.rb handler and the sensu-handler gem keeping it from firing until there’s 180 occurrences.

How to overwrite this in the mailer.rb?

-James

On Jul 23, 2015, at 10:45 AM, James Jelinek jelinek@gmail.com wrote:

I’ve tried adding occurrences 3 to my check_disk.json to see if I can get the handler to fire like this:

{

“checks”: {

“disk”: {

“command”: “/etc/sensu/plugins/check-disk.sh -w 80 -c 90 -d /”,

“interval”: 10,

“handlers”: [“default”, “mailer”],

“occurrences”: 3,

“subscribers”: [

“test”

]

}

}

}

I’m still seeing the output from the handler of “only handling every 180 occurrences” even after restarting sensu-server and sense-api.

-James

On Jul 23, 2015, at 10:37 AM, James Jelinek jelinek@gmail.com wrote:

This shows the handler getting fired I think:

{“timestamp”:“2015-07-23T10:34:30.705274-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,"smtp_password”:“password”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: mfp/disk\n”]}

But the 180 occurrences I think is the problem. How can I override this for the handler to make sure the occurrences/refresh is lower so that it actually sends the email?

I was looking through the sensu-users list and found something you touched on last year about this but I’m not totally sure that’s what I need.

-James

On Jul 23, 2015, at 10:19 AM, James Jelinek jelinek@gmail.com wrote:

Ok so for fun I am trying to cat the mailer.json handler definition into the handler itself to see if it’s a ruby problem and I get this on output:

[root@monitor conf.d]# cat mailer.json |/opt/sensu/embedded/bin/ruby …/handlers/mailer.rb

/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `+’: no implicit conversion of nil into String (TypeError)

from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:128:in `filter_silenced'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:31:in `filter'
from /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:54:in `block in <class:Handler>’

I think the problem is it’s passing NIL because I am not generating an actual event since calling from the command line.

Any thoughts on where else to look into the logs to see what/where this handler is firing or not firing and how to debug?

I’m kind of hitting a wall here.

-James

On Jul 23, 2015, at 10:09 AM, James Jelinek jelinek@gmail.com wrote:

Ok nevermind, sensu-handler is included in the gems when you install the sense-plugin gem. And it’s listed as /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb

So what else can I look at to see what’s going on here? I thank the stars for this mailing list, but a lot of this stuff should be more documented. That’s something I’m going to start doing and contribute.

Thanks for your help Kyle.

On Jul 23, 2015, at 10:05 AM, James Jelinek jelinek@gmail.com wrote:

One thing I noticed with this is that the gem sensu-handler (which is a requirement of mailer.rb) is not installed on my sensu master. When I try to install via omnibus like this

GEM_PATH=/opt/sensu/embedded/lib/ruby/gems/2.0.0 /opt/sensu/embedded/bin/gem install sensu-handler

I get the following:

ERROR: Could not find a valid gem ‘sensu-handler’ (>= 0) in any repository

I’m using embedded ruby (omnibus) and so far installing other gems like mail and timeout worked just fine with the gem install command. This could be why it’s not sending out email or firing properly.

Any thoughts?

On Jul 23, 2015, at 9:44 AM, Kyle Anderson kyle@xkyle.com wrote:

https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions

Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.

On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek jelinek@gmail.com wrote:

Ok so it looks like I wasn’t specifying the handler type or command. So
here’s what I have so far:

{

“handlers”: {

“mailer”: {

“type”: “pipe”,

“command”: “/etc/sensu/handlers/mailer.rb”,

“admin_gui”: “http://25.149.38.106:5555/”,

“mail_from”: "helpdesk@mydomain.com",

“mail_to”: "james.jelinek@mydomain.com",

“smtp_address”: “smtp.gmail.com”,

“smtp_port”: “587”,

“smtp_username”: "helpdesk@mydomain.com",

“smtp_password”: “password`”,

“smtp_domain”: “mydomain.com

}

}

}

Does this look right? I have a check in check_disk.json that is failing and
it’s showing on the dashboard, but I’m not receiving any emails. Here’s a
sample of the log from /var/sensu/sensu-server.log

{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/mailer.rb",“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com",“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only
handling every 180 occurrences: mfp/disk\n”]}

It looks like the mailer check is firing and the handler is getting
executed, but I’m not sure if I have it setup right or not.

The docs are very sparse so I’m sort of flying blind here. Any help is
greatly appreciated guys!

-James


#15

Ok so after banging my head all day I jumped into IRC and talked with the dev team. Basically it looked like a filtering issue with the sensu-plugin/handler. So we neutered the filtering in the pagerduty handler and alerts started firing.

After further conversation, it came down to my config.json file needed the following to be set:

“api”: {

"host": "localhost",

"port": 4567

}

I had not set the host entry as it didn’t instruct me to do so in the sensu docs. By default sensu-plugin needs this to work right. So after setting this, removing the neutering, and setting my intervals/threshold low, I started firing off events!

Problem solved.

···

On Thursday, July 23, 2015 at 8:17:04 AM UTC-5, James Jelinek wrote:

I have sensu pretty much setup and running. I used the mailer.rb handler from the sensu-community-plugins repo and put it in /etc/sensu/handlers. I also configured mailer.json and put it in /etc/sensu/conf.d/mailer.json

Here is what my mailer.json looks like.

{

“mailer”: {

"admin_gui": "[http://25.149.38.106:5555/](http://25.149.38.106:5555/)",
"mail_from": "helpdesk@mydomain.com",
"mail_to": "james.jelinek@mydomain.com",
"smtp_address": "[smtp.gmail.com](http://smtp.gmail.com)",
"smtp_port": "587",
"smtp_username": "helpdesk@mydomain.com",
"smtp_password": "password`",
"smtp_domain": "[mydomain.com](http://mydomain.com)"

}

}

Then I add the handler to a check_disk.json check that I have setup as a test like so:

{

“checks”: {

"disk": {
  "command": "/etc/sensu/plugins/check-disk.sh -w 80 -c 90 -d /",
  "interval": 10,
  "handlers": ["default", "mailer"],
  "subscribers": [
    "test"
  ]
}

}

}

I tail the sensu-server.log and I get the following error when the check_disk check runs:

{“timestamp”:“2015-07-23T08:10:59.376218-0500”,“level”:“info”,“message”:“processing event”,“event”:{“id”:“c9474c1f-fbde-4605-b5d5-d40c70655fe1”,“client”:{“name”:“mfp”,“address”:“25.147.26.157”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437656971},“check”:{“command”:"/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90",“interval”:10,“handlers”:[“default”,“mailer”],“subscribers”:[“test”],“name”:“disk”,“issued”:1437657059,“executed”:1437656980,“duration”:0.011,“output”:“CRITICAL - /dev/md1 space used=94% | ‘/dev/md1 usage’=94%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:6531,“action”:“create”}}

{“timestamp”:“2015-07-23T08:10:59.378436-0500”,“level”:“error”,“message”:“unknown handler”,“handler_name”:“mailer”}

So even though I have it setup properly (I think) as a handler and a json definition, I get ,“level”:“error”,“message”:“unknown handler”,“handler_name”:“mailer”}

Any thoughts on why this is happening?

Thanks in advance guys!

-James