Ok here’s an outpour from my /var/sensu/sensu-server.log (stripping out the credentials). It looks like the check gets fired then on error the handler gets called. But I’m not sure exactly what I’m looking at here. It seems like it’s firing the check and then the handler. But I’m not sure what else to look at from there.
Where else in my log can I look?
{“timestamp”:“2015-07-23T09:57:29.326564-0500”,“level”:“info”,“message”:“processing event”,“event”:{“id”:“c7ddd6c8-c75b-496f-8b91-73c76f6992dc”,“client”:{“name”:“wwv”,“address”:“25.140.134.150”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437663447},“check”:{“command”:“/etc/sensu/plugins/check-disk.sh -w 80 -c 90 -d /”,“interval”:10,“handlers”:[“default”,“mailer”],“subscribers”:[“test”],“name”:“disk”,“issued”:1437663449,“executed”:1437663449,“duration”:0.009,“output”:“CRITICAL - / space used=97% | ‘/ usage’=97%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:7107,“action”:“create”}}
{“timestamp”:“2015-07-23T09:57:29.335682-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“cat”,“name”:“default”},“output”:[“{"id":"c7ddd6c8-c75b-496f-8b91-73c76f6992dc","client":{"name":"wwv","address":"25.140.134.150","subscriptions":["test"],"version":"0.20.0","timestamp":1437663447},"check":{"command":"/etc/sensu/plugins/check-disk.sh -w 80 -c 90 -d /","interval":10,"handlers":["default","mailer"],"subscribers":["test"],"name":"disk","issued":1437663449,"executed":1437663449,"duration":0.009,"output":"CRITICAL - / space used=97% | ‘/ usage’=97%;80;90;\n","status":2,"history":["2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2"],"total_state_change":0},"occurrences":7107,"action":"create"}”]}
{“timestamp”:“2015-07-23T09:57:29.378473-0500”,“level”:“info”,“message”:“processing event”,“event”:{“id”:“3784bf61-36af-4941-ac85-6861dd2bf67b”,“client”:{“name”:“mfp”,“address”:“25.147.26.157”,“subscriptions”:[“test”],“version”:“0.20.0”,“timestamp”:1437663368},“check”:{“command”:“/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90”,“interval”:10,“handlers”:[“default”,“mailer”],“subscribers”:[“test”],“name”:“disk”,“issued”:1437663449,“executed”:1437663370,“duration”:0.01,“output”:“CRITICAL - /dev/md1 space used=94% | ‘/dev/md1 usage’=94%;80;90;\n”,“status”:2,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”],“total_state_change”:0},“occurrences”:7102,“action”:“create”}}
{“timestamp”:“2015-07-23T09:57:29.391024-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“cat”,“name”:“default”},“output”:[“{"id":"3784bf61-36af-4941-ac85-6861dd2bf67b","client":{"name":"mfp","address":"25.147.26.157","subscriptions":["test"],"version":"0.20.0","timestamp":1437663368},"check":{"command":"/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90","interval":10,"handlers":["default","mailer"],"subscribers":["test"],"name":"disk","issued":1437663449,"executed":1437663370,"duration":0.01,"output":"CRITICAL - /dev/md1 space used=94% | ‘/dev/md1 usage’=94%;80;90;\n","status":2,"history":["2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2"],"total_state_change":0},"occurrences":7102,"action":"create"}”]}
{“timestamp”:“2015-07-23T09:57:30.148795-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“/etc/sensu/handlers/mailer.rb”,“admin_gui”:“http://25.149.38.106:5555/”,“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com",“smtp_address”:“smtp.gmail.com”,“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com",“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: wwv/disk\n”]}
{“timestamp”:“2015-07-23T09:57:30.222175-0500”,“level”:“info”,“message”:“publishing check request”,“payload”:{“name”:“memory”,“issued”:1437663450,“command”:“/etc/sensu/plugins/check-mem.sh -w 1024 -c 512”},“subscribers”:[“test”]}
{“timestamp”:“2015-07-23T09:57:30.536387-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“/etc/sensu/handlers/mailer.rb”,“admin_gui”:“http://25.149.38.106:5555/”,“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com",“smtp_address”:“smtp.gmail.com”,“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com",“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: ws/disk\n”]}
{“timestamp”:“2015-07-23T09:57:30.573765-0500”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“/etc/sensu/handlers/mailer.rb”,“admin_gui”:“http://25.149.38.106:5555/”,“mail_from”:"helpdesk@mydomain.com",“mail_to”:"james.jelinek@mydomain.com",“smtp_address”:“smtp.gmail.com”,“smtp_port”:“587”,“smtp_username”:"helpdesk@mydomain.com",“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:[“only handling every 180 occurrences: mfp/disk\n”]}
···
On Jul 23, 2015, at 9:44 AM, Kyle Anderson kyle@xkyle.com wrote:
https://sensuapp.org/docs/0.16/checks#common-custom-check-definitions
Occurrences is throttling the handler. Try to find a part of your logs
where it actually fired to get more info.
On Thu, Jul 23, 2015 at 6:31 AM, James Jelinek jelinek@gmail.com wrote:
Ok so it looks like I wasn’t specifying the handler type or command. So
here’s what I have so far:
{
“handlers”: {
“mailer”: {
"type": "pipe",
"command": "/etc/sensu/handlers/mailer.rb",
"admin_gui": "http://25.149.38.106:5555/",
"mail_from": "helpdesk@mydomain.com",
"mail_to": "james.jelinek@mydomain.com",
"smtp_address": "smtp.gmail.com",
"smtp_port": "587",
"smtp_username": "helpdesk@mydomain.com",
"smtp_password": "password`",
"smtp_domain": "mydomain.com"
}
}
}
Does this look right? I have a check in check_disk.json that is failing and
it’s showing on the dashboard, but I’m not receiving any emails. Here’s a
sample of the log from /var/sensu/sensu-server.log
{“timestamp”:“2015-07-23T08:27:30.261687-0500”,“level”:“info”,“message”:“handler
output”,“handler”:{“type”:“pipe”,“command”:“/etc/sensu/handlers/mailer.rb”,“admin_gui”:“http://25.149.38.106:5555/",“mail_from”:“helpdesk@mydomain.com”,“mail_to”:“james.jelinek@mydomain.com”,“smtp_address”:“smtp.gmail.com”,“smtp_port”:“587”,“smtp_username”:“helpdesk@mydomain.com”,“smtp_password”:“Password`”,“smtp_domain”:“mydomain.com”,“name”:“mailer”},“output”:["only
handling every 180 occurrences: mfp/disk\n”]}
It looks like the mailer check is firing and the handler is getting
executed, but I’m not sure if I have it setup right or not.
The docs are very sparse so I’m sort of flying blind here. Any help is
greatly appreciated guys!
-James
