Does anyone have troubleshooting tips for determining why a custom handler will not run in a timely manner? According to the client logs I have checks running properly (exiting with ‘0’) every 30 seconds.
Yesterday I was only seeing the handler run about once every five hours. Today the handler has yet to run. Is there a way to determine if the server is actually receiving the results of these checks? So far I’ve reviewed the sensu-client.log/sensu-server.log, and the rabbit log, but they have not been illuminating. The only thing I can say for sure is that there are new amqp connections coming in from the client periodically.
Handler:
[root@sensu02 rabbitmq]# cat /etc/sensu/conf.d/handlers/splunk.json
{
“handlers”: {
“splunk”: {
“type”: “pipe”,
“command”: “sensu_to_splunk > /tmp/s2s.out”
}
}
}
Client(canary) output:
[root@sencnl01 sensu]# tail -1 /var/log/sensu/sensu-client.log
{“timestamp”:“2014-01-14T15:27:34.617658-0800”,“level”:“info”,“message”:“publishing check result”,“payload”:{“client”:“sencnl0”,“check”:{“command”:“billing_order_check /app/monitor/conf/billing_order_check.json”,“handlers”:[“default”,“splunk”],“standalone”:true,“interval”:30,“name”:“billing_order_monitor”,“issued”:1389742053,“executed”:1389742053,“output”:"{\n “monitor_class”: “OrderCheck”,\n “timestamp”: “2014-01-14 15:27:34 -0800”,\n “canary_config”: {\n “file”: “/app/monitor/conf/billing_order_check.json”,\n “source”: “command line”\n },\n “messages”: {\n },\n “status”: {\n “code”: 0,\n “result”: “PASSED”,\n “summary”: “Order check: happy as a clam.”\n }\n}\n","status":0,“duration”:0.9389999999999999}}}
Any pointers would be appreciated.
Thanks!
-Travis
Changing the handler to type “metric” fixed the problem.
···
On Tuesday, January 14, 2014 3:32:23 PM UTC-8, Travis Bear wrote:
Does anyone have troubleshooting tips for determining why a custom handler will not run in a timely manner? According to the client logs I have checks running properly (exiting with ‘0’) every 30 seconds.
Yesterday I was only seeing the handler run about once every five hours. Today the handler has yet to run. Is there a way to determine if the server is actually receiving the results of these checks? So far I’ve reviewed the sensu-client.log/sensu-server.log, and the rabbit log, but they have not been illuminating. The only thing I can say for sure is that there are new amqp connections coming in from the client periodically.
Handler:
[root@sensu02 rabbitmq]# cat /etc/sensu/conf.d/handlers/splunk.json
{
“handlers”: {
“splunk”: {
“type”: “pipe”,
“command”: “sensu_to_splunk > /tmp/s2s.out”
}
}
}
Client(canary) output:
[root@sencnl01 sensu]# tail -1 /var/log/sensu/sensu-client.log
{“timestamp”:“2014-01-14T15:27:34.617658-0800”,“level”:“info”,“message”:“publishing check result”,“payload”:{“client”:“sencnl0”,“check”:{“command”:“billing_order_check /app/monitor/conf/billing_order_check.json”,“handlers”:[“default”,“splunk”],“standalone”:true,“interval”:30,“name”:“billing_order_monitor”,“issued”:1389742053,“executed”:1389742053,“output”:“{\n "monitor_class": "OrderCheck",\n "timestamp": "2014-01-14 15:27:34 -0800",\n "canary_config": {\n "file": "/app/monitor/conf/billing_order_check.json",\n "source": "command line"\n },\n "messages": {\n },\n "status": {\n "code": 0,\n "result": "PASSED",\n "summary": "Order check: happy as a clam."\n }\n}\n”,"status":0,“duration”:0.9389999999999999}}}
Any pointers would be appreciated.
Thanks!
-Travis
I think we figured out the source of this problem - the checks were being ran on the client and generating output, but since it was ‘ok’ (return code 0) the server never passed the event to the handlers. Changing the type to ‘metric’ caused the server to start handling the event.
···
On Wednesday, January 15, 2014 3:22:51 PM UTC-8, Travis Bear wrote:
Changing the handler to type “metric” fixed the problem.
On Tuesday, January 14, 2014 3:32:23 PM UTC-8, Travis Bear wrote:
Does anyone have troubleshooting tips for determining why a custom handler will not run in a timely manner? According to the client logs I have checks running properly (exiting with ‘0’) every 30 seconds.
Yesterday I was only seeing the handler run about once every five hours. Today the handler has yet to run. Is there a way to determine if the server is actually receiving the results of these checks? So far I’ve reviewed the sensu-client.log/sensu-server.log, and the rabbit log, but they have not been illuminating. The only thing I can say for sure is that there are new amqp connections coming in from the client periodically.
Handler:
[root@sensu02 rabbitmq]# cat /etc/sensu/conf.d/handlers/splunk.json
{
“handlers”: {
“splunk”: {
“type”: “pipe”,
“command”: “sensu_to_splunk > /tmp/s2s.out”
}
}
}
Client(canary) output:
[root@sencnl01 sensu]# tail -1 /var/log/sensu/sensu-client.log
{“timestamp”:“2014-01-14T15:27:34.617658-0800”,“level”:“info”,“message”:“publishing check result”,“payload”:{“client”:“sencnl0”,“check”:{“command”:“billing_order_check /app/monitor/conf/billing_order_check.json”,“handlers”:[“default”,“splunk”],“standalone”:true,“interval”:30,“name”:“billing_order_monitor”,“issued”:1389742053,“executed”:1389742053,“output”:“{\n "monitor_class": "OrderCheck",\n "timestamp": "2014-01-14 15:27:34 -0800",\n "canary_config": {\n "file": "/app/monitor/conf/billing_order_check.json",\n "source": "command line"\n },\n "messages": {\n },\n "status": {\n "code": 0,\n "result": "PASSED",\n "summary": "Order check: happy as a clam."\n }\n}\n”,"status":0,“duration”:0.9389999999999999}}}
Any pointers would be appreciated.
Thanks!
-Travis