Check that loops

I am working on creating my first check using the sensu plugin. I am calling a REST API, that returns a JSON file with an array of services names and the status that service is in. If the status is set to false, then I would send an alert.

I have all the logic working, what I am missing is how to not have the check exist when it finds one service down but has not finished looking at all the other services. I want to loop through all the services, and send an alert for any of the services that are down. It appears when you send critical “message” the check script will exit. I tried looking through various community plugins for an example of what I am trying to do but have not been able to find any.

The services that I pull back from the API can be dynamic, so passing the check script a list of services to check individually won’t work.

I notice that a metric check will not exit, if I changed the check to be a metric, is there a way to have a handler send an email if anything is marked critical, or do metric event handlers always send the output of the check no matter the results?

I recently wrote a check that does something similar and may be useful as a guide for you. Check out the send_{ok,warning,crtical} functions and the sensu_client_sock() function:

https://gist.github.com/joemiller/5806570

This check is executed by sensu, looks up all hosts (‘endpoints’) in our env, pings them with fping, and sends an event for each to the local sensu client socket via udp/3030. The events all come from the same client as far as sensu is concerned but the names of the generated check results are _ping_check.

···

On Tue, Jun 18, 2013 at 8:26 AM, Quenten Griffith qgriffith@gmail.com wrote:

I am working on creating my first check using the sensu plugin. I am calling a REST API, that returns a JSON file with an array of services names and the status that service is in. If the status is set to false, then I would send an alert.

I have all the logic working, what I am missing is how to not have the check exist when it finds one service down but has not finished looking at all the other services. I want to loop through all the services, and send an alert for any of the services that are down. It appears when you send critical “message” the check script will exit. I tried looking through various community plugins for an example of what I am trying to do but have not been able to find any.

The services that I pull back from the API can be dynamic, so passing the check script a list of services to check individually won’t work.

I notice that a metric check will not exit, if I changed the check to be a metric, is there a way to have a handler send an email if anything is marked critical, or do metric event handlers always send the output of the check no matter the results?

Thank you for the quick response and the code example. I should be able to use this from what I am trying to do.

···

On Tuesday, June 18, 2013 11:58:17 AM UTC-4, Joe Miller wrote:

I recently wrote a check that does something similar and may be useful as a guide for you. Check out the send_{ok,warning,crtical} functions and the sensu_client_sock() function:

https://gist.github.com/joemiller/5806570

This check is executed by sensu, looks up all hosts (‘endpoints’) in our env, pings them with fping, and sends an event for each to the local sensu client socket via udp/3030. The events all come from the same client as far as sensu is concerned but the names of the generated check results are _ping_check.

On Tue, Jun 18, 2013 at 8:26 AM, Quenten Griffith qgri...@gmail.com wrote:

I am working on creating my first check using the sensu plugin. I am calling a REST API, that returns a JSON file with an array of services names and the status that service is in. If the status is set to false, then I would send an alert.

I have all the logic working, what I am missing is how to not have the check exist when it finds one service down but has not finished looking at all the other services. I want to loop through all the services, and send an alert for any of the services that are down. It appears when you send critical “message” the check script will exit. I tried looking through various community plugins for an example of what I am trying to do but have not been able to find any.

The services that I pull back from the API can be dynamic, so passing the check script a list of services to check individually won’t work.

I notice that a metric check will not exit, if I changed the check to be a metric, is there a way to have a handler send an email if anything is marked critical, or do metric event handlers always send the output of the check no matter the results?

You can simplify it further if you don’t need the forkmanager bits. I included those because I was worried about down hosts causing the check to take too long so I wanted the fpings to run in parallel.

···

On Tue, Jun 18, 2013 at 10:26 AM, Quenten Griffith qgriffith@gmail.com wrote:

Thank you for the quick response and the code example. I should be able to use this from what I am trying to do.

On Tuesday, June 18, 2013 11:58:17 AM UTC-4, Joe Miller wrote:

I recently wrote a check that does something similar and may be useful as a guide for you. Check out the send_{ok,warning,crtical} functions and the sensu_client_sock() function:

https://gist.github.com/joemiller/5806570

This check is executed by sensu, looks up all hosts (‘endpoints’) in our env, pings them with fping, and sends an event for each to the local sensu client socket via udp/3030. The events all come from the same client as far as sensu is concerned but the names of the generated check results are _ping_check.

On Tue, Jun 18, 2013 at 8:26 AM, Quenten Griffith qgri...@gmail.com wrote:

I am working on creating my first check using the sensu plugin. I am calling a REST API, that returns a JSON file with an array of services names and the status that service is in. If the status is set to false, then I would send an alert.

I have all the logic working, what I am missing is how to not have the check exist when it finds one service down but has not finished looking at all the other services. I want to loop through all the services, and send an alert for any of the services that are down. It appears when you send critical “message” the check script will exit. I tried looking through various community plugins for an example of what I am trying to do but have not been able to find any.

The services that I pull back from the API can be dynamic, so passing the check script a list of services to check individually won’t work.

I notice that a metric check will not exit, if I changed the check to be a metric, is there a way to have a handler send an email if anything is marked critical, or do metric event handlers always send the output of the check no matter the results?

I have it working and all I had to do was use your methods that do the socket connect and the send_*. Now I loop through my results and if the status i false I send it to the send_critical. Thanks so much for sending me this. It saved me a lot of time, and saved me from making some ugly code. I was going to try to push things into hashes and then use output to form the message while reading the values from the hash for anything that failed.

···

On Tuesday, June 18, 2013 2:02:25 PM UTC-4, Joe Miller wrote:

You can simplify it further if you don’t need the forkmanager bits. I included those because I was worried about down hosts causing the check to take too long so I wanted the fpings to run in parallel.

On Tue, Jun 18, 2013 at 10:26 AM, Quenten Griffith qgri...@gmail.com wrote:

Thank you for the quick response and the code example. I should be able to use this from what I am trying to do.

On Tuesday, June 18, 2013 11:58:17 AM UTC-4, Joe Miller wrote:

I recently wrote a check that does something similar and may be useful as a guide for you. Check out the send_{ok,warning,crtical} functions and the sensu_client_sock() function:

https://gist.github.com/joemiller/5806570

This check is executed by sensu, looks up all hosts (‘endpoints’) in our env, pings them with fping, and sends an event for each to the local sensu client socket via udp/3030. The events all come from the same client as far as sensu is concerned but the names of the generated check results are _ping_check.

On Tue, Jun 18, 2013 at 8:26 AM, Quenten Griffith qgri...@gmail.com wrote:

I am working on creating my first check using the sensu plugin. I am calling a REST API, that returns a JSON file with an array of services names and the status that service is in. If the status is set to false, then I would send an alert.

I have all the logic working, what I am missing is how to not have the check exist when it finds one service down but has not finished looking at all the other services. I want to loop through all the services, and send an alert for any of the services that are down. It appears when you send critical “message” the check script will exit. I tried looking through various community plugins for an example of what I am trying to do but have not been able to find any.

The services that I pull back from the API can be dynamic, so passing the check script a list of services to check individually won’t work.

I notice that a metric check will not exit, if I changed the check to be a metric, is there a way to have a handler send an email if anything is marked critical, or do metric event handlers always send the output of the check no matter the results?

In my check json file I have handlers set

“handlers”: [

“default”, “mailer”, “logevent”

]

I thought that if I changed your code to use option :handlers instead of option :handler, and then changed the send_* methods to send ‘handlers’ => config[:handlers] it would send the results from my check json file. However it seems to be ignoring what I have set, and using the default of default.

···

On Tuesday, June 18, 2013 2:02:25 PM UTC-4, Joe Miller wrote:

You can simplify it further if you don’t need the forkmanager bits. I included those because I was worried about down hosts causing the check to take too long so I wanted the fpings to run in parallel.

On Tue, Jun 18, 2013 at 10:26 AM, Quenten Griffith qgri...@gmail.com wrote:

Thank you for the quick response and the code example. I should be able to use this from what I am trying to do.

On Tuesday, June 18, 2013 11:58:17 AM UTC-4, Joe Miller wrote:

I recently wrote a check that does something similar and may be useful as a guide for you. Check out the send_{ok,warning,crtical} functions and the sensu_client_sock() function:

https://gist.github.com/joemiller/5806570

This check is executed by sensu, looks up all hosts (‘endpoints’) in our env, pings them with fping, and sends an event for each to the local sensu client socket via udp/3030. The events all come from the same client as far as sensu is concerned but the names of the generated check results are _ping_check.

On Tue, Jun 18, 2013 at 8:26 AM, Quenten Griffith qgri...@gmail.com wrote:

I am working on creating my first check using the sensu plugin. I am calling a REST API, that returns a JSON file with an array of services names and the status that service is in. If the status is set to false, then I would send an alert.

I have all the logic working, what I am missing is how to not have the check exist when it finds one service down but has not finished looking at all the other services. I want to loop through all the services, and send an alert for any of the services that are down. It appears when you send critical “message” the check script will exit. I tried looking through various community plugins for an example of what I am trying to do but have not been able to find any.

The services that I pull back from the API can be dynamic, so passing the check script a list of services to check individually won’t work.

I notice that a metric check will not exit, if I changed the check to be a metric, is there a way to have a handler send an email if anything is marked critical, or do metric event handlers always send the output of the check no matter the results?

What does the actual generated json look like that is sent to sensu-client? Check the client logs (triggering debug logging with sig USR1 might help too). Or run netcat on port 3031 and have the check send there.

···

On Tue, Jun 18, 2013 at 1:00 PM, Quenten Griffith qgriffith@gmail.com wrote:

In my check json file I have handlers set

“handlers”: [

“default”, “mailer”, “logevent”

]

I thought that if I changed your code to use option :handlers instead of option :handler, and then changed the send_* methods to send ‘handlers’ => config[:handlers] it would send the results from my check json file. However it seems to be ignoring what I have set, and using the default of default.

On Tuesday, June 18, 2013 2:02:25 PM UTC-4, Joe Miller wrote:

You can simplify it further if you don’t need the forkmanager bits. I included those because I was worried about down hosts causing the check to take too long so I wanted the fpings to run in parallel.

On Tue, Jun 18, 2013 at 10:26 AM, Quenten Griffith qgri...@gmail.com wrote:

Thank you for the quick response and the code example. I should be able to use this from what I am trying to do.

On Tuesday, June 18, 2013 11:58:17 AM UTC-4, Joe Miller wrote:

I recently wrote a check that does something similar and may be useful as a guide for you. Check out the send_{ok,warning,crtical} functions and the sensu_client_sock() function:

https://gist.github.com/joemiller/5806570

This check is executed by sensu, looks up all hosts (‘endpoints’) in our env, pings them with fping, and sends an event for each to the local sensu client socket via udp/3030. The events all come from the same client as far as sensu is concerned but the names of the generated check results are _ping_check.

On Tue, Jun 18, 2013 at 8:26 AM, Quenten Griffith qgri...@gmail.com wrote:

I am working on creating my first check using the sensu plugin. I am calling a REST API, that returns a JSON file with an array of services names and the status that service is in. If the status is set to false, then I would send an alert.

I have all the logic working, what I am missing is how to not have the check exist when it finds one service down but has not finished looking at all the other services. I want to loop through all the services, and send an alert for any of the services that are down. It appears when you send critical “message” the check script will exit. I tried looking through various community plugins for an example of what I am trying to do but have not been able to find any.

The services that I pull back from the API can be dynamic, so passing the check script a list of services to check individually won’t work.

I notice that a metric check will not exit, if I changed the check to be a metric, is there a way to have a handler send an email if anything is marked critical, or do metric event handlers always send the output of the check no matter the results?