Using a single check definition to monitor multiple items

Hi there,

I have a check definition configured on the master to check for an SSL certificate expiry. On my client, I may have a variable number of SSL certificates that I’d like to monitor the expiry times for. What is the best way to approach this without having to create multiple check definitions on the server that do the same thing? Is there a way to “loop” over a variable number of certs?

I had this same problem last year but with dns resolution checks. I
contemplated submitting a patch to do 'ansible' like templating in the
commands (currently you can only do very basic token substitution) ... but,
I ended up just writing a python check looping over the hosts and resolving
each one reporting warnings/errors counters depending on which failed etc.

If you're using ansible/salt or some other configuration manager you could
use templates to auto generate multiple check json definitions without
having to actually write the checks separately yourself, so the check name
would be auto-generated as well, like:

ssl_host_check_<host>.json

Should be trivial with any configuration management tool. For my DNS use
case that was not a good fit, so a single check script worked (we pull the
list of domains dynamically from a db).

···

On Tue, Feb 28, 2017 at 6:39 PM, Hany Fahim <hany@vmfarms.com> wrote:

Hi there,

I have a check definition configured on the master to check for an SSL
certificate expiry. On my client, I may have a variable number of SSL
certificates that I'd like to monitor the expiry times for. What is the
best way to approach this without having to create multiple check
definitions on the server that do the same thing? Is there a way to "loop"
over a variable number of certs?

Thanks for the quick response! I was hoping to avoid doing that and using something native within Sensu itself. I’m currently doing something similar, but need this to scale to many thousands of systems, each with variable number of certs.

···

On Tuesday, February 28, 2017 at 6:49:03 PM UTC-5, Moises Silva wrote:

On Tue, Feb 28, 2017 at 6:39 PM, Hany Fahim ha...@vmfarms.com wrote:

Hi there,

I have a check definition configured on the master to check for an SSL certificate expiry. On my client, I may have a variable number of SSL certificates that I’d like to monitor the expiry times for. What is the best way to approach this without having to create multiple check definitions on the server that do the same thing? Is there a way to “loop” over a variable number of certs?

I had this same problem last year but with dns resolution checks. I contemplated submitting a patch to do ‘ansible’ like templating in the commands (currently you can only do very basic token substitution) … but, I ended up just writing a python check looping over the hosts and resolving each one reporting warnings/errors counters depending on which failed etc.

If you’re using ansible/salt or some other configuration manager you could use templates to auto generate multiple check json definitions without having to actually write the checks separately yourself, so the check name would be auto-generated as well, like:

ssl_host_check_.json

Should be trivial with any configuration management tool. For my DNS use case that was not a good fit, so a single check script worked (we pull the list of domains dynamically from a db).

Yeah for something like that I'd say may be use a Go program fetching your
list of domains from a db and spawn the checks in parallel (or a slice of
the total to not overwhelm the system, but that depends on where you're
running this of course). Even if you had a way to do it in a single sensu
check, thousands of requests wouldn't be scalable I think in a single
check, if one tcp connection hangs, you're other requests will too for a
while, etc.

If you prefer Ruby you can write the check as a sensu extension or even a a
completely external utility in any language triggered by a cron job that
only reports failures to sensu via the sensu client socket (which now
supports HTTP for reporting checks status).

···

On Tue, Feb 28, 2017 at 6:50 PM, Hany Fahim <hany@vmfarms.com> wrote:

Thanks for the quick response! I was hoping to avoid doing that and using
something native within Sensu itself. I'm currently doing something
similar, but need this to scale to many thousands of systems, each with
variable number of certs.

Looks like I’ll have to head down a route like this. Thanks for your help!

···

On Tuesday, February 28, 2017 at 6:55:40 PM UTC-5, Moises Silva wrote:

On Tue, Feb 28, 2017 at 6:50 PM, Hany Fahim ha...@vmfarms.com wrote:

Thanks for the quick response! I was hoping to avoid doing that and using something native within Sensu itself. I’m currently doing something similar, but need this to scale to many thousands of systems, each with variable number of certs.

Yeah for something like that I’d say may be use a Go program fetching your list of domains from a db and spawn the checks in parallel (or a slice of the total to not overwhelm the system, but that depends on where you’re running this of course). Even if you had a way to do it in a single sensu check, thousands of requests wouldn’t be scalable I think in a single check, if one tcp connection hangs, you’re other requests will too for a while, etc.

If you prefer Ruby you can write the check as a sensu extension or even a a completely external utility in any language triggered by a cron job that only reports failures to sensu via the sensu client socket (which now supports HTTP for reporting checks status).