Can you make a simple 'pipe' handler respect 'occurences' and 'refresh'?


#1

Dear All,

I’ve been struggling with that one…

I work in a limited environment (it’s a cool company but there are some things I cannot do) where:

  1. the only version (at this time) of sensu that I can use is 0.16.0 (and it’s not the rpm available on Sensu’s website, unfortunately).

  2. I cannot do ‘gem install’ of any kind (breaks supportability) but I can add rpm packages or simple self-contained scripts.

The latter prevents me from using most of the sensu-plugins developed by the community because they include a gem dependency of some kind at some step or another…

I’m limited to the following gems:

gem list|grep sensu

sensu (0.16.0)

sensu-em (2.4.0)

sensu-extension (1.0.0)

sensu-extensions (1.0.0)

sensu-logger (1.0.0)

sensu-settings (1.2.0)

sensu-spawn (1.1.0)

sensu-transport (2.4.0)

So I’m trying to have Sensu send alerts by e-mail and make it -NOT- send e-mail -EVERY- minute for every alert…

My checks include something like this:

“ntpd_health”: {

“handlers”: [“email”],

“command”: “{{ sensu_plugins_path }}/check_ntpmon --okpeers 1”,

“subscribers”: [

“overcld_generic”

],

“interval”: 60, “occurences”: 2, “refresh”: 1800

},

And the handler is a simple pipe:

{

“handlers”:

{

“email”: {

“command”: “python -mjson.tool|mail -v -s ‘Sensu alert’ -r sensu-mail@company.org -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S ssl-verify=ignore -S nss-config-dir=/etc/pki/nssdb recipient@company.org”,

“type”: “pipe”

}

}

}

Alerts by e-mail work… but of course ‘occurences’ and ‘refresh’ are being ignored and I get an e-mail per alert per minute…

(on https://sensuapp.org/docs/0.16/handlers, it -DOES- say that " Warning: These trivial (pipe) handlers will not respect the popular refresh and occurences settings ",

So I’m not very surprised…

Is there a way to make a the handler a bit ‘smarter’ ? Or alternatively is there a mail handler that only had one of the above ‘gems’ as a requirement and that I could use?

I’m so new to Sensu, I’m a bit at loss…

Thanks for reading,

Vincent S. Cojot


#2

Can you install gems in a bundle? If so you could install gems in an
isolated bundle to your heart's content and then just point sensu at
those "binaries" ?
The steps would be:
- make a Gemfile with the gems you want to use (sensu-plugins-mailer, etc)
- bundle install --path something
- bundle exec mail_something ?
I'm sure this *could* work, but seems like a lot of work to get around
this limitation. Don't do this if you are not comfortable with
bundler/ruby.

Other alternatives are installing the gems you needs and converting
them to rpms with fpm (I've previously done this in a similar
environment to you)
So like:
    fpm -s gem -t deb sensu-handler; rpm -Uvh sensu-plugin*.rpm
Without knowing more about your ruby setup it is hard to know if this
is a good idea or not.

Can you confirm when you run gem list | grep sensu, is that against
the omnibus gem? Or system gem? (sounds like it may be system gem
if it wasn't from a package from the website)

As a real last resort, as you see those handlers just read stdin ->
stdout. Your python code *could* do filtering on its own without the
sensu-handler gem.
(You would be re-implementing
https://github.com/sensu-plugins/sensu-plugin/blob/f1889faa5144650e2d8ca35816d659dd455c49e5/lib/sensu-handler.rb#L118,
and probably other filter_ functions)
Basically filling in the blanks here:
https://github.com/sensu/sensu-plugin-python/blob/master/sensu_plugin/handler.py#L17

···

On Mon, Mar 7, 2016 at 9:25 AM, Vincent Cojot <vincent.cojot@gmail.com> wrote:

Dear All,

I've been struggling with that one..

I work in a limited environment (it's a cool company but there are some
things I cannot do) where:

1) the only version (at this time) of sensu that I can use is 0.16.0 (and
it's not the rpm available on Sensu's website, unfortunately).

2) I cannot do 'gem install' of any kind (breaks supportability) but I can
add rpm packages or simple self-contained scripts.

The latter prevents me from using most of the sensu-plugins developed by the
community because they include a gem dependency of some kind at some step or
another..
I'm limited to the following gems:

# gem list|grep sensu
sensu (0.16.0)
sensu-em (2.4.0)
sensu-extension (1.0.0)
sensu-extensions (1.0.0)
sensu-logger (1.0.0)
sensu-settings (1.2.0)
sensu-spawn (1.1.0)
sensu-transport (2.4.0)

So I'm trying to have Sensu send alerts by e-mail and make it -NOT- send
e-mail -EVERY- minute for every alert..

My checks include something like this:
    "ntpd_health": {
      "handlers": ["email"],
      "command": "{{ sensu_plugins_path }}/check_ntpmon --okpeers 1",
      "subscribers": [
        "overcld_generic"
      ],
      "interval": 60, "occurences": 2, "refresh": 1800
    },

And the handler is a simple pipe:
{
   "handlers":
   {
    "email": {
        "command": "python -mjson.tool|mail -v -s 'Sensu alert' -r
sensu-mail@company.org -S smtp-use-starttls -S ssl-verify=ignore -S
smtp-auth=login -S ssl-verify=ignore -S nss-config-dir=/etc/pki/nssdb
recipient@company.org",
        "type": "pipe"
    }
}
}

Alerts by e-mail work.. but of course 'occurences' and 'refresh' are being
ignored and I get an e-mail per alert per minute..
(on https://sensuapp.org/docs/0.16/handlers, it -DOES- say that " Warning:
These trivial (pipe) handlers will not respect the popular refresh and
occurences settings ",
So I'm not very surprised..

Is there a way to make a the handler a bit 'smarter' ? Or alternatively is
there a mail handler that only had one of the above 'gems' as a requirement
and that I could use?

I'm so new to Sensu, I'm a bit at loss..

Thanks for reading,

Vincent S. Cojot


#3

Hi Kyle,
Thanks for your answer. I didn’t know you could turn gems into rpms. I will look into that.

For the record, here’s how I solved this issue (And I wish to thank David Moreau-Simard for his help).

I added a ‘filters’ field as well as a ‘severities’ field to my e-mail handler… Here’s the relevant part of that json:

{

“handlers”:

{

“email”: {

“command”: “python -mjson.tool|mail -v -S smtp=mx.company.com -s ‘Sensu alert (SC)’ -r …”,

“filters”: [

“filter_check_retry_occurrences”

],

“severities”: [

“ok”,

“warning”,

“critical”,

“unknown”

],

“type”: “pipe”

}

}

The filter (here ‘filter_check_retry_occurrences’) was written by David and it is quite simple in essence, adding ‘occurences’ checking where it isn’t already used.

The code was taken from this repo:

filter_check_retry_occurrences.rb had to be installed under /etc/sensu/extensions in order to be loaded properly…

That was it!

Kind regards,

Vincent

···

On Monday, March 7, 2016 at 10:28:30 PM UTC-5, Kyle Anderson wrote:

Can you install gems in a bundle? If so you could install gems in an
isolated bundle to your heart’s content and then just point sensu at
those “binaries” ?
The steps would be:

  • make a Gemfile with the gems you want to use (sensu-plugins-mailer, etc)
  • bundle install --path something
  • bundle exec mail_something ?
    I’m sure this could work, but seems like a lot of work to get around
    this limitation. Don’t do this if you are not comfortable with
    bundler/ruby.

Other alternatives are installing the gems you needs and converting
them to rpms with fpm (I’ve previously done this in a similar
environment to you)
So like:
fpm -s gem -t deb sensu-handler; rpm -Uvh sensu-plugin*.rpm
Without knowing more about your ruby setup it is hard to know if this
is a good idea or not.

Can you confirm when you run gem list | grep sensu, is that against
the omnibus gem? Or system gem? (sounds like it may be system gem
if it wasn’t from a package from the website)

As a real last resort, as you see those handlers just read stdin ->
stdout. Your python code could do filtering on its own without the
sensu-handler gem.
(You would be re-implementing
https://github.com/sensu-plugins/sensu-plugin/blob/f1889faa5144650e2d8ca35816d659dd455c49e5/lib/sensu-handler.rb#L118,
and probably other filter_ functions)
Basically filling in the blanks here:
https://github.com/sensu/sensu-plugin-python/blob/master/sensu_plugin/handler.py#L17

On Mon, Mar 7, 2016 at 9:25 AM, Vincent Cojot vincen...@gmail.com wrote:

Dear All,

I’ve been struggling with that one…

I work in a limited environment (it’s a cool company but there are some
things I cannot do) where:

  1. the only version (at this time) of sensu that I can use is 0.16.0 (and
    it’s not the rpm available on Sensu’s website, unfortunately).

  2. I cannot do ‘gem install’ of any kind (breaks supportability) but I can
    add rpm packages or simple self-contained scripts.

The latter prevents me from using most of the sensu-plugins developed by the
community because they include a gem dependency of some kind at some step or
another…
I’m limited to the following gems:

gem list|grep sensu

sensu (0.16.0)
sensu-em (2.4.0)
sensu-extension (1.0.0)
sensu-extensions (1.0.0)
sensu-logger (1.0.0)
sensu-settings (1.2.0)
sensu-spawn (1.1.0)
sensu-transport (2.4.0)

So I’m trying to have Sensu send alerts by e-mail and make it -NOT- send
e-mail -EVERY- minute for every alert…

My checks include something like this:
“ntpd_health”: {
“handlers”: [“email”],
“command”: “{{ sensu_plugins_path }}/check_ntpmon --okpeers 1”,
“subscribers”: [
“overcld_generic”
],
“interval”: 60, “occurences”: 2, “refresh”: 1800
},

And the handler is a simple pipe:
{
“handlers”:
{
“email”: {
“command”: “python -mjson.tool|mail -v -s ‘Sensu alert’ -r
sensu...@company.org -S smtp-use-starttls -S ssl-verify=ignore -S
smtp-auth=login -S ssl-verify=ignore -S nss-config-dir=/etc/pki/nssdb
reci...@company.org”,
“type”: “pipe”
}
}
}

Alerts by e-mail work… but of course ‘occurences’ and ‘refresh’ are being
ignored and I get an e-mail per alert per minute…
(on https://sensuapp.org/docs/0.16/handlers, it -DOES- say that " Warning:
These trivial (pipe) handlers will not respect the popular refresh and
occurences settings ",
So I’m not very surprised…

Is there a way to make a the handler a bit ‘smarter’ ? Or alternatively is
there a mail handler that only had one of the above ‘gems’ as a requirement
and that I could use?

I’m so new to Sensu, I’m a bit at loss…

Thanks for reading,

Vincent S. Cojot