Can you make a simple 'pipe' handler respect 'occurences' and 'refresh'?

Vincent_Cojot · March 7, 2016, 5:25pm

Dear All,

I’ve been struggling with that one…

I work in a limited environment (it’s a cool company but there are some things I cannot do) where:

the only version (at this time) of sensu that I can use is 0.16.0 (and it’s not the rpm available on Sensu’s website, unfortunately).
I cannot do ‘gem install’ of any kind (breaks supportability) but I can add rpm packages or simple self-contained scripts.

The latter prevents me from using most of the sensu-plugins developed by the community because they include a gem dependency of some kind at some step or another…

I’m limited to the following gems:

gem list|grep sensu

sensu (0.16.0)

sensu-em (2.4.0)

sensu-extension (1.0.0)

sensu-extensions (1.0.0)

sensu-logger (1.0.0)

sensu-settings (1.2.0)

sensu-spawn (1.1.0)

sensu-transport (2.4.0)

So I’m trying to have Sensu send alerts by e-mail and make it -NOT- send e-mail -EVERY- minute for every alert…

My checks include something like this:

“ntpd_health”: {

“handlers”: [“email”],

“command”: “{{ sensu_plugins_path }}/check_ntpmon --okpeers 1”,

“subscribers”: [

“overcld_generic”

],

“interval”: 60, “occurences”: 2, “refresh”: 1800

},

And the handler is a simple pipe:

{

“handlers”:

{

“email”: {

“command”: “python -mjson.tool|mail -v -s ‘Sensu alert’ -r sensu-mail@company.org -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S ssl-verify=ignore -S nss-config-dir=/etc/pki/nssdb recipient@company.org”,

“type”: “pipe”

}

Alerts by e-mail work… but of course ‘occurences’ and ‘refresh’ are being ignored and I get an e-mail per alert per minute…

(on https://sensuapp.org/docs/0.16/handlers, it -DOES- say that " Warning: These trivial (pipe) handlers will not respect the popular refresh and occurences settings ",

So I’m not very surprised…

Is there a way to make a the handler a bit ‘smarter’ ? Or alternatively is there a mail handler that only had one of the above ‘gems’ as a requirement and that I could use?

I’m so new to Sensu, I’m a bit at loss…

Thanks for reading,

Vincent S. Cojot

Kyle_Anderson · March 8, 2016, 3:28am

Can you install gems in a bundle? If so you could install gems in an
isolated bundle to your heart's content and then just point sensu at
those "binaries" ?
The steps would be:
- make a Gemfile with the gems you want to use (sensu-plugins-mailer, etc)
- bundle install --path something
- bundle exec mail_something ?
I'm sure this *could* work, but seems like a lot of work to get around
this limitation. Don't do this if you are not comfortable with
bundler/ruby.

Other alternatives are installing the gems you needs and converting
them to rpms with fpm (I've previously done this in a similar
environment to you)
So like:
fpm -s gem -t deb sensu-handler; rpm -Uvh sensu-plugin*.rpm
Without knowing more about your ruby setup it is hard to know if this
is a good idea or not.

Can you confirm when you run gem list | grep sensu, is that against
the omnibus gem? Or system gem? (sounds like it may be system gem
if it wasn't from a package from the website)

As a real last resort, as you see those handlers just read stdin ->
stdout. Your python code *could* do filtering on its own without the
sensu-handler gem.
(You would be re-implementing
https://github.com/sensu-plugins/sensu-plugin/blob/f1889faa5144650e2d8ca35816d659dd455c49e5/lib/sensu-handler.rb#L118,
and probably other filter_ functions)
Basically filling in the blanks here:
https://github.com/sensu/sensu-plugin-python/blob/master/sensu_plugin/handler.py#L17

···

On Mon, Mar 7, 2016 at 9:25 AM, Vincent Cojot <vincent.cojot@gmail.com> wrote:

Dear All,

I've been struggling with that one..

I work in a limited environment (it's a cool company but there are some
things I cannot do) where:

1) the only version (at this time) of sensu that I can use is 0.16.0 (and
it's not the rpm available on Sensu's website, unfortunately).

2) I cannot do 'gem install' of any kind (breaks supportability) but I can
add rpm packages or simple self-contained scripts.

The latter prevents me from using most of the sensu-plugins developed by the
community because they include a gem dependency of some kind at some step or
another..
I'm limited to the following gems:

# gem list|grep sensu
sensu (0.16.0)
sensu-em (2.4.0)
sensu-extension (1.0.0)
sensu-extensions (1.0.0)
sensu-logger (1.0.0)
sensu-settings (1.2.0)
sensu-spawn (1.1.0)
sensu-transport (2.4.0)

So I'm trying to have Sensu send alerts by e-mail and make it -NOT- send
e-mail -EVERY- minute for every alert..

My checks include something like this:
    "ntpd_health": {
      "handlers": ["email"],
      "command": "{{ sensu_plugins_path }}/check_ntpmon --okpeers 1",
      "subscribers": [
        "overcld_generic"
      ],
      "interval": 60, "occurences": 2, "refresh": 1800
    },

And the handler is a simple pipe:
{
   "handlers":
   {
    "email": {
        "command": "python -mjson.tool|mail -v -s 'Sensu alert' -r
sensu-mail@company.org -S smtp-use-starttls -S ssl-verify=ignore -S
smtp-auth=login -S ssl-verify=ignore -S nss-config-dir=/etc/pki/nssdb
recipient@company.org",
        "type": "pipe"
    }
}
}

Alerts by e-mail work.. but of course 'occurences' and 'refresh' are being
ignored and I get an e-mail per alert per minute..
(on https://sensuapp.org/docs/0.16/handlers, it -DOES- say that " Warning:
These trivial (pipe) handlers will not respect the popular refresh and
occurences settings ",
So I'm not very surprised..

Is there a way to make a the handler a bit 'smarter' ? Or alternatively is
there a mail handler that only had one of the above 'gems' as a requirement
and that I could use?

I'm so new to Sensu, I'm a bit at loss..

Thanks for reading,

Vincent S. Cojot

Vincent_Cojot · April 7, 2016, 8:42pm

Hi Kyle,
Thanks for your answer. I didn’t know you could turn gems into rpms. I will look into that.

For the record, here’s how I solved this issue (And I wish to thank David Moreau-Simard for his help).

I added a ‘filters’ field as well as a ‘severities’ field to my e-mail handler… Here’s the relevant part of that json:

{

“handlers”:

{

“email”: {

“command”: “python -mjson.tool|mail -v -S smtp=mx.company.com -s ‘Sensu alert (SC)’ -r …”,

“filters”: [

“filter_check_retry_occurrences”

],

“severities”: [

“ok”,

“warning”,

“critical”,

“unknown”

],

“type”: “pipe”

}

The filter (here ‘filter_check_retry_occurrences’) was written by David and it is quite simple in essence, adding ‘occurences’ checking where it isn’t already used.

The code was taken from this repo:

github.com

rdo-infra/ansible-role-rdomonitoring/blob/master/files/filter_check_retry_occurrences.rb

#   Copyright Red Hat, Inc. All Rights Reserved.
#
#   Licensed under the Apache License, Version 2.0 (the "License"); you may
#   not use this file except in compliance with the License. You may obtain
#   a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#   WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#   License for the specific language governing permissions and limitations
#   under the License.
#
#  Credit and thanks for workflow implementation to Yelp:
#  https://github.com/Yelp/sensu_handlers/blob/master/files/num_occurrences_filter.rb

module Sensu::Extension
  class CheckRetryOccurrences < Filter

This file has been truncated. show original

filter_check_retry_occurrences.rb had to be installed under /etc/sensu/extensions in order to be loaded properly…

That was it!

Kind regards,

Vincent

···

On Monday, March 7, 2016 at 10:28:30 PM UTC-5, Kyle Anderson wrote:

Can you install gems in a bundle? If so you could install gems in an
isolated bundle to your heart’s content and then just point sensu at
those “binaries” ?
The steps would be:

make a Gemfile with the gems you want to use (sensu-plugins-mailer, etc)

bundle install --path something

bundle exec mail_something ?
I’m sure this could work, but seems like a lot of work to get around
this limitation. Don’t do this if you are not comfortable with
bundler/ruby.

Other alternatives are installing the gems you needs and converting
them to rpms with fpm (I’ve previously done this in a similar
environment to you)
So like:
fpm -s gem -t deb sensu-handler; rpm -Uvh sensu-plugin*.rpm
Without knowing more about your ruby setup it is hard to know if this
is a good idea or not.

Can you confirm when you run gem list | grep sensu, is that against
the omnibus gem? Or system gem? (sounds like it may be system gem
if it wasn’t from a package from the website)

As a real last resort, as you see those handlers just read stdin →
stdout. Your python code could do filtering on its own without the
sensu-handler gem.
(You would be re-implementing
https://github.com/sensu-plugins/sensu-plugin/blob/f1889faa5144650e2d8ca35816d659dd455c49e5/lib/sensu-handler.rb#L118,
and probably other filter_ functions)
Basically filling in the blanks here:
https://github.com/sensu/sensu-plugin-python/blob/master/sensu_plugin/handler.py#L17

On Mon, Mar 7, 2016 at 9:25 AM, Vincent Cojot vincen...@gmail.com wrote:

Dear All,

I’ve been struggling with that one…

I work in a limited environment (it’s a cool company but there are some
things I cannot do) where:

the only version (at this time) of sensu that I can use is 0.16.0 (and
it’s not the rpm available on Sensu’s website, unfortunately).

I cannot do ‘gem install’ of any kind (breaks supportability) but I can
add rpm packages or simple self-contained scripts.

The latter prevents me from using most of the sensu-plugins developed by the
community because they include a gem dependency of some kind at some step or
another…
I’m limited to the following gems:

gem list|grep sensu

sensu (0.16.0)
sensu-em (2.4.0)
sensu-extension (1.0.0)
sensu-extensions (1.0.0)
sensu-logger (1.0.0)
sensu-settings (1.2.0)
sensu-spawn (1.1.0)
sensu-transport (2.4.0)

So I’m trying to have Sensu send alerts by e-mail and make it -NOT- send
e-mail -EVERY- minute for every alert…

My checks include something like this:
“ntpd_health”: {
“handlers”: [“email”],
“command”: “{{ sensu_plugins_path }}/check_ntpmon --okpeers 1”,
“subscribers”: [
“overcld_generic”
],
“interval”: 60, “occurences”: 2, “refresh”: 1800
},

And the handler is a simple pipe:
{
“handlers”:
{
“email”: {
“command”: “python -mjson.tool|mail -v -s ‘Sensu alert’ -r
sensu...@company.org -S smtp-use-starttls -S ssl-verify=ignore -S
smtp-auth=login -S ssl-verify=ignore -S nss-config-dir=/etc/pki/nssdb
reci...@company.org”,
“type”: “pipe”
}
}
}

Alerts by e-mail work… but of course ‘occurences’ and ‘refresh’ are being
ignored and I get an e-mail per alert per minute…
(on https://sensuapp.org/docs/0.16/handlers, it -DOES- say that " Warning:
These trivial (pipe) handlers will not respect the popular refresh and
occurences settings ",
So I’m not very surprised…

Is there a way to make a the handler a bit ‘smarter’ ? Or alternatively is
there a mail handler that only had one of the above ‘gems’ as a requirement
and that I could use?

I’m so new to Sensu, I’m a bit at loss…

Thanks for reading,

Vincent S. Cojot

Topic		Replies	Views
Initial handler setup Sensu Classic (EOL)	3	427	November 22, 2018
Does Pipe hander supports "occurrences" and "refresh" settings? Sensu Classic (EOL)	0	421	July 11, 2014
Embedded Ruby and Gems for Plugins and other questions Sensu Classic (EOL)	1	715	July 21, 2015
Sensu 0.29 occurences/refresh logic Sensu Classic (EOL)	4	525	November 22, 2018
Problem with handlers that inherit from Sensu::Handler Sensu Classic (EOL)	6	562	July 13, 2015

Can you make a simple 'pipe' handler respect 'occurences' and 'refresh'?

gem list|grep sensu

gem list|grep sensu

Related topics