How to tell a handler that it should only take action on state change?

Sensu Classic (EOL)
1

I’d like to get an e-mail everytime some check changes its state by returning a different exit code.

I’m using the mailer handler from the community repository and I notice it inherits from Sensu::Handler (from sensu-plugin gem). Looking at its implementation and a the server logs it seems everytime my checker exits with an error code, the handlers get an event with action “create”.

By default the action won’t be called for multiple “create” events until the timeout specified in the “refresh” setting, so I get this message in the server logs:

“only handling every 60 occurrences” (interval is set to 30s and default refresh is 1800s or 30min)

But I don’t want to get an e-mail reminding me every 30 minutes (or any other time set in the “refresh” setting). I just want to get another e-mail once it’s resolved. Would that be possible?

Thanks in advance,

Rodrigo.

2

just set a very high refresh interval in check definition

···

On 03/19/2014 06:57 PM, Rodrigo Rosenfeld Rosas wrote:

I'd like to get an e-mail everytime some check changes its state by returning a different exit code.

I'm using the mailer handler from the community repository and I notice it inherits from Sensu::Handler (from sensu-plugin gem). Looking at its implementation and a the server logs it seems everytime my checker exits with an error code, the handlers get an event with action "create".

By default the action won't be called for multiple "create" events until the timeout specified in the "refresh" setting, so I get this message in the server logs:

"only handling every 60 occurrences" (interval is set to 30s and default refresh is 1800s or 30min)

But I don't want to get an e-mail reminding me every 30 minutes (or any other time set in the "refresh" setting). I just want to get another e-mail once it's resolved. Would that be possible?

Thanks in advance,
Rodrigo.

3

Ok, thank you. Sorry, I didn’t subscribe to this topic it seems, so I haven’t seen this answer before…

I’d really prefer for Sensu to only send me a notification of changes, but if this is not possible I’ll set a high refresh interval as a work-around.

Thanks,

Rodrigo.

···

Em quinta-feira, 20 de março de 2014 06h15min01s UTC-3, Alexander Piavlo escreveu:

just set a very high refresh interval in check definition

On 03/19/2014 06:57 PM, Rodrigo Rosenfeld Rosas wrote:

I’d like to get an e-mail everytime some check changes its state by
returning a different exit code.

I’m using the mailer handler from the community repository and I
notice it inherits from Sensu::Handler (from sensu-plugin gem).
Looking at its implementation and a the server logs it seems everytime
my checker exits with an error code, the handlers get an event with
action “create”.

By default the action won’t be called for multiple “create” events
until the timeout specified in the “refresh” setting, so I get this
message in the server logs:

“only handling every 60 occurrences” (interval is set to 30s and
default refresh is 1800s or 30min)

But I don’t want to get an e-mail reminding me every 30 minutes (or
any other time set in the “refresh” setting). I just want to get
another e-mail once it’s resolved. Would that be possible?

Thanks in advance,
Rodrigo.

4

Is there a better way of doing this rather than having a high refresh interval in the check. This is because I might want to know if the issue is resolved as soon as it is resolved. A high refresh interval on check will defer the resolved mail as well till the time the check is invoked again.

···

On Tuesday, 13 May 2014 05:42:20 UTC+5:30, Rodrigo Rosenfeld Rosas wrote:

Ok, thank you. Sorry, I didn’t subscribe to this topic it seems, so I haven’t seen this answer before…

I’d really prefer for Sensu to only send me a notification of changes, but if this is not possible I’ll set a high refresh interval as a work-around.

Thanks,

Rodrigo.

Em quinta-feira, 20 de março de 2014 06h15min01s UTC-3, Alexander Piavlo escreveu:

just set a very high refresh interval in check definition

On 03/19/2014 06:57 PM, Rodrigo Rosenfeld Rosas wrote:

I’d like to get an e-mail everytime some check changes its state by
returning a different exit code.

I’m using the mailer handler from the community repository and I
notice it inherits from Sensu::Handler (from sensu-plugin gem).
Looking at its implementation and a the server logs it seems everytime
my checker exits with an error code, the handlers get an event with
action “create”.

By default the action won’t be called for multiple “create” events
until the timeout specified in the “refresh” setting, so I get this
message in the server logs:

“only handling every 60 occurrences” (interval is set to 30s and
default refresh is 1800s or 30min)

But I don’t want to get an e-mail reminding me every 30 minutes (or
any other time set in the “refresh” setting). I just want to get
another e-mail once it’s resolved. Would that be possible?

Thanks in advance,
Rodrigo.

5

Also, the same goes for detecting an issue as well. A high refresh interval on the check could notify of the error quite late which could be undesirable.

···

On Tuesday, 5 August 2014 16:26:06 UTC+5:30, Abhishek Jain wrote:

Is there a better way of doing this rather than having a high refresh interval in the check. This is because I might want to know if the issue is resolved as soon as it is resolved. A high refresh interval on check will defer the resolved mail as well till the time the check is invoked again.

On Tuesday, 13 May 2014 05:42:20 UTC+5:30, Rodrigo Rosenfeld Rosas wrote:

Ok, thank you. Sorry, I didn’t subscribe to this topic it seems, so I haven’t seen this answer before…

I’d really prefer for Sensu to only send me a notification of changes, but if this is not possible I’ll set a high refresh interval as a work-around.

Thanks,

Rodrigo.

Em quinta-feira, 20 de março de 2014 06h15min01s UTC-3, Alexander Piavlo escreveu:

just set a very high refresh interval in check definition

On 03/19/2014 06:57 PM, Rodrigo Rosenfeld Rosas wrote:

I’d like to get an e-mail everytime some check changes its state by
returning a different exit code.

I’m using the mailer handler from the community repository and I
notice it inherits from Sensu::Handler (from sensu-plugin gem).
Looking at its implementation and a the server logs it seems everytime
my checker exits with an error code, the handlers get an event with
action “create”.

By default the action won’t be called for multiple “create” events
until the timeout specified in the “refresh” setting, so I get this
message in the server logs:

“only handling every 60 occurrences” (interval is set to 30s and
default refresh is 1800s or 30min)

But I don’t want to get an e-mail reminding me every 30 minutes (or
any other time set in the “refresh” setting). I just want to get
another e-mail once it’s resolved. Would that be possible?

Thanks in advance,
Rodrigo.

6

The refresh interval only affects things on the create stage:

Resolve events are not affected by it.

···

On Tue, Aug 5, 2014 at 3:57 AM, Abhishek Jain <abhi111jain@gmail.com> wrote:

Also, the same goes for detecting an issue as well. A high refresh interval
on the check could notify of the error quite late which could be
undesirable.

On Tuesday, 5 August 2014 16:26:06 UTC+5:30, Abhishek Jain wrote:

Is there a better way of doing this rather than having a high refresh
interval in the check. This is because I might want to know if the issue is
resolved as soon as it is resolved. A high refresh interval on check will
defer the resolved mail as well till the time the check is invoked again.

On Tuesday, 13 May 2014 05:42:20 UTC+5:30, Rodrigo Rosenfeld Rosas wrote:

Ok, thank you. Sorry, I didn't subscribe to this topic it seems, so I
haven't seen this answer before...

I'd really prefer for Sensu to only send me a notification of changes,
but if this is not possible I'll set a high refresh interval as a
work-around.

Thanks,
Rodrigo.

Em quinta-feira, 20 de março de 2014 06h15min01s UTC-3, Alexander Piavlo >>> escreveu:

just set a very high refresh interval in check definition

On 03/19/2014 06:57 PM, Rodrigo Rosenfeld Rosas wrote:
> I'd like to get an e-mail everytime some check changes its state by
> returning a different exit code.
>
> I'm using the mailer handler from the community repository and I
> notice it inherits from Sensu::Handler (from sensu-plugin gem).
> Looking at its implementation and a the server logs it seems everytime
> my checker exits with an error code, the handlers get an event with
> action "create".
>
> By default the action won't be called for multiple "create" events
> until the timeout specified in the "refresh" setting, so I get this
> message in the server logs:
>
> "only handling every 60 occurrences" (interval is set to 30s and
> default refresh is 1800s or 30min)
>
> But I don't want to get an e-mail reminding me every 30 minutes (or
> any other time set in the "refresh" setting). I just want to get
> another e-mail once it's resolved. Would that be possible?
>
> Thanks in advance,
> Rodrigo.
>

7

Thanks for the response Kyle. That solves the “resolve” part of things where it immediately gets handled. But I still have one query: If it affects the “create” event then even the handling of the first such create occurrence gets delayed by the refresh interval, right? What if I still want to be immediately notified when the first such occurrence happens but not be notified again till it gets resolved. Is there a work around for that as well, is my question!

···

On Tuesday, 5 August 2014 20:11:47 UTC+5:30, Kyle Anderson wrote:

The refresh interval only affects things on the create stage:
https://github.com/sensu/sensu-plugin/blob/master/lib/sensu-handler.rb#L99

Resolve events are not affected by it.

On Tue, Aug 5, 2014 at 3:57 AM, Abhishek Jain abhi1...@gmail.com wrote:

Also, the same goes for detecting an issue as well. A high refresh interval
on the check could notify of the error quite late which could be
undesirable.

On Tuesday, 5 August 2014 16:26:06 UTC+5:30, Abhishek Jain wrote:

Is there a better way of doing this rather than having a high refresh
interval in the check. This is because I might want to know if the issue is
resolved as soon as it is resolved. A high refresh interval on check will
defer the resolved mail as well till the time the check is invoked again.

On Tuesday, 13 May 2014 05:42:20 UTC+5:30, Rodrigo Rosenfeld Rosas wrote:

Ok, thank you. Sorry, I didn’t subscribe to this topic it seems, so I
haven’t seen this answer before…

I’d really prefer for Sensu to only send me a notification of changes,
but if this is not possible I’ll set a high refresh interval as a
work-around.

Thanks,
Rodrigo.

Em quinta-feira, 20 de março de 2014 06h15min01s UTC-3, Alexander Piavlo > >>> escreveu:

just set a very high refresh interval in check definition

On 03/19/2014 06:57 PM, Rodrigo Rosenfeld Rosas wrote:

I’d like to get an e-mail everytime some check changes its state by
returning a different exit code.

I’m using the mailer handler from the community repository and I
notice it inherits from Sensu::Handler (from sensu-plugin gem).
Looking at its implementation and a the server logs it seems everytime
my checker exits with an error code, the handlers get an event with
action “create”.

By default the action won’t be called for multiple “create” events
until the timeout specified in the “refresh” setting, so I get this
message in the server logs:

“only handling every 60 occurrences” (interval is set to 30s and
default refresh is 1800s or 30min)

But I don’t want to get an e-mail reminding me every 30 minutes (or
any other time set in the “refresh” setting). I just want to get
another e-mail once it’s resolved. Would that be possible?

Thanks in advance,
Rodrigo.

8

Yes, you are right. Refresh is the number of seconds before taking action:
https://github.com/sensu/sensu-docs/blame/6ece8e06ffc28217d0dd130fbab697d644710b1d/source/docs/0.13/checks.md#L382

And that is what it does here:

So....

This is a bit of a tricky situation. On the one hand, sensu-core has
to provide some sort of function for this (maybe) and give names to
these parameters, so it does, and the sensu-community handlers respect
them.

But, I personally never liked "refresh" "occurrences" and "interval".
So my team at my company decided as a group to say "screw it" and do
our own, we chose "check_every", "alert_after", and "realert_every",
and wrote our on filter_repeated function that works the way we
wanted. (create an event on the first occurrence after the alert_after
threshold. Re-alert every n time afterwards. We have a special "-1"
for exponential backoff. "0" would do what you would want in this
case, realert never.)

So the workaround is to take advantage of the flexibility of Sensu and
blaze your own trail, making handlers work for you, the way you want
them. (by overriding the filter_repeated function)
(Yes, I intend to open source mine to serve as examples so you can
copy. Stay tuned)

Long term, I don't know if there are plans to make these check
tune-ables any better or different. Changing these would break lots of
behavior. But, if you felt strongly that this is an issue, I would
open an issue and see what the sensu-core people say? (I'll vote for
you?)

···

--

So. Long story short: Yes, "refresh" delays the create event.
Yes, there is a work around, it is to... add a filter_repeated
function in your handler that does what you want.

def filter_repeated
  if @event['occurrences'] > 1 && @event['action'] == 'create'
    bail 'only handling the first occurrence of this check.'
  end
end

Sorry for the long-winded answer. Obviously this is something I've
thought a lot about and wrestled with.

On Tue, Aug 5, 2014 at 10:18 PM, Abhishek Jain <abhi111jain@gmail.com> wrote:

Thanks for the response Kyle. That solves the "resolve" part of things where
it immediately gets handled. But I still have one query: If it affects the
"create" event then even the handling of the first such create occurrence
gets delayed by the refresh interval, right? What if I still want to be
immediately notified when the first such occurrence happens but not be
notified again till it gets resolved. Is there a work around for that as
well, is my question!

On Tuesday, 5 August 2014 20:11:47 UTC+5:30, Kyle Anderson wrote:

The refresh interval only affects things on the create stage:
https://github.com/sensu/sensu-plugin/blob/master/lib/sensu-handler.rb#L99

Resolve events are not affected by it.

On Tue, Aug 5, 2014 at 3:57 AM, Abhishek Jain <abhi1...@gmail.com> wrote:
> Also, the same goes for detecting an issue as well. A high refresh
> interval
> on the check could notify of the error quite late which could be
> undesirable.
>
>
> On Tuesday, 5 August 2014 16:26:06 UTC+5:30, Abhishek Jain wrote:
>>
>> Is there a better way of doing this rather than having a high refresh
>> interval in the check. This is because I might want to know if the
>> issue is
>> resolved as soon as it is resolved. A high refresh interval on check
>> will
>> defer the resolved mail as well till the time the check is invoked
>> again.
>>
>> On Tuesday, 13 May 2014 05:42:20 UTC+5:30, Rodrigo Rosenfeld Rosas >> >> wrote:
>>>
>>> Ok, thank you. Sorry, I didn't subscribe to this topic it seems, so I
>>> haven't seen this answer before...
>>>
>>> I'd really prefer for Sensu to only send me a notification of changes,
>>> but if this is not possible I'll set a high refresh interval as a
>>> work-around.
>>>
>>> Thanks,
>>> Rodrigo.
>>>
>>> Em quinta-feira, 20 de março de 2014 06h15min01s UTC-3, Alexander >> >>> Piavlo >> >>> escreveu:
>>>>
>>>>
>>>> just set a very high refresh interval in check definition
>>>>
>>>> On 03/19/2014 06:57 PM, Rodrigo Rosenfeld Rosas wrote:
>>>> > I'd like to get an e-mail everytime some check changes its state by
>>>> > returning a different exit code.
>>>> >
>>>> > I'm using the mailer handler from the community repository and I
>>>> > notice it inherits from Sensu::Handler (from sensu-plugin gem).
>>>> > Looking at its implementation and a the server logs it seems
>>>> > everytime
>>>> > my checker exits with an error code, the handlers get an event with
>>>> > action "create".
>>>> >
>>>> > By default the action won't be called for multiple "create" events
>>>> > until the timeout specified in the "refresh" setting, so I get this
>>>> > message in the server logs:
>>>> >
>>>> > "only handling every 60 occurrences" (interval is set to 30s and
>>>> > default refresh is 1800s or 30min)
>>>> >
>>>> > But I don't want to get an e-mail reminding me every 30 minutes (or
>>>> > any other time set in the "refresh" setting). I just want to get
>>>> > another e-mail once it's resolved. Would that be possible?
>>>> >
>>>> > Thanks in advance,
>>>> > Rodrigo.
>>>> >
>>>>
>

9

Thanks Kyle for putting in the effort to describe the issue and the resolution so exhaustively. It helps a lot really! On the strongly felt issue part, yes, I do feel that this is something that sensu-core should support as it would be a common problem across many people using it. So I would urge you to still go ahead and raise it up with the sensu-core people. It would be great if you could also share a link to the issue that you would open. Thanks again!

···

On Wednesday, 6 August 2014 21:33:56 UTC+5:30, Kyle Anderson wrote:

Yes, you are right. Refresh is the number of seconds before taking action:
https://github.com/sensu/sensu-docs/blame/6ece8e06ffc28217d0dd130fbab697d644710b1d/source/docs/0.13/checks.md#L382

And that is what it does here:
https://github.com/sensu/sensu-plugin/blob/1c75c7f7f831f800395ba681ffe03de734a52ab7/lib/sensu-handler.rb#L100-L101

So…

This is a bit of a tricky situation. On the one hand, sensu-core has
to provide some sort of function for this (maybe) and give names to
these parameters, so it does, and the sensu-community handlers respect
them.

But, I personally never liked “refresh” “occurrences” and “interval”.
So my team at my company decided as a group to say “screw it” and do
our own, we chose “check_every”, “alert_after”, and “realert_every”,
and wrote our on filter_repeated function that works the way we
wanted. (create an event on the first occurrence after the alert_after
threshold. Re-alert every n time afterwards. We have a special “-1”
for exponential backoff. “0” would do what you would want in this
case, realert never.)

So the workaround is to take advantage of the flexibility of Sensu and
blaze your own trail, making handlers work for you, the way you want
them. (by overriding the filter_repeated function)
(Yes, I intend to open source mine to serve as examples so you can
copy. Stay tuned)

Long term, I don’t know if there are plans to make these check
tune-ables any better or different. Changing these would break lots of
behavior. But, if you felt strongly that this is an issue, I would
open an issue and see what the sensu-core people say? (I’ll vote for
you?)

–

So. Long story short: Yes, “refresh” delays the create event.
Yes, there is a work around, it is to… add a filter_repeated
function in your handler that does what you want.

def filter_repeated
if @event[‘occurrences’] > 1 && @event[‘action’] == ‘create’
bail ‘only handling the first occurrence of this check.’
end
end

Sorry for the long-winded answer. Obviously this is something I’ve
thought a lot about and wrestled with.

On Tue, Aug 5, 2014 at 10:18 PM, Abhishek Jain abhi1...@gmail.com wrote:

Thanks for the response Kyle. That solves the “resolve” part of things where
it immediately gets handled. But I still have one query: If it affects the
“create” event then even the handling of the first such create occurrence
gets delayed by the refresh interval, right? What if I still want to be
immediately notified when the first such occurrence happens but not be
notified again till it gets resolved. Is there a work around for that as
well, is my question!

On Tuesday, 5 August 2014 20:11:47 UTC+5:30, Kyle Anderson wrote:

The refresh interval only affects things on the create stage:
https://github.com/sensu/sensu-plugin/blob/master/lib/sensu-handler.rb#L99

Resolve events are not affected by it.

On Tue, Aug 5, 2014 at 3:57 AM, Abhishek Jain abhi1...@gmail.com wrote:

Also, the same goes for detecting an issue as well. A high refresh
interval
on the check could notify of the error quite late which could be
undesirable.

On Tuesday, 5 August 2014 16:26:06 UTC+5:30, Abhishek Jain wrote:

Is there a better way of doing this rather than having a high refresh
interval in the check. This is because I might want to know if the
issue is
resolved as soon as it is resolved. A high refresh interval on check
will
defer the resolved mail as well till the time the check is invoked
again.

On Tuesday, 13 May 2014 05:42:20 UTC+5:30, Rodrigo Rosenfeld Rosas > >> >> wrote:

Ok, thank you. Sorry, I didn’t subscribe to this topic it seems, so I
haven’t seen this answer before…

I’d really prefer for Sensu to only send me a notification of changes,
but if this is not possible I’ll set a high refresh interval as a
work-around.

Thanks,
Rodrigo.

Em quinta-feira, 20 de março de 2014 06h15min01s UTC-3, Alexander > >> >>> Piavlo > >> >>> escreveu:

just set a very high refresh interval in check definition

On 03/19/2014 06:57 PM, Rodrigo Rosenfeld Rosas wrote:

I’d like to get an e-mail everytime some check changes its state by
returning a different exit code.

I’m using the mailer handler from the community repository and I
notice it inherits from Sensu::Handler (from sensu-plugin gem).
Looking at its implementation and a the server logs it seems
everytime
my checker exits with an error code, the handlers get an event with
action “create”.

By default the action won’t be called for multiple “create” events
until the timeout specified in the “refresh” setting, so I get this
message in the server logs:

“only handling every 60 occurrences” (interval is set to 30s and
default refresh is 1800s or 30min)

But I don’t want to get an e-mail reminding me every 30 minutes (or
any other time set in the “refresh” setting). I just want to get
another e-mail once it’s resolved. Would that be possible?

Thanks in advance,
Rodrigo.

10

Hehe, I was thinking you would open the issue.
I'm not even sure it is a bug?

But, how about I'll get the code out first (I'll post to the list when
it is ready), and if the ideas/implementation in it are valuable, then
they can be adopted.
If not, that is just as good :slight_smile:

In my opinion, Sensu, being pre 1.0, is still up for breaking changes
(and it does).
I'm not afraid (personally) of breaking compatibility with all
existing handlers in a point-release if the sensu maintainers would
accept the pull request :slight_smile:

···

On Thu, Aug 7, 2014 at 1:01 AM, Abhishek Jain <abhi111jain@gmail.com> wrote:

Thanks Kyle for putting in the effort to describe the issue and the
resolution so exhaustively. It helps a lot really! On the strongly felt
issue part, yes, I do feel that this is something that sensu-core should
support as it would be a common problem across many people using it. So I
would urge you to still go ahead and raise it up with the sensu-core people.
It would be great if you could also share a link to the issue that you would
open. Thanks again!

On Wednesday, 6 August 2014 21:33:56 UTC+5:30, Kyle Anderson wrote:

Yes, you are right. Refresh is the number of seconds before taking action:

https://github.com/sensu/sensu-docs/blame/6ece8e06ffc28217d0dd130fbab697d644710b1d/source/docs/0.13/checks.md#L382

And that is what it does here:

https://github.com/sensu/sensu-plugin/blob/1c75c7f7f831f800395ba681ffe03de734a52ab7/lib/sensu-handler.rb#L100-L101

So....

This is a bit of a tricky situation. On the one hand, sensu-core has
to provide some sort of function for this (maybe) and give names to
these parameters, so it does, and the sensu-community handlers respect
them.

But, I personally never liked "refresh" "occurrences" and "interval".
So my team at my company decided as a group to say "screw it" and do
our own, we chose "check_every", "alert_after", and "realert_every",
and wrote our on filter_repeated function that works the way we
wanted. (create an event on the first occurrence after the alert_after
threshold. Re-alert every n time afterwards. We have a special "-1"
for exponential backoff. "0" would do what you would want in this
case, realert never.)

So the workaround is to take advantage of the flexibility of Sensu and
blaze your own trail, making handlers work for you, the way you want
them. (by overriding the filter_repeated function)
(Yes, I intend to open source mine to serve as examples so you can
copy. Stay tuned)

Long term, I don't know if there are plans to make these check
tune-ables any better or different. Changing these would break lots of
behavior. But, if you felt strongly that this is an issue, I would
open an issue and see what the sensu-core people say? (I'll vote for
you?)

--

So. Long story short: Yes, "refresh" delays the create event.
Yes, there is a work around, it is to... add a filter_repeated
function in your handler that does what you want.

def filter_repeated
  if @event['occurrences'] > 1 && @event['action'] == 'create'
    bail 'only handling the first occurrence of this check.'
  end
end

Sorry for the long-winded answer. Obviously this is something I've
thought a lot about and wrestled with.

On Tue, Aug 5, 2014 at 10:18 PM, Abhishek Jain <abhi1...@gmail.com> wrote:
> Thanks for the response Kyle. That solves the "resolve" part of things
> where
> it immediately gets handled. But I still have one query: If it affects
> the
> "create" event then even the handling of the first such create
> occurrence
> gets delayed by the refresh interval, right? What if I still want to be
> immediately notified when the first such occurrence happens but not be
> notified again till it gets resolved. Is there a work around for that as
> well, is my question!
>
>
> On Tuesday, 5 August 2014 20:11:47 UTC+5:30, Kyle Anderson wrote:
>>
>> The refresh interval only affects things on the create stage:
>>
>> https://github.com/sensu/sensu-plugin/blob/master/lib/sensu-handler.rb#L99
>>
>> Resolve events are not affected by it.
>>
>> On Tue, Aug 5, 2014 at 3:57 AM, Abhishek Jain <abhi1...@gmail.com> >> >> wrote:
>> > Also, the same goes for detecting an issue as well. A high refresh
>> > interval
>> > on the check could notify of the error quite late which could be
>> > undesirable.
>> >
>> >
>> > On Tuesday, 5 August 2014 16:26:06 UTC+5:30, Abhishek Jain wrote:
>> >>
>> >> Is there a better way of doing this rather than having a high
>> >> refresh
>> >> interval in the check. This is because I might want to know if the
>> >> issue is
>> >> resolved as soon as it is resolved. A high refresh interval on check
>> >> will
>> >> defer the resolved mail as well till the time the check is invoked
>> >> again.
>> >>
>> >> On Tuesday, 13 May 2014 05:42:20 UTC+5:30, Rodrigo Rosenfeld Rosas >> >> >> wrote:
>> >>>
>> >>> Ok, thank you. Sorry, I didn't subscribe to this topic it seems, so
>> >>> I
>> >>> haven't seen this answer before...
>> >>>
>> >>> I'd really prefer for Sensu to only send me a notification of
>> >>> changes,
>> >>> but if this is not possible I'll set a high refresh interval as a
>> >>> work-around.
>> >>>
>> >>> Thanks,
>> >>> Rodrigo.
>> >>>
>> >>> Em quinta-feira, 20 de março de 2014 06h15min01s UTC-3, Alexander >> >> >>> Piavlo >> >> >>> escreveu:
>> >>>>
>> >>>>
>> >>>> just set a very high refresh interval in check definition
>> >>>>
>> >>>> On 03/19/2014 06:57 PM, Rodrigo Rosenfeld Rosas wrote:
>> >>>> > I'd like to get an e-mail everytime some check changes its state
>> >>>> > by
>> >>>> > returning a different exit code.
>> >>>> >
>> >>>> > I'm using the mailer handler from the community repository and I
>> >>>> > notice it inherits from Sensu::Handler (from sensu-plugin gem).
>> >>>> > Looking at its implementation and a the server logs it seems
>> >>>> > everytime
>> >>>> > my checker exits with an error code, the handlers get an event
>> >>>> > with
>> >>>> > action "create".
>> >>>> >
>> >>>> > By default the action won't be called for multiple "create"
>> >>>> > events
>> >>>> > until the timeout specified in the "refresh" setting, so I get
>> >>>> > this
>> >>>> > message in the server logs:
>> >>>> >
>> >>>> > "only handling every 60 occurrences" (interval is set to 30s and
>> >>>> > default refresh is 1800s or 30min)
>> >>>> >
>> >>>> > But I don't want to get an e-mail reminding me every 30 minutes
>> >>>> > (or
>> >>>> > any other time set in the "refresh" setting). I just want to get
>> >>>> > another e-mail once it's resolved. Would that be possible?
>> >>>> >
>> >>>> > Thanks in advance,
>> >>>> > Rodrigo.
>> >>>> >
>> >>>>
>> >

11

Hi Kyle,

First off, I would like to apologize if I’m saying something stupid here, I’m very new to Sensu. (Btw, please ignore the e-mail adress, it is a postfix relay :wink: )

My question is, where can I find the sensu-handler.rb? Is it a file present in each installation? Or do I need to download this separately? (Which would be analogue to let’s say the Apache check)
If not present and a separate download is needed, what handler should I point to in my check? Now it is default and e-mail.

My goal is the same as you guys, when sensu says something is wrong: only send one mail and then leave the error be.

Thanks in advance for your help.

12

My question is, where can I find the sensu-handler.rb? Is it a file present
in each installation?

Yes. It is part of the sensu-plugin gem, it comes with sensu.

Or do I need to download this separately? (Which would

be analogue to let's say the Apache check)
If not present and a separate download is needed, what handlers should I use
for my check?

No. Almost all sensu handlers written in ruby already inherit the
class (class YourHandler < Sensu::Handler)
You should use whatever handlers you want. My favorite handler is my
JIRA handler:

When something is wrong I get a ticket, and when it is fixed, the
ticket auto-closes. No email spam. Painless alerts for low-urgency
issues.

My goal is the same as you guys, when sensu says something is wrong: only
send one mail and then leave the error be.

Yep, you can overwrite the filter_repeated behavior to however you see fit.
Here is an example of a handler that doesn't filter at all:

Here is my custom filter_repeated function. My favorite feature is
exponential backoff:

···

Thanks in advance for your help.

13

Hi Kyle,

Would it be possible to just overwrite that sensu-plugin gem?
I found the handler.rb in the directory: /opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.1.0/lib
However when I modified it by the filter_repeated you gave above the change had no affect, so I’m thinking I’m modifying the wrong file? (In the wrong directory?)
Could you point me to the one I should modify?

When no filter is defined by using “def filter_repeated; end”. I should get no e-mail correct?

BTW: Once I got plain e-mail working, I will definitely try out the JIRA plugin, that one looks promising, thanks for pointing me to its existence.

14

I wouldn't try to overwrite the gem, just the function who's behavior
you want to change (filter_repeated)

"def filter_repeated; end" makes the function do nothing, which means
it doesn't filter anything! (you will get every email)

I'll just write the code for you:

def filter_repeated
  if @event['action'] == 'create' && @event['occurrences'] != 1
    bail "Only firing on the first occurrence. We are at #
#{@event['occurrences']}"
  end
end

pretty sure this is what you want.

···

On Sat, Nov 22, 2014 at 9:30 AM, <sensuwarning@gmail.com> wrote:

Hi Kyle,

Would it be possible to just overwrite that sensu-plugin gem?
I found the handler.rb in the directory:
/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.1.0/lib
However when I modified it by the filter_repeated you gave above the change
had no affect, so I'm thinking I'm modifying the wrong file? (In the wrong
directory?)
Could you point me to the one I should modify?

When no filter is defined by using "def filter_repeated; end". I should get
no e-mail correct?

BTW: Once I got plain e-mail working, I will definitely try out the JIRA
plugin, that one looks promising, thanks for pointing me to its existence.

15

From what I would say my very basic knowledge of programming, this indeed seems like the thing I want.
However, the question remains, where do I place this “new function”?

On a side note: I don’t expect you to do all the work for me, maybe you could point me to a resource where I can find how to replace ruby functions?

Kind regards.

And thanks for the helpful answers you already gave me.

16

You can stick it right in your handler. The act of declaring the
function locally will mean the handler will use it over the inherited
one.

Here are two examples:

···

On Sun, Dec 7, 2014 at 9:06 AM, <sensuwarning@gmail.com> wrote:

From what I would say my very basic knowledge of programming, this indeed
seems like the thing I want.
However, the question remains, where do I place this "new function"?

On a side note: I don't expect you to do all the work for me, maybe you
could point me to a resource where I can find how to replace ruby functions?

Kind regards.

And thanks for the helpful answers you already gave me.

17

Hi,

Could you please tell me how to send notification only on error case.

In my header I put

{
  "handlers": {
    "error_mailer ": {
      "command": "/etc/sensu/handlers/handler-mailer.rb /etc/sensu/conf.d/mailer.json",
      "severities": [
        "warning",
        "critical",
        "unknown"
      ],
      "type": "pipe"
    }
  }
}

But I receive mail and when status Critical was changed to OK

середа, 6 серпня 2014 р. 18:03:56 UTC+2 користувач Kyle Anderson написав:

···

Yes, you are right. Refresh is the number of seconds before taking action:
https://github.com/sensu/sensu-docs/blame/6ece8e06ffc28217d0dd130fbab697d644710b1d/source/docs/0.13/checks.md#L382

And that is what it does here:
https://github.com/sensu/sensu-plugin/blob/1c75c7f7f831f800395ba681ffe03de734a52ab7/lib/sensu-handler.rb#L100-L101

So…

This is a bit of a tricky situation. On the one hand, sensu-core has
to provide some sort of function for this (maybe) and give names to
these parameters, so it does, and the sensu-community handlers respect
them.

But, I personally never liked “refresh” “occurrences” and “interval”.
So my team at my company decided as a group to say “screw it” and do
our own, we chose “check_every”, “alert_after”, and “realert_every”,
and wrote our on filter_repeated function that works the way we
wanted. (create an event on the first occurrence after the alert_after
threshold. Re-alert every n time afterwards. We have a special “-1”
for exponential backoff. “0” would do what you would want in this
case, realert never.)

So the workaround is to take advantage of the flexibility of Sensu and
blaze your own trail, making handlers work for you, the way you want
them. (by overriding the filter_repeated function)
(Yes, I intend to open source mine to serve as examples so you can
copy. Stay tuned)

Long term, I don’t know if there are plans to make these check
tune-ables any better or different. Changing these would break lots of
behavior. But, if you felt strongly that this is an issue, I would
open an issue and see what the sensu-core people say? (I’ll vote for
you?)

–

So. Long story short: Yes, “refresh” delays the create event.
Yes, there is a work around, it is to… add a filter_repeated
function in your handler that does what you want.

def filter_repeated
if @event[‘occurrences’] > 1 && @event[‘action’] == ‘create’
bail ‘only handling the first occurrence of this check.’
end
end

Sorry for the long-winded answer. Obviously this is something I’ve
thought a lot about and wrestled with.

On Tue, Aug 5, 2014 at 10:18 PM, Abhishek Jain abhi1...@gmail.com wrote:

Thanks for the response Kyle. That solves the “resolve” part of things where
it immediately gets handled. But I still have one query: If it affects the
“create” event then even the handling of the first such create occurrence
gets delayed by the refresh interval, right? What if I still want to be
immediately notified when the first such occurrence happens but not be
notified again till it gets resolved. Is there a work around for that as
well, is my question!

On Tuesday, 5 August 2014 20:11:47 UTC+5:30, Kyle Anderson wrote:

The refresh interval only affects things on the create stage:
https://github.com/sensu/sensu-plugin/blob/master/lib/sensu-handler.rb#L99

Resolve events are not affected by it.

On Tue, Aug 5, 2014 at 3:57 AM, Abhishek Jain abhi1...@gmail.com wrote:

Also, the same goes for detecting an issue as well. A high refresh
interval
on the check could notify of the error quite late which could be
undesirable.

On Tuesday, 5 August 2014 16:26:06 UTC+5:30, Abhishek Jain wrote:

Is there a better way of doing this rather than having a high refresh
interval in the check. This is because I might want to know if the
issue is
resolved as soon as it is resolved. A high refresh interval on check
will
defer the resolved mail as well till the time the check is invoked
again.

On Tuesday, 13 May 2014 05:42:20 UTC+5:30, Rodrigo Rosenfeld Rosas > >> >> wrote:

Ok, thank you. Sorry, I didn’t subscribe to this topic it seems, so I
haven’t seen this answer before…

I’d really prefer for Sensu to only send me a notification of changes,
but if this is not possible I’ll set a high refresh interval as a
work-around.

Thanks,
Rodrigo.

Em quinta-feira, 20 de março de 2014 06h15min01s UTC-3, Alexander > >> >>> Piavlo > >> >>> escreveu:

just set a very high refresh interval in check definition

On 03/19/2014 06:57 PM, Rodrigo Rosenfeld Rosas wrote:

I’d like to get an e-mail everytime some check changes its state by
returning a different exit code.

I’m using the mailer handler from the community repository and I
notice it inherits from Sensu::Handler (from sensu-plugin gem).
Looking at its implementation and a the server logs it seems
everytime
my checker exits with an error code, the handlers get an event with
action “create”.

By default the action won’t be called for multiple “create” events
until the timeout specified in the “refresh” setting, so I get this
message in the server logs:

“only handling every 60 occurrences” (interval is set to 30s and
default refresh is 1800s or 30min)

But I don’t want to get an e-mail reminding me every 30 minutes (or
any other time set in the “refresh” setting). I just want to get
another e-mail once it’s resolved. Would that be possible?

Thanks in advance,
Rodrigo.