Handler behavior when check goes from warning to critical

Wyatt_Walter · August 21, 2014, 9:27pm

Hi all,

Hopefully this is an easy one to figure out. We have setup a couple of handlers using the community ones for notifications, one which emails the team, and another which creates an event in PagerDuty. Both specify severities they should handle:

mailer.json:

“severities”: [

“ok”,

“warning”,

“critical”,

“unknown”

],

…
…

``

pagerduty.json:

…
“severities”: [

“critical”,

“ok”

],
…

``

Normally, this works quite well for us to only get paged about critical events. Where we’ve run into trouble recently is the situation where a check goes from warning to critical. The PagerDuty handler does fire after the alert hits critical, however, the event has already occurred more than once, and the refresh value hasn’t been hit yet, so it takes no action. ‘“output”:“only handling every 1440 occurrences”’ get logged.

Anyone has a solution to not filter the event when the event goes from a lower severity to a higher severity? We really don’t want to lower our refresh interval (to make this workable and actually have it fired we’d need it to be far too low anyway).

Kyle_Anderson · August 23, 2014, 4:08pm

Hmm, sensu really only has the concept of creating an event (non-zero)
or closing an event (zero status code):

github.com

sensu/sensu/blob/c81c50a54bb746cfc6be952e75ed5ffcda2ccfb1/lib/sensu/server.rb#L416-L428


      
          if check[:status] != 0 || is_flapping
            if previous_occurrence && check[:status] == previous_occurrence[:check][:status]
              event[:occurrences] = previous_occurrence[:occurrences] + 1
            end
            event[:action] = is_flapping ? :flapping : :create
            @redis.hset('events:' + client[:name], check[:name], MultiJson.dump(event)) do
              unless check[:handle] == false
                handle_event(event)
              end
            end
          elsif previous_occurrence
            event[:occurrences] = previous_occurrence[:occurrences]
            event[:action] = :resolve

You only get one filter per check, the only way I can think to do what
you want is to have two checks, one with the relaxed threshold that
goes to email, and the other one with a real critical threshold that
goes to PD (and doesn't fire early).

···

On Thu, Aug 21, 2014 at 2:27 PM, Wyatt Walter <wwalter@sugarcrm.com> wrote:

Hi all,

Hopefully this is an easy one to figure out. We have setup a couple of
handlers using the community ones for notifications, one which emails the
team, and another which creates an event in PagerDuty. Both specify
severities they should handle:

mailer.json:
...
      "severities": [
        "ok",
        "warning",
        "critical",
        "unknown"
      ],
...

pagerduty.json:
...
      "severities": [
        "critical",
        "ok"
      ],
...

Normally, this works quite well for us to only get paged about critical
events. Where we've run into trouble recently is the situation where a check
goes from warning to critical. The PagerDuty handler does fire after the
alert hits critical, however, the event has already occurred more than once,
and the refresh value hasn't been hit yet, so it takes no action.
'"output":"only handling every 1440 occurrences"' get logged.

Anyone has a solution to not filter the event when the event goes from a
lower severity to a higher severity? We really don't want to lower our
refresh interval (to make this workable and actually have it fired we'd need
it to be far too low anyway).

Wyatt_Walter · September 10, 2014, 4:40am

This is very weird. Today we had a check go from warning to critical and it acted as I was expecting. Maybe the behavior I saw before was just me being impatient when I didn’t see a page.

···

On Saturday, August 23, 2014 11:08:56 AM UTC-5, Kyle Anderson wrote:

Hmm, sensu really only has the concept of creating an event (non-zero)

or closing an event (zero status code):

https://github.com/sensu/sensu/blob/c81c50a54bb746cfc6be952e75ed5ffcda2ccfb1/lib/sensu/server.rb#L416-L428

You only get one filter per check, the only way I can think to do what

you want is to have two checks, one with the relaxed threshold that

goes to email, and the other one with a real critical threshold that

goes to PD (and doesn’t fire early).

On Thu, Aug 21, 2014 at 2:27 PM, Wyatt Walter wwa...@sugarcrm.com wrote:

Hi all,

Hopefully this is an easy one to figure out. We have setup a couple of

handlers using the community ones for notifications, one which emails the

team, and another which creates an event in PagerDuty. Both specify

severities they should handle:

mailer.json:

…
  "severities": [
    "ok",
    "warning",
    "critical",
    "unknown"
  ],
…

pagerduty.json:

…
  "severities": [
    "critical",
    "ok"
  ],
…

Normally, this works quite well for us to only get paged about critical

events. Where we’ve run into trouble recently is the situation where a check

goes from warning to critical. The PagerDuty handler does fire after the

alert hits critical, however, the event has already occurred more than once,

and the refresh value hasn’t been hit yet, so it takes no action.

‘“output”:“only handling every 1440 occurrences”’ get logged.

Anyone has a solution to not filter the event when the event goes from a

lower severity to a higher severity? We really don’t want to lower our

refresh interval (to make this workable and actually have it fired we’d need

it to be far too low anyway).

Topic		Replies	Views
Notification handlers, resolved events for certain severities Sensu Classic (EOL)	4	476	November 22, 2018
PagerDuty - Resolving a Critical, that has only gone back to a Warning (not to OK) Sensu Classic (EOL)	0	458	August 11, 2016
Filter seems is not working for the events that send to pagerduty Sensu Go	9	601	April 21, 2021
Handler not always sending resolve events Sensu Classic (EOL)	3	1453	December 3, 2019
How to tell a handler that it should only take action on state change? Sensu Classic (EOL)	17	1060	November 22, 2018

Handler behavior when check goes from warning to critical

Related topics