# Handler behavior when check goes from warning to critical

**URL:** <https://discourse.sensu.io/t/handler-behavior-when-check-goes-from-warning-to-critical/237>\
**Category:** Sensu Classic (EOL)\
**Created:** [August 21, 2014, 9:27pm UTC](https://discourse.sensu.io/t/handler-behavior-when-check-goes-from-warning-to-critical/237 "2014-08-21T21:27:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wyatt\_Walter](https://avatars.discourse-cdn.com/v4/letter/w/ed655f/32.png) [@Wyatt\_Walter](https://discourse.sensu.io/u/Wyatt_Walter)\
**Post date:** [August 21, 2014, 9:27pm UTC](https://discourse.sensu.io/t/handler-behavior-when-check-goes-from-warning-to-critical/237/1 "2014-08-21T21:27:43Z")

</div>

Hi all,

Hopefully this is an easy one to figure out. We have setup a couple of handlers using the community ones for notifications, one which emails the team, and another which creates an event in PagerDuty. Both specify severities they should handle:

mailer.json:

“severities”: [

“ok”,

“warning”,

“critical”,

“unknown”

],

…  
…

``

pagerduty.json:

…  
“severities”: [

“critical”,

“ok”

],  
…

``

Normally, this works quite well for us to only get paged about critical events. Where we’ve run into trouble recently is the situation where a check goes from warning to critical. The PagerDuty handler does fire after the alert hits critical, however, the event has already occurred more than once, and the refresh value hasn’t been hit yet, so it takes no action. ‘“output”:“only handling every 1440 occurrences”’ get logged.

Anyone has a solution to not filter the event when the event goes from a lower severity to a higher severity? We really don’t want to lower our refresh interval (to make this workable and actually have it fired we’d need it to be far too low anyway).

---

<div class="post-metadata">

**Author:** ![Kyle\_Anderson](https://avatars.discourse-cdn.com/v4/letter/k/e5b9ba/32.png) [@Kyle\_Anderson](https://discourse.sensu.io/u/Kyle_Anderson)\
**Post date:** [August 23, 2014, 4:08pm UTC](https://discourse.sensu.io/t/handler-behavior-when-check-goes-from-warning-to-critical/237/2 "2014-08-23T16:08:53Z")

</div>

Hmm, sensu really only has the concept of creating an event (non-zero)  
or closing an event (zero status code):

> <https://github.com/sensu/sensu/blob/c81c50a54bb746cfc6be952e75ed5ffcda2ccfb1/lib/sensu/server.rb#L416-L428>

You only get one filter per check, the only way I can think to do what  
you want is to have two checks, one with the relaxed threshold that  
goes to email, and the other one with a real critical threshold that  
goes to PD (and doesn't fire early).

> **···**
>
> On Thu, Aug 21, 2014 at 2:27 PM, Wyatt Walter \<wwalter@sugarcrm.com\> wrote:
> 
> > Hi all,
> > 
> > Hopefully this is an easy one to figure out. We have setup a couple of  
> > handlers using the community ones for notifications, one which emails the  
> > team, and another which creates an event in PagerDuty. Both specify  
> > severities they should handle:
> > 
> > mailer.json:  
> > ...  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"severities": [  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"ok",  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"warning",  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"critical",  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"unknown"  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;],  
> > ...
> > 
> > pagerduty.json:  
> > ...  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"severities": [  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"critical",  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"ok"  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;],  
> > ...
> > 
> > Normally, this works quite well for us to only get paged about critical  
> > events. Where we've run into trouble recently is the situation where a check  
> > goes from warning to critical. The PagerDuty handler does fire after the  
> > alert hits critical, however, the event has already occurred more than once,  
> > and the refresh value hasn't been hit yet, so it takes no action.  
> > '"output":"only handling every 1440 occurrences"' get logged.
> > 
> > Anyone has a solution to not filter the event when the event goes from a  
> > lower severity to a higher severity? We really don't want to lower our  
> > refresh interval (to make this workable and actually have it fired we'd need  
> > it to be far too low anyway).

---

<div class="post-metadata">

**Author:** ![Wyatt\_Walter](https://avatars.discourse-cdn.com/v4/letter/w/ed655f/32.png) [@Wyatt\_Walter](https://discourse.sensu.io/u/Wyatt_Walter)\
**Post date:** [September 10, 2014, 4:40am UTC](https://discourse.sensu.io/t/handler-behavior-when-check-goes-from-warning-to-critical/237/3 "2014-09-10T04:40:11Z")

</div>

This is very weird. Today we had a check go from warning to critical and it acted as I was expecting. Maybe the behavior I saw before was just me being impatient when I didn’t see a page.

> **···**
>
> On Saturday, August 23, 2014 11:08:56 AM UTC-5, Kyle Anderson wrote:
> 
> > Hmm, sensu really only has the concept of creating an event (non-zero)
> > 
> > or closing an event (zero status code):
> > 
> > [https://github.com/sensu/sensu/blob/c81c50a54bb746cfc6be952e75ed5ffcda2ccfb1/lib/sensu/server.rb#L416-L428](https://github.com/sensu/sensu/blob/c81c50a54bb746cfc6be952e75ed5ffcda2ccfb1/lib/sensu/server.rb#L416-L428)
> > 
> > You only get one filter per check, the only way I can think to do what
> > 
> > you want is to have two checks, one with the relaxed threshold that
> > 
> > goes to email, and the other one with a real critical threshold that
> > 
> > goes to PD (and doesn’t fire early).
> > 
> > On Thu, Aug 21, 2014 at 2:27 PM, Wyatt Walter [wwa...@sugarcrm.com](mailto:wwa...@sugarcrm.com) wrote:
> > 
> > > Hi all,
> > 
> > > 
> > 
> > > Hopefully this is an easy one to figure out. We have setup a couple of
> > 
> > > handlers using the community ones for notifications, one which emails the
> > 
> > > team, and another which creates an event in PagerDuty. Both specify
> > 
> > > severities they should handle:
> > 
> > > 
> > 
> > > mailer.json:
> > 
> > > …
> > 
> > > ```
> > > "severities": [
> > > 
> > > ```
> > 
> > > ```
> > > "ok",
> > > 
> > > ```
> > 
> > > ```
> > > "warning",
> > > 
> > > ```
> > 
> > > ```
> > > "critical",
> > > 
> > > ```
> > 
> > > ```
> > > "unknown"
> > > 
> > > ```
> > 
> > > ```
> > > ],
> > > 
> > > ```
> > 
> > > …
> > 
> > > 
> > 
> > > pagerduty.json:
> > 
> > > …
> > 
> > > ```
> > > "severities": [
> > > 
> > > ```
> > 
> > > ```
> > > "critical",
> > > 
> > > ```
> > 
> > > ```
> > > "ok"
> > > 
> > > ```
> > 
> > > ```
> > > ],
> > > 
> > > ```
> > 
> > > …
> > 
> > > 
> > 
> > > Normally, this works quite well for us to only get paged about critical
> > 
> > > events. Where we’ve run into trouble recently is the situation where a check
> > 
> > > goes from warning to critical. The PagerDuty handler does fire after the
> > 
> > > alert hits critical, however, the event has already occurred more than once,
> > 
> > > and the refresh value hasn’t been hit yet, so it takes no action.
> > 
> > > ‘“output”:“only handling every 1440 occurrences”’ get logged.
> > 
> > > 
> > 
> > > Anyone has a solution to not filter the event when the event goes from a
> > 
> > > lower severity to a higher severity? We really don’t want to lower our
> > 
> > > refresh interval (to make this workable and actually have it fired we’d need
> > 
> > > it to be far too low anyway).
> > 
> > >