Using multiple handler filters

Sensu Go
1

I have three filters for my handler: is_incident, not_silenced, occurrences_3

However, I also want to receive alerts for resolution events, so I added an OR condition to is_incident: is_incident || is_resolution

After adding that, I’m getting alerts for all events including ‘ok’ events, which makes me believe events are not getting filtered by the is_incident filter anymore. How should I add the OR condition?

2

The is_incident filter will allow is_resolution events to pass through. That is documented here. I think your occurrences_3 filter may be causing your issue.

Might I suggest the fatigue check filter? It should allow what you are wanting to work, with the side benefit of being tunable per-check and entity based on annotations.

3

Thanks for your reply. My occurrences filter is indeed the ‘problem’. Should I be using occurrences_watermark to achieve the following?

  • 2 occurrences (non-zero): handle event by email handler
  • 3 occurrences (non-zero): handle event by Pushover handler
  • State change from non-zero to zero: incident resolved event alert to email handler only
4

Here’s what I would do to achieve those specific objectives.

One filter for the email, this covers bullet points 1 & 3:

---
type: EventFilter
api_version: core/v2
metadata:
  name: filter_for_email
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.occurrences == 2 || event.is_resolution

Combined with the following defined for the email handler:

filters:
- is_incident
- not_silenced
- filter_for_email

And for the pushover handler covering bullet point 2:

---
type: EventFilter
api_version: core/v2
metadata:
  name: filter_for_pushover
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.occurrences == 3

With the following defined for the pushover handler:

filters:
- is_incident
- not_silenced
- filter_for_pushover