Is_incident filter use when event.check.occurrences is greater than 1?

Sensu Go
1

I have one sensitive alert that fails every so often, might be due to network latency, and then it recovers immediately.

I wanted to tell Sensu to not alert me unless it fails at least 3 times, so I set event.check.occurrences to 3 in its filter.

Is there a way to tune a filter to be like is_incident for more than 1 occurrence? When I have it applied, I don’t get recovery notifications because it seems like is_incident only allows the first occurrence of the recovery through.

With is_incident removed though, I will get recovery notices where there were no alerts - ie if the check went critical just once, no alert, but then when it goes back to normal for 3 straight occurrences, I get a recovery notification.

2

Perhaps try this filter asset?

3

That looks promising, will try it out in the morning!

Thanks.

4

The more I look at it I’m just not comfortable with the way assets work now.

/var/cache/sensu/sensu-backend/68720865127fbc7c2fe16ca4d7bbf2a187a2df703f4b4acae1c93e8a66556e9079e1270521999b5871473e6c851f51b34097c54fdb8d18eedb7064df9019adc8/bin/sensu-slack-handler

That is too unwieldy and not human-readable, and I can’t see any reason why something that will be installed and used for the life of a monitoring system - like a slack handler or alert filter - should be in a volatile cache area anyway.

If you change Go to handle assets more the way Core did, with an embedded folder that wasn’t volatile, I would really warm up to them, but I don’t want to use them as-is and they’re too baked in to Go to avoid using them now.

Going to look in to Prometheus. Thank you for the responses and help.