Apply alert fatigue filter on continuous failing event subsequence

Sensu Go
1

Feature Suggestion

We are researching to use sensu-go observability pipeline as an alert service. So call api to only generate Failing event to inpuPreformatted textt data. There is no Passing event being input to pipeline. So we don’t know the fixed interval of check as well.
Alert fatigue strategy we need:

  • Alert when occurrences == 3
  • Avoid duplicated alert in one hour/day

Possible Implementation

I found handle_when filter maybe can implement similar function. But it seems not possible in Sensu-Go when migrate from sensu core.
Could you offer and describe some other solutions if existed.

Context

This is my create event request body:

{
    "entity": {
        "entity_class": "proxy",
        "metadata": {
            "name": "{{$someHost}}",
            "namespace": "default"
        }
    },
    "check": {
        "issued": {{$timestamp}},
        "executed": {{$timestamp}},
        "output": "Server error",
        "state": "failing",
        "status": 2,
        "handlers": [
            "email"
        ],
        "metadata": {
            "name": "urlWatch",
            "labels":{
                "serverTypeId": "131",
                "zoneType": "URL Watch Zone"
            },
            "annotations":{
                "fatigue_check/occurrences": "4",
                "fatigue_check/interval": "600",
                "fatigue_check/allow_resolution": "false"
            }
        }
    },
    "timestamp": {{$timestamp}}
}
2

Github issue link: https://github.com/sensu/sensu-go/issues/4025

3

Hey!

This is an interesting use case. Have you thought about creating your own javascript filter asset and having it compare the check history status and timestamps? You can extend filters by providing your own javascript based runtime assets. And you can share those assets via Sensu Bonsai.

The Sensu Go event model records the return status and timestamp in the check history attribute, for up to the last 20 check executions I believe. So it should be possible for you to use that information in a javascript asset to implent the filter logic you need.

We don’t have a javascript filter asset template repository yet (we need need to put one together to help people get started with building their own filter assets) But if you want to look at an example of a javascript filter asset, take a look at:
https://github.com/nixwiz/sensu-go-fatigue-check-filter

Hopefully this helps.