Disabling handlers

Sensu Go
1

Is there a way to disable handlers? My use case is to disable the pagerduty and email handlers during any planned activities on sensu servers, so that there is no noise on incidents or emails. Also, to enable it back post activity.

Regards,

2

Hey,

I believe the Sensu silencing resource might be what you are looking for.
You can create silences that will match all checks running on a specific entity for example, and as long as the handler definitions use the not_silenced built in filter as part of their configuration, the handler should not fire for any event that is associated with an matching silence.
Ref:
https://docs.sensu.io/sensu-go/latest/observability-pipeline/observe-process/silencing/

3

I wanted a way to disable notifications without creating silence, so I added a filter called emergency_notification_control.

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "emergency_notification_control"
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.has_check"
    ]
  }
}

Run this in for each namespace in your cluster, and add it to the filter list for each handler you want to control. You can then block notifications by toggling the filter action to “deny” either using the dashboard or by running a sensuctl filter update emergency_notification_control. For me, since some of the Sensu clusters I manage have over a dozen namespaces, I wrote a little script to walk through each one and change the filter action.

2 Likes
4

I think adding an event filter is great, does this filter keepalives as well.

5

Yes, it’ll also filter keepalives. As a recent example, we were experiencing some heavy iowait within our etcd cluster, and this resulted in some false-positive keepalive aIerts going out. Engaging the filter gave us time to fix etcd without any further panic-inducing false alarms.

2 Likes