Sensu Go / Slack Handler / Alerting in Slack like crazy

Sensu Go
1

I am running the latest Sensu Go 5.19.0 (open source compiled version), and when I add the slack handler using the version from Bonsai even with my definition having ‘is_incident’ as filter, it alerts to my slack channel like crazy.

Here is my Definition, not sure what I am missing, but right now I had to delete the handler because it just alerts like a mad-man.

---
type: Handler
api_version: core/v2
metadata:
  created_by: admin
  name: slack
  namespace: default
spec:
  command: sensu-slack-handler --channel '#alerting-it'
  env_vars:
  - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxxx
  filters:
    - is_incident
  handlers: null
  runtime_assets:
  - sensu-slack-handler
  secrets: null
  timeout: 0
  type: pipe

here is example of my channel, it keeps repeating the alerts my slack channel and was even posting OK messages for items that weren’t critical to begin with. It ended up posting over 100+ alerts the minute i added the handler?

As you can see just below it alerted twice within a few minutes of www.banksiteservices.com, check_lsyncd being critical?

image
image532×609 47 KB

2

Depending on your use cases, you can either use the fatigue plugin or create a ‘state_change_only’ filter and apply that as well to the handler(Note that the filter will only trigger alerts when the check transitions between states, it will not do hourly reminders like the fatigue plugin).

type: EventFilter
api_version: core/v2
metadata:
  name: state_change_only
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.occurrences == 1
  runtime_assets: null
1 Like