Slack handler not firing

[Capt-Tom]

Hello, it’s me again.

I’m now attempting to hook up my handlers and have stalled. I’ve gone through a few pieces of documentation:
https://docs.sensu.io/sensu-go/latest/reference/handlers/#keepalive-event-handlers
https://docs.sensu.io/sensu-go/latest/getting-started/sample-app/#create-a-sensu-pipeline-to-slack
https://docs.sensu.io/sensu-go/latest/guides/send-slack-alerts/

Given my experience with the environment I feel I have an acceptable understanding of how all this should function, yet the handler is not firing, or at least no messages are arriving.

I have manually verified my webhook URL.

My assets:

root@default-ubuntu-1804:~# sensuctl asset list
            Name                                                           URL                                                  Hash
 ────────────────────────── ───────────────────────────────────────────────────────────────────────────────────────────────── ─────────
  b3fc-ruby-runtime          //assets.bonsai.sensu.io/.../b3f-ruby-runtime_2.1.0_ruby-2.4.4_debian_linux_amd64.tar.gz          a603f43
  b3fc-sensu-checks          //assets.bonsai.sensu.io/.../b3fc-sensu-checks_1.1.0_ruby-2.3.0_debian_linux_amd64.tar.gz         5fdb2df
  sensu-plugins-cpu-checks   //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.1.0_debian_linux_amd64.tar.gz             f0435fd
  sensu-plugins-monitoring   //assets.bonsai.sensu.io/.../monitoring-plugins-debian_2.3.0_linux_amd64.tar.gz                   9f72313
  sensu-plugins-syslog-ng    //assets.bonsai.sensu.io/.../sensu-plugins-syslog-ng_1.1.1_ruby-2.3.0_debian_linux_amd64.tar.gz   20ce4f1
  sensu-slack-handler        //assets.bonsai.sensu.io/.../b3f-ruby-runtime_2.1.0_ruby-2.4.4_debian_linux_amd64.tar.gz          dc4ef75

My checks, output truncated:

root@default-ubuntu-1804:~# sensuctl check list
          Name                                                           Command                                                  Interval   Cron   Timeout   TTL   Subscriptions    Handlers                     Assets                     Hooks   Publish?   Stdin?   Metric Format   Metric Handlers
 ─────────────────────── ─────────────────────────────────────────────────────────────────────────────────────────────────────── ────────── ────── ───────── ───── ──────────────── ────────── ──────────────────────────────────────────── ─────── ────────── ──────── ─────────────── ─────────────────
  check-syslog            metrics-syslog-ng.rb                                                                                          60              500   500   system           slack      sensu-plugins-syslog-ng,b3fc-ruby-runtime            true       false
  check_swap              check_swap -c 1 -w 1                                                                                          60               10   500   system           slack      sensu-plugins-monitoring                             true       false
  cpu-usage               check-cpu.rb -c 95 -w 90                                                                                      60               10   500   system           slack      b3fc-ruby-runtime,sensu-plugins-cpu-checks           true       false

My handlers:

root@default-ubuntu-1804:~# sensuctl handler list
    Name      Type   Timeout   Filters   Mutator                            Execute                                                                  Environment Variables                          Assets
 ─────────── ────── ───────── ───────── ───────── ──────────────────────────────────────────────────────────── ──────────────────────────────────────────────────────────────────────── ─────────────────────
  keepalive   set          0                       CALL: slack
  slack       pipe         0                       RUN:  sensu-slack-handler --channel '#channel-name'   SLACK_WEBHOOK_URL=https://hooks.slack.com/services/redacted           sensu-slack-handler

I’m running the sensu-rhel docker image. I’m creating this topic because I’ve tried both removing an agent to fire force the keepalive into a warning status, as well as running stress on a machine to pin its CPU. Neither resulted in a message posted.

sensu-backend version 5.18.0, build 670381e539b7af2ee9dff3863249950d71db7b53, built 2020-02-25T22:59:19Z`

I’m not receiving any log lines indicating an issue from the backend container, and I’ve checked topics here and on github.

I’ve tried both the -w flag in the sensu-slack-handler command as well as loading the webhook URL through the environmental variables array in the handler object.

[Capt-Tom]

Does the slack handler asset need to be included in the check runtime assets as well?

[todd]

No, the handler asset only needs to be included for the handler itself.

One thing I notice is that the URL for your sensu-slack-handler asset is pointed to a ruby runtime.

[Capt-Tom]

I consciously checked for this, I can’t believe I missed that.

Interestingly I did paste over the hash correctly. It matches https://bonsai.sensu.io/assets/sensu/sensu-slack-handler/.

Weird there was no mismatch error. Thank you very much for correcting my carelessness.

[todd]

Glad to have helped find it.

Now you’re going to want some filters on those handlers so you’re not getting every event into Slack.