sending WARN and CRITICAL to separate Slack channels


#1

Hi everyone
I am required to report WARN and CRITICAL messages to separate Slack channels, did a preliminary search for this but couldnt find much. In most cases in checks we specify slack handler and that sends all messages to one channel. Is there a way to send notifications to different slack channels depending on the severity of the alert?

Thanks


#2

One way to do this with stock handlers is to configure two handlers:
one handler for each channel. Then configure the different handlers
respond to different severities:
https://sensuapp.org/docs/latest/handlers#definition-attributes (see
"severities")
So you might have a slack-criticals handler that operates on CRITICAL
and slack-warnings handler that operates on.. everything else? It
would be up to you.

It does require you to have each check send to both handlers. (or make
a handler set)

At Yelp we have lots of teams and lots of channels, so it was worth it
for us to handle this with a custom handler, sending "paging" alerts
to one channel and non-paging alerts to a different one, depending on
your team:

···

On Sat, Jun 27, 2015 at 12:16 AM, <aiman@flipagram.com> wrote:

Hi everyone
I am required to report WARN and CRITICAL messages to separate Slack
channels, did a preliminary search for this but couldnt find much. In most
cases in checks we specify slack handler and that sends all messages to one
channel. Is there a way to send notifications to different slack channels
depending on the severity of the alert?

Thanks


#3

Perfect, thanks a lot for your help Kyle, I will definitely evaluate this.

Thanks

···

On Sat, Jun 27, 2015 at 6:30 PM, Kyle Anderson kyle@xkyle.com wrote:

One way to do this with stock handlers is to configure two handlers:

one handler for each channel. Then configure the different handlers

respond to different severities:

https://sensuapp.org/docs/latest/handlers#definition-attributes (see

“severities”)

So you might have a slack-criticals handler that operates on CRITICAL

and slack-warnings handler that operates on… everything else? It

would be up to you.

It does require you to have each check send to both handlers. (or make

a handler set)

At Yelp we have lots of teams and lots of channels, so it was worth it

for us to handle this with a custom handler, sending “paging” alerts

to one channel and non-paging alerts to a different one, depending on

your team:

https://github.com/Yelp/sensu_handlers/blob/master/files/nodebot.rb#L12-L21

On Sat, Jun 27, 2015 at 12:16 AM, aiman@flipagram.com wrote:

Hi everyone

I am required to report WARN and CRITICAL messages to separate Slack

channels, did a preliminary search for this but couldnt find much. In most

cases in checks we specify slack handler and that sends all messages to one

channel. Is there a way to send notifications to different slack channels

depending on the severity of the alert?

Thanks

Aiman Parvaiz

Lead Systems Architect
aiman@flipagram.com

cell: 213-300-6377

http://flipagram.com/apz