Sensu Enterprise Built-in Handler : Email Configuration

Sensu Classic (EOL)
1

Hi experts,

I would like to seek for your advice how filters are applied to Sensu built in handler. Currently we are using Event stream to send out all events toTCP and now we want to configure the Email handler to handle critical events.

The expectation or the goal is for the email handler to deal w/ CRITICAL events only. Only when Check status is set to critical then it would send an email once. If the same event has not been solved within an hour then it should trigger the email to be sent out once every 30 mins. We have an SMTP connection already set up (Postfix) and working but we cannot get the desired output so I’ve been playing around on the configurations and have been running on different issues and questions:

  1. If no filter is applied to the email handler, and then we define the “handlers”: [“email”] to a check which is running on an interval of 60 seconds, email is received once and that’s it. Is it not supposed to send an email every event by default?

  2. Is it correct that even if I don’t define event_stream and email in *handlers { } *as set handlers (default.json), event_stream and email are expected to work as long as we have the correct event_stream and email configuration? and so I can delete my default.json configuration in the system. If that is the case, where should I apply the filter (Filter.json)?

  3. And then if you could help us come up with the right filter and configurations so that only critical events are handled and it must send an email once but if not resolved after an hour, it should send out once every 30 mins.

Below are the configurations we have in our system so far.

Check.json (on the client server)

{
“checks”: {
“event-queues”: {
“type”: “metric”,
“command”: “/opt/sensu/embedded/bin/event-queues.rb”,
“interval”: 60,
“standalone”: true,
“handlers”: [“event_stream”,“email”]
}
}
}

``

Email.json (Sensu Server)

{
“email”: {
“smtp”: {
“address”: “localhost”,
“port”: 25
},
“to”: “peter@gmail.com”,
“from”: “hostmaster@sensu1”
}
}

``

Filter.json (Sensu Server)

{
“filters”: {
“example_filter”: {
“negate”: false,
“attributes”: {
“status”: 2
}
}
}
}

``

default.json (Sensu Server)

{
“handlers”: {
“default”: {
“type”: “set”,
“handlers”: [
“event_stream”
]
}
}
}

``

I really hope you could help us. Thank you so much in advance! :slight_smile:

Regards.

2

Detailed replies inline.

I would like to seek for your advice how filters are applied to Sensu built in handler. Currently we are using Event stream to send out all events toTCP and now we want to configure the Email handler to handle critical events.

The expectation or the goal is for the email handler to deal w/ CRITICAL events only. Only when Check status is set to critical then it would send an email once. If the same event has not been solved within an hour then it should trigger the email to be sent out once every 30 mins. We have an SMTP connection already set up (Postfix) and working but we cannot get the desired output so I’ve been playing around on the configurations and have been running on different issues and questions:

  1. If no filter is applied to the email handler, and then we define the “handlers”: [“email”] to a check which is running on an interval of 60 seconds, email is received once and that’s it. Is it not supposed to send an email every event by default?

No. The default - especially with plugins that are not up to date - is to send one notification and warn in the log. (something about only handling every 30 occurrences).

  1. Is it correct that even if I don’t define event_stream and email in *handlers { } *as set handlers (default.json), event_stream and email are expected to work as long as we have the correct event_stream and email configuration? and so I can delete my default.json configuration in the system. If that is the case, where should I apply the filter (Filter.json)?

Filters are applied at the check definition (example at the end of this mail). I think that if you have an array with handlers, they are treated individually - meaning that you get notifications on the ones that work instead of all or none.

  1. And then if you could help us come up with the right filter and configurations so that only critical events are handled and it must send an email once but if not resolved after an hour, it should send out once every 30 mins.

You should be able to pierce that together from the example (at the end of this mail) as well as this blog post and the Readme of the Sensu-plugin framework. Please don’t hesitate to ask further questions should our instruction be unclear. You will want to check the “occurrences” and “refresh” parameters.

/etc/sensu/conf.d/handlers/mail.json

{

“handlers”: {

“mail”: {

“command”: “handler-mailer.rb -t /etc/sensu/handlers/mail_template.erb -j mail”,

“type”: “pipe”,

“filters”: [

“occurences”

],

“severities”: [

“critical”

]

}

},

“mail”: {

“mail_from”: “REDACTED”,

“mail_to”: “REDACTED”

}

}

/etc/sensu/handlers/mail_template.erb

<%= @event[‘client’][‘name’] %>: <%= @event[‘check’][‘name’] %>

=== Output ===

<%= @event[‘check’][‘output’] %>

=== Meta ===

Impact: <%= @event[‘check’][‘impact’] %>

Suggestion: <%= @event[‘check’][‘suggestion’] %>

Dashboard: https://REDACTED/#/client/sensu/<%= @event[“client”][“name”] %>?check=<%= @event[“check”][“name”] %>

Runbook: <%= @event[‘check’][‘runbook’] %>

Timestamp: <%= Time.at(@event[‘check’][‘issued’]) %>

Address: <%= @event[‘client’][‘address’] %>

Operating System: <%= @event[‘client’][‘operating_system’] %>

/etc/sensu/conf.d/sensu-plugin.json

{

“sensu_plugin”: {

“disable_deprecated_filtering”: true

}

}

/etc/sensu/conf.d/checks/some_tls_check.json

{

“checks”: {

“check_tls_certificates_assignments”: {

“command”: “check-ssl-host.rb --warning 15 --critical 5 --host REDACTED --port 443”,

“handlers”: [

“mattermost”,

“mail”

],

“interval”: 300,

“source”: “REDACTED”,

“subscribers”: [

“proxy”

],

“standalone”: false,

“occurrences”: 1,

“refresh”: 86400,

“impact”: “Secure connections will throw warnings or not work at all.”,

“suggestion”: “Check connectivity. Renew the TLS certificate for the subdomain REDACTED.”,

“runbook”: “https://REDACTED/administrators/documentation/blob/master/runbooks/sensu/certificates.markdown”

}

}

}

``

···

On Friday, July 14, 2017 at 5:32:39 AM UTC+2, Kaye wrote:

3

Hi Alexander,

Thank you so much for the info! really appreciate the very detailed response! :slight_smile:

Yeah I think for regular sensu handler, filter is applied in the handler definition and then handler is defined as an array in the check. But in our case, we are using the email enterprise event handler w/ the configuration Email.json (Sensu Server) so I don’t really define it as a pipe handler in the definition as seen in https://sensuapp.org/docs/latest/enterprise/integrations/email.html

what I did initially was to add the “filters”: [“occurrences”] in the email.json configuration but it was not working,. hence, I specified the “filters” in the check definition as seen below. Is it the correct way of doing it then with Sensu Enterprise?

{

“checks”: {

“agent-tracking-feed”: {

“type”: “metric”,

“command”: “/opt/sensu/embedded/bin/metrics-agent-tracking.rb -s 30”,

“interval”: 60,

“standalone”: true,

“payload-type”: “json”,

  "filters": ["recurrence-5-30"]

}
}
}

``

Thanks again! :slight_smile:

···

On Friday, July 14, 2017 at 11:32:39 AM UTC+8, Kaye wrote:

Hi experts,

I would like to seek for your advice how filters are applied to Sensu built in handler. Currently we are using Event stream to send out all events toTCP and now we want to configure the Email handler to handle critical events.

The expectation or the goal is for the email handler to deal w/ CRITICAL events only. Only when Check status is set to critical then it would send an email once. If the same event has not been solved within an hour then it should trigger the email to be sent out once every 30 mins. We have an SMTP connection already set up (Postfix) and working but we cannot get the desired output so I’ve been playing around on the configurations and have been running on different issues and questions:

  1. If no filter is applied to the email handler, and then we define the “handlers”: [“email”] to a check which is running on an interval of 60 seconds, email is received once and that’s it. Is it not supposed to send an email every event by default?
  1. Is it correct that even if I don’t define event_stream and email in *handlers { } *as set handlers (default.json), event_stream and email are expected to work as long as we have the correct event_stream and email configuration? and so I can delete my default.json configuration in the system. If that is the case, where should I apply the filter (Filter.json)?
  1. And then if you could help us come up with the right filter and configurations so that only critical events are handled and it must send an email once but if not resolved after an hour, it should send out once every 30 mins.

Below are the configurations we have in our system so far.

Check.json (on the client server)

{
“checks”: {
“event-queues”: {
“type”: “metric”,
“command”: “/opt/sensu/embedded/bin/event-queues.rb”,
“interval”: 60,
“standalone”: true,
“handlers”: [“event_stream”,“email”]
}
}
}

``

Email.json (Sensu Server)

{
“email”: {
“smtp”: {
“address”: “localhost”,
“port”: 25
},
“to”: “peter@gmail.com”,
“from”: “hostmaster@sensu1”
}
}

``

Filter.json (Sensu Server)

{
“filters”: {
“example_filter”: {
“negate”: false,
“attributes”: {
“status”: 2
}
}
}
}

``

default.json (Sensu Server)

{
“handlers”: {
“default”: {
“type”: “set”,
“handlers”: [
“event_stream”
]
}
}
}

``

I really hope you could help us. Thank you so much in advance! :slight_smile:

Regards.

4

Also if possible for “refresh” to be triggered on state change or a certain set of condition? If event has been resolved for instance?

Thanks a lot!

···

On Friday, July 14, 2017 at 11:32:39 AM UTC+8, Kaye wrote:

Hi experts,

I would like to seek for your advice how filters are applied to Sensu built in handler. Currently we are using Event stream to send out all events toTCP and now we want to configure the Email handler to handle critical events.

The expectation or the goal is for the email handler to deal w/ CRITICAL events only. Only when Check status is set to critical then it would send an email once. If the same event has not been solved within an hour then it should trigger the email to be sent out once every 30 mins. We have an SMTP connection already set up (Postfix) and working but we cannot get the desired output so I’ve been playing around on the configurations and have been running on different issues and questions:

  1. If no filter is applied to the email handler, and then we define the “handlers”: [“email”] to a check which is running on an interval of 60 seconds, email is received once and that’s it. Is it not supposed to send an email every event by default?
  1. Is it correct that even if I don’t define event_stream and email in *handlers { } *as set handlers (default.json), event_stream and email are expected to work as long as we have the correct event_stream and email configuration? and so I can delete my default.json configuration in the system. If that is the case, where should I apply the filter (Filter.json)?
  1. And then if you could help us come up with the right filter and configurations so that only critical events are handled and it must send an email once but if not resolved after an hour, it should send out once every 30 mins.

Below are the configurations we have in our system so far.

Check.json (on the client server)

{
“checks”: {
“event-queues”: {
“type”: “metric”,
“command”: “/opt/sensu/embedded/bin/event-queues.rb”,
“interval”: 60,
“standalone”: true,
“handlers”: [“event_stream”,“email”]
}
}
}

``

Email.json (Sensu Server)

{
“email”: {
“smtp”: {
“address”: “localhost”,
“port”: 25
},
“to”: “peter@gmail.com”,
“from”: “hostmaster@sensu1”
}
}

``

Filter.json (Sensu Server)

{
“filters”: {
“example_filter”: {
“negate”: false,
“attributes”: {
“status”: 2
}
}
}
}

``

default.json (Sensu Server)

{
“handlers”: {
“default”: {
“type”: “set”,
“handlers”: [
“event_stream”
]
}
}
}

``

I really hope you could help us. Thank you so much in advance! :slight_smile:

Regards.

5

This behavior is already implied. E.g.

  • something has a critical error (sends an e-mail)

  • is critical within the refresh period (does nothing)

  • is OK again within refresh period (sends another e-mail, resets refresh period)

  • OR is critical after refresh period (sends another e-mail)

···

On Tuesday, July 18, 2017 at 8:50:22 AM UTC+2, Kaye wrote:

Also if possible for “refresh” to be triggered on state change or a certain set of condition? If event has been resolved for instance?

6

I’m sorry, I can’t help you with Sensu Enterprise. :confused:

···

On Tuesday, July 18, 2017 at 6:17:54 AM UTC+2, Kaye wrote:

Yeah I think for regular sensu handler, filter is applied in the handler definition and then handler is defined as an array in the check. But in our case, we are using the email enterprise event handler w/ the configuration Email.json (Sensu Server) so I don’t really define it as a pipe handler in the definition as seen in https://sensuapp.org/docs/latest/enterprise/integrations/email.html

Is it the correct way of doing it then with Sensu Enterprise?

7

Hi Kaye,

Sensu Enterprise automatically applies the “handle_when” filter to handler integrations (built-in handlers). In Sensu Enterprise 2.0.0 and later, this “handle_when” filter is effectively an alias for the “occurrences” filter extension included in Sensu Core. As such I expect that “handle_when” is already being applied to your email handler integration. Were this is not the case I’d expect you to receive an email for every non-OK execution, i.e. every 60 seconds, regardless of the “occurrences” and “refresh” values in your check definition.

That said, Sensu Enterprise handler integrations support specifying both filters and severities in their respective configuration scopes. so specifying “filters”: [“occurrences”] in the “email” scope should work in Sensu Enterprise 2.0.0 and later. Whether using Sensu Core or Sensu Enterprise, a “filters” attribute specified in a check definition will be considered a custom attribute and have no affect on check result handling.

From what you’ve written I understand the desired result is to receive emails only for critical events. Based on the configuration you provided earlier, will you please try configuring the email handler integration as follows?

{

“email”: {

"smtp": {

  "address": "localhost",

  "port": 25

},

"to": "pe...@gmail.com",

"from": "hostmaster@sensu1",

"severities": ["critical"]

}

}

Using “severities” to filter messages this way will still allow resolve messages to be handled.

Regards,

Cameron

···

On Monday, July 17, 2017 at 10:17:54 PM UTC-6, Kaye wrote:

Hi Alexander,

Thank you so much for the info! really appreciate the very detailed response! :slight_smile:

Yeah I think for regular sensu handler, filter is applied in the handler definition and then handler is defined as an array in the check. But in our case, we are using the email enterprise event handler w/ the configuration Email.json (Sensu Server) so I don’t really define it as a pipe handler in the definition as seen in https://sensuapp.org/docs/latest/enterprise/integrations/email.html

what I did initially was to add the “filters”: [“occurrences”] in the email.json configuration but it was not working,. hence, I specified the “filters” in the check definition as seen below. Is it the correct way of doing it then with Sensu Enterprise?

{

“checks”: {

“agent-tracking-feed”: {

“type”: “metric”,

“command”: “/opt/sensu/embedded/bin/metrics-agent-tracking.rb -s 30”,

“interval”: 60,

“standalone”: true,

“payload-type”: “json”,

  "filters": ["recurrence-5-30"]

}
}
}

``

Thanks again! :slight_smile:

On Friday, July 14, 2017 at 11:32:39 AM UTC+8, Kaye wrote:

Hi experts,

I would like to seek for your advice how filters are applied to Sensu built in handler. Currently we are using Event stream to send out all events toTCP and now we want to configure the Email handler to handle critical events.

The expectation or the goal is for the email handler to deal w/ CRITICAL events only. Only when Check status is set to critical then it would send an email once. If the same event has not been solved within an hour then it should trigger the email to be sent out once every 30 mins. We have an SMTP connection already set up (Postfix) and working but we cannot get the desired output so I’ve been playing around on the configurations and have been running on different issues and questions:

  1. If no filter is applied to the email handler, and then we define the “handlers”: [“email”] to a check which is running on an interval of 60 seconds, email is received once and that’s it. Is it not supposed to send an email every event by default?
  1. Is it correct that even if I don’t define event_stream and email in *handlers { } *as set handlers (default.json), event_stream and email are expected to work as long as we have the correct event_stream and email configuration? and so I can delete my default.json configuration in the system. If that is the case, where should I apply the filter (Filter.json)?
  1. And then if you could help us come up with the right filter and configurations so that only critical events are handled and it must send an email once but if not resolved after an hour, it should send out once every 30 mins.

Below are the configurations we have in our system so far.

Check.json (on the client server)

{
“checks”: {
“event-queues”: {
“type”: “metric”,
“command”: “/opt/sensu/embedded/bin/event-queues.rb”,
“interval”: 60,
“standalone”: true,
“handlers”: [“event_stream”,“email”]
}
}
}

``

Email.json (Sensu Server)

{
“email”: {
“smtp”: {
“address”: “localhost”,
“port”: 25
},
“to”: “pe...@gmail.com”,
“from”: “hostmaster@sensu1”
}
}

``

Filter.json (Sensu Server)

{
“filters”: {
“example_filter”: {
“negate”: false,
“attributes”: {
“status”: 2
}
}
}
}

``

default.json (Sensu Server)

{
“handlers”: {
“default”: {
“type”: “set”,
“handlers”: [
“event_stream”
]
}
}
}

``

I really hope you could help us. Thank you so much in advance! :slight_smile:

Regards.

8

Hi Alexander,

You’ve been a great help! Thanks a lot! :slight_smile:

···

On Friday, July 14, 2017 at 11:32:39 AM UTC+8, Kaye wrote:

Hi experts,

I would like to seek for your advice how filters are applied to Sensu built in handler. Currently we are using Event stream to send out all events toTCP and now we want to configure the Email handler to handle critical events.

The expectation or the goal is for the email handler to deal w/ CRITICAL events only. Only when Check status is set to critical then it would send an email once. If the same event has not been solved within an hour then it should trigger the email to be sent out once every 30 mins. We have an SMTP connection already set up (Postfix) and working but we cannot get the desired output so I’ve been playing around on the configurations and have been running on different issues and questions:

  1. If no filter is applied to the email handler, and then we define the “handlers”: [“email”] to a check which is running on an interval of 60 seconds, email is received once and that’s it. Is it not supposed to send an email every event by default?
  1. Is it correct that even if I don’t define event_stream and email in *handlers { } *as set handlers (default.json), event_stream and email are expected to work as long as we have the correct event_stream and email configuration? and so I can delete my default.json configuration in the system. If that is the case, where should I apply the filter (Filter.json)?
  1. And then if you could help us come up with the right filter and configurations so that only critical events are handled and it must send an email once but if not resolved after an hour, it should send out once every 30 mins.

Below are the configurations we have in our system so far.

Check.json (on the client server)

{
“checks”: {
“event-queues”: {
“type”: “metric”,
“command”: “/opt/sensu/embedded/bin/event-queues.rb”,
“interval”: 60,
“standalone”: true,
“handlers”: [“event_stream”,“email”]
}
}
}

``

Email.json (Sensu Server)

{
“email”: {
“smtp”: {
“address”: “localhost”,
“port”: 25
},
“to”: “peter@gmail.com”,
“from”: “hostmaster@sensu1”
}
}

``

Filter.json (Sensu Server)

{
“filters”: {
“example_filter”: {
“negate”: false,
“attributes”: {
“status”: 2
}
}
}
}

``

default.json (Sensu Server)

{
“handlers”: {
“default”: {
“type”: “set”,
“handlers”: [
“event_stream”
]
}
}
}

``

I really hope you could help us. Thank you so much in advance! :slight_smile:

Regards.

9

Hi Cameron,

Totally clear! :slight_smile: thanks so much for explaining in details. Yes the desired output is to handle only critical events and to somehow send an email for 1st occurrence but if same event has not been resolved after 3600 seconds then it should send a reminder email every 1800 seconds until event has been resolved. Would this be possible to be controlled in the email or filter scope? and is there a way to use 2 different subject templates for email? Because this way, if we have several notification emails in the queue then we could easily determine in the subject if it’s a new event or just a reminder of unresolved event?

Thank you again and Regards

···

On Friday, July 14, 2017 at 11:32:39 AM UTC+8, Kaye wrote:

Hi experts,

I would like to seek for your advice how filters are applied to Sensu built in handler. Currently we are using Event stream to send out all events toTCP and now we want to configure the Email handler to handle critical events.

The expectation or the goal is for the email handler to deal w/ CRITICAL events only. Only when Check status is set to critical then it would send an email once. If the same event has not been solved within an hour then it should trigger the email to be sent out once every 30 mins. We have an SMTP connection already set up (Postfix) and working but we cannot get the desired output so I’ve been playing around on the configurations and have been running on different issues and questions:

  1. If no filter is applied to the email handler, and then we define the “handlers”: [“email”] to a check which is running on an interval of 60 seconds, email is received once and that’s it. Is it not supposed to send an email every event by default?
  1. Is it correct that even if I don’t define event_stream and email in *handlers { } *as set handlers (default.json), event_stream and email are expected to work as long as we have the correct event_stream and email configuration? and so I can delete my default.json configuration in the system. If that is the case, where should I apply the filter (Filter.json)?
  1. And then if you could help us come up with the right filter and configurations so that only critical events are handled and it must send an email once but if not resolved after an hour, it should send out once every 30 mins.

Below are the configurations we have in our system so far.

Check.json (on the client server)

{
“checks”: {
“event-queues”: {
“type”: “metric”,
“command”: “/opt/sensu/embedded/bin/event-queues.rb”,
“interval”: 60,
“standalone”: true,
“handlers”: [“event_stream”,“email”]
}
}
}

``

Email.json (Sensu Server)

{
“email”: {
“smtp”: {
“address”: “localhost”,
“port”: 25
},
“to”: “peter@gmail.com”,
“from”: “hostmaster@sensu1”
}
}

``

Filter.json (Sensu Server)

{
“filters”: {
“example_filter”: {
“negate”: false,
“attributes”: {
“status”: 2
}
}
}
}

``

default.json (Sensu Server)

{
“handlers”: {
“default”: {
“type”: “set”,
“handlers”: [
“event_stream”
]
}
}
}

``

I really hope you could help us. Thank you so much in advance! :slight_smile:

Regards.

10

Hi Kaye,

The handle_when filter honors a “refresh” value in any check definition, an integer describing how many seconds must pass before an ongoing alert should be handled again. With a refresh of 1800 seconds in your check definition, recurring notifications should be limited to once every half hour.

It should be possible to filter out the one extra alert during the first hour by creating a series of filters to match each possible check interval, but I haven’t had a chance to figure this out yet myself.

Cameron

···

On Tuesday, July 18, 2017 at 10:04:03 PM UTC-6, Kaye wrote:

Hi Cameron,

Totally clear! :slight_smile: thanks so much for explaining in details. Yes the desired output is to handle only critical events and to somehow send an email for 1st occurrence but if same event has not been resolved after 3600 seconds then it should send a reminder email every 1800 seconds until event has been resolved. Would this be possible to be controlled in the email or filter scope? and is there a way to use 2 different subject templates for email? Because this way, if we have several notification emails in the queue then we could easily determine in the subject if it’s a new event or just a reminder of unresolved event?

Thank you again and Regards

On Friday, July 14, 2017 at 11:32:39 AM UTC+8, Kaye wrote:

Hi experts,

I would like to seek for your advice how filters are applied to Sensu built in handler. Currently we are using Event stream to send out all events toTCP and now we want to configure the Email handler to handle critical events.

The expectation or the goal is for the email handler to deal w/ CRITICAL events only. Only when Check status is set to critical then it would send an email once. If the same event has not been solved within an hour then it should trigger the email to be sent out once every 30 mins. We have an SMTP connection already set up (Postfix) and working but we cannot get the desired output so I’ve been playing around on the configurations and have been running on different issues and questions:

  1. If no filter is applied to the email handler, and then we define the “handlers”: [“email”] to a check which is running on an interval of 60 seconds, email is received once and that’s it. Is it not supposed to send an email every event by default?
  1. Is it correct that even if I don’t define event_stream and email in *handlers { } *as set handlers (default.json), event_stream and email are expected to work as long as we have the correct event_stream and email configuration? and so I can delete my default.json configuration in the system. If that is the case, where should I apply the filter (Filter.json)?
  1. And then if you could help us come up with the right filter and configurations so that only critical events are handled and it must send an email once but if not resolved after an hour, it should send out once every 30 mins.

Below are the configurations we have in our system so far.

Check.json (on the client server)

{
“checks”: {
“event-queues”: {
“type”: “metric”,
“command”: “/opt/sensu/embedded/bin/event-queues.rb”,
“interval”: 60,
“standalone”: true,
“handlers”: [“event_stream”,“email”]
}
}
}

``

Email.json (Sensu Server)

{
“email”: {
“smtp”: {
“address”: “localhost”,
“port”: 25
},
“to”: “pe...@gmail.com”,
“from”: “hostmaster@sensu1”
}
}

``

Filter.json (Sensu Server)

{
“filters”: {
“example_filter”: {
“negate”: false,
“attributes”: {
“status”: 2
}
}
}
}

``

default.json (Sensu Server)

{
“handlers”: {
“default”: {
“type”: “set”,
“handlers”: [
“event_stream”
]
}
}
}

``

I really hope you could help us. Thank you so much in advance! :slight_smile:

Regards.