Notification handlers, resolved events for certain severities

Sensu Classic (EOL)
1

Hi,

We've been trying to implement a policy where we get emailed about
critical states and warnings only appear on the dashboard.

This seems simple, using the mailer from sensu-community-plugins:
https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/mailer.rb

{
  "handlers": {
    "mailer": {
      "command": "/etc/sensu/handlers/mailer.rb",
      "severities": [
        "unknown",
        "critical"
      ],
      "type": "pipe"
    }
  }
}

This sends emails when warning events are resolved, but not when they
are created. This causes extra email noise and confusion.

It seems quite odd to not send state transitions from ok -> warning
but send state transitions from warning -> ok. How do other people
implement these sort of policies?

For now we've patched mailer.rb to do nothing if it gets an event with
'action': 'resolve' whose previous status was not in the list of
severities that the handler should care about:

It's a bit of an ugly hack for the handler to do this, so maybe the
desired behaviour instead is that the server doesn't send such events
to the handler, something like this patch:
https://github.com/bleach/sensu/commit/b2833a2369a3ecfb8c9ebb61849537fbc9f14ff1

Or maybe instead of severities we should define the state transitions
that a handler should receive:

"transitions":
   [ {"from": "ok", "to": "critical" },
     { "from": "critical", "to": "ok"},
     { "from": "critical", "to": "warning"} ]

Do any of the above proposed ways of changing the current behaviour make sense?

Cheers,
Graham

2

This would suggest that event data should contain something along the lines of “last_status”, so that the severity filter can catch events where the last status doesn’t meet the severity condition and the action is “resolve”.

···

On Wed, Jan 15, 2014 at 10:51 AM, Graham Bleach graham@gdb.me wrote:

Hi,

We’ve been trying to implement a policy where we get emailed about

critical states and warnings only appear on the dashboard.

This seems simple, using the mailer from sensu-community-plugins:

https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/mailer.rb

{

“handlers”: {

"mailer": {

  "command": "/etc/sensu/handlers/mailer.rb",

  "severities": [

    "unknown",

    "critical"

  ],

  "type": "pipe"

}

}

}

This sends emails when warning events are resolved, but not when they

are created. This causes extra email noise and confusion.

It seems quite odd to not send state transitions from ok → warning

but send state transitions from warning → ok. How do other people

implement these sort of policies?

For now we’ve patched mailer.rb to do nothing if it gets an event with

‘action’: ‘resolve’ whose previous status was not in the list of

severities that the handler should care about:

https://github.com/bleach/sensu-community-plugins/commit/ee0ca28b67aa1e797b7cbde8180aa1e0e6ffd0d4

It’s a bit of an ugly hack for the handler to do this, so maybe the

desired behaviour instead is that the server doesn’t send such events

to the handler, something like this patch:

https://github.com/bleach/sensu/commit/b2833a2369a3ecfb8c9ebb61849537fbc9f14ff1

Or maybe instead of severities we should define the state transitions

that a handler should receive:

“transitions”:

[ {“from”: “ok”, “to”: “critical” },

 { "from": "critical", "to": "ok"},

 { "from": "critical", "to": "warning"} ]

Do any of the above proposed ways of changing the current behaviour make sense?

Cheers,

Graham

3

Hey Graham,

I have a fix here https://github.com/sensu/sensu/pull/685

Sean.

···

On Thu, Jan 16, 2014 at 1:28 PM, Sean Porter portertech@gmail.com wrote:

This would suggest that event data should contain something along the lines of “last_status”, so that the severity filter can catch events where the last status doesn’t meet the severity condition and the action is “resolve”.

On Wed, Jan 15, 2014 at 10:51 AM, Graham Bleach graham@gdb.me wrote:

Hi,

We’ve been trying to implement a policy where we get emailed about

critical states and warnings only appear on the dashboard.

This seems simple, using the mailer from sensu-community-plugins:

https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/mailer.rb

{

“handlers”: {

"mailer": {

  "command": "/etc/sensu/handlers/mailer.rb",

  "severities": [

    "unknown",

    "critical"

  ],

  "type": "pipe"

}

}

}

This sends emails when warning events are resolved, but not when they

are created. This causes extra email noise and confusion.

It seems quite odd to not send state transitions from ok → warning

but send state transitions from warning → ok. How do other people

implement these sort of policies?

For now we’ve patched mailer.rb to do nothing if it gets an event with

‘action’: ‘resolve’ whose previous status was not in the list of

severities that the handler should care about:

https://github.com/bleach/sensu-community-plugins/commit/ee0ca28b67aa1e797b7cbde8180aa1e0e6ffd0d4

It’s a bit of an ugly hack for the handler to do this, so maybe the

desired behaviour instead is that the server doesn’t send such events

to the handler, something like this patch:

https://github.com/bleach/sensu/commit/b2833a2369a3ecfb8c9ebb61849537fbc9f14ff1

Or maybe instead of severities we should define the state transitions

that a handler should receive:

“transitions”:

[ {“from”: “ok”, “to”: “critical” },

 { "from": "critical", "to": "ok"},

 { "from": "critical", "to": "warning"} ]

Do any of the above proposed ways of changing the current behaviour make sense?

Cheers,

Graham

4

Fix merged into master.

https://github.com/sensu/sensu/commit/e2bfb5e8c95869ba5e9aac290de3c0b4a1419632

Will be in the next release, early next week.

Sean.

···

On Thu, Jan 16, 2014 at 2:02 PM, Sean Porter portertech@gmail.com wrote:

Hey Graham,

I have a fix here https://github.com/sensu/sensu/pull/685

Sean.

On Thu, Jan 16, 2014 at 1:28 PM, Sean Porter portertech@gmail.com wrote:

This would suggest that event data should contain something along the lines of “last_status”, so that the severity filter can catch events where the last status doesn’t meet the severity condition and the action is “resolve”.

On Wed, Jan 15, 2014 at 10:51 AM, Graham Bleach graham@gdb.me wrote:

Hi,

We’ve been trying to implement a policy where we get emailed about

critical states and warnings only appear on the dashboard.

This seems simple, using the mailer from sensu-community-plugins:

https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/mailer.rb

{

“handlers”: {

"mailer": {

  "command": "/etc/sensu/handlers/mailer.rb",

  "severities": [

    "unknown",

    "critical"

  ],

  "type": "pipe"

}

}

}

This sends emails when warning events are resolved, but not when they

are created. This causes extra email noise and confusion.

It seems quite odd to not send state transitions from ok → warning

but send state transitions from warning → ok. How do other people

implement these sort of policies?

For now we’ve patched mailer.rb to do nothing if it gets an event with

‘action’: ‘resolve’ whose previous status was not in the list of

severities that the handler should care about:

https://github.com/bleach/sensu-community-plugins/commit/ee0ca28b67aa1e797b7cbde8180aa1e0e6ffd0d4

It’s a bit of an ugly hack for the handler to do this, so maybe the

desired behaviour instead is that the server doesn’t send such events

to the handler, something like this patch:

https://github.com/bleach/sensu/commit/b2833a2369a3ecfb8c9ebb61849537fbc9f14ff1

Or maybe instead of severities we should define the state transitions

that a handler should receive:

“transitions”:

[ {“from”: “ok”, “to”: “critical” },

 { "from": "critical", "to": "ok"},

 { "from": "critical", "to": "warning"} ]

Do any of the above proposed ways of changing the current behaviour make sense?

Cheers,

Graham