Passing handler name to the actual handler command?

[Matthew_Barr]

tl;dr:

Handlers have no idea what sensu handler “name” they were called as. They can’t lookup their settings based on how they were called. Can’t have 2 of the same handler command with different settings with most community plugins, nor is there a way to have a setting change based on which handler name was called. Should we add a handler hash to the event output?

(All code below main body, for readability.)

Let’s assume I have a memory check( #1) :

I’d like to notify Ops (mailer, #2) if there’s any systems !OK state, and Eng (mail_eng, #3) if there’s any critical. No problem - just use a different handler, w/ different to:’s.

However, if you want to use most of the community plugins, you can’t actually use 2 different sets of config settings, because they hard code the setting key they use to typically their name.

From https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/mailer.rb#L41 :
mail_to = settings[‘mailer’]['mail_to’]

If you don’t define the handler name as mailer.rb, it won’t use the settings.

This is an issue for pretty much every community handler plugin, as far as I can see.

Sensu doesn’t emit the name of the handler being called as part of the json event output to pipe commands.

I’d like to propose that we add a handler section to an event, as it’s emitted.

( Probably best in: https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L296 )

We could put the handler settings there, so that handlers could be simplified, and not have to look up their values from the settings

I’ve heard suggestions to use a custom field on the check, but that still doesn’t let me change which group gets alerted per severity.

Open to other ideas as to how to deal w/ this, obviously… but this just seems like a nice programatic way to deal w/ these kinds of issues.

I’ve been able to hack around this in a fashion for right now by using a (sym)link for the mailer.rb file to mail_eng.rb, and changing the line above to be :

basename=File.basename($0,File.extname($0))

mail_to = settings[basename][‘mail_to’]

Which sets basename to be name of the script called (mail_eng, in this case.)

#1 : Check Def:
{
“checks”: {
“check_mem”: {
“command”: “/etc/sensu/plugins/check_mem.sh -w 90 -c 95”,
“occurrences”: 2,
“handlers”: [
“mailer",
“mail_eng”
],
“standalone”: false,
“subscribers”: [
“allhosts"
],
“interval”: 60
}
}
}

#2: mailer Handler:
{
“mailer": {
“mail_to”: “mbarr@example.com”,
“mail_from”: “sensu@example.com”,
},
“handlers”: {
“mailer”: {
“severities”: [
“warning”,
“critical”,
“unknown”
],
“command”: “/etc/sensu/handlers/mailer.rb”,
“type”: “pipe”
}
}
}

#3: mail_eng Handler:
{
“mail_eng”: {
“mail_to”: “eng-alerts@example.com”,
“mail_from”: “sensu@example.com”,
},
“handlers”: {
“mail_eng”: {
“severities”: [
“critical”
],
“type”: “pipe”,
“command”: “/etc/sensu/handlers/mail_eng.rb”
}
}
}

[Kyle_Anderson]

Hmm, crap
I probably shouldn't have recommended something I hadn't done myself
personally

The only other suggestion I have is the same one from the previous
email, where the check itself emits its email recipients, and the
handler is modified to read that out of the hash.

I could see the argument for the handler "knowing" what its name is,
or what it's specific settings hash is, but maybe not.

All the settings are there, the only thing we are missing is the
"name" to lookup the specific settings, instead of hardcoding it in
the handler itself, assuming it is the only *instance*. (like you
said)

I don't know what to say. This seems like a pretty common use case,
and I don't think the that everyone who deploys Sensu should have to
re-invent this wheel with their own custom handlers.

I vote with you that the handler invocation:
https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L316
Should pass event_data merged with { "calling_handler" => handler }
and each community plugins handler be modified to look up
settings[calling_handler]['mail_to'], to allow multiple instances of a
handler be configured.

In the end, it does seem like handlers themselves are *almost* config
All of my handlers in my deployment ended up needing to be modified
to fit our business needs. But this kind of change would allow them to
be slightly more re-usable.

···

On Mon, Apr 28, 2014 at 8:41 AM, Matthew Barr <mbarr@kensho.com> wrote:

tl;dr:

Handlers have no idea what sensu handler "name" they were called as. They
can’t lookup their settings based on how they were called. Can’t have 2 of
the same handler command with different settings with most community
plugins, nor is there a way to have a setting change based on which handler
name was called. Should we add a handler hash to the event output?

(All code below main body, for readability.)

Let’s assume I have a memory check( #1) :

I’d like to notify Ops (mailer, #2) if there’s any systems !OK state, and
Eng (mail_eng, #3) if there’s any critical. No problem - just use a
different handler, w/ different to:’s.

However, if you want to use most of the community plugins, you can’t
actually use 2 different sets of config settings, because they hard code the
setting key they use to typically their name.

From
https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/mailer.rb#L41
:
mail_to = settings['mailer']['mail_to’]

If you don’t define the handler name as mailer.rb, it won’t use the
settings.

This is an issue for pretty much every community handler plugin, as far as I
can see.

Sensu doesn’t emit the name of the handler being called as part of the json
event output to pipe commands.
I’d like to propose that we add a handler section to an event, as it’s
emitted.

( Probably best in:
https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L296 )

We could put the handler settings there, so that handlers could be
simplified, and not have to look up their values from the settings

I’ve heard suggestions to use a custom field on the check, but that still
doesn’t let me change which group gets alerted per severity.

Open to other ideas as to how to deal w/ this, obviously... but this just
seems like a nice programatic way to deal w/ these kinds of issues.

I’ve been able to hack around this in a fashion for right now by using a
(sym)link for the mailer.rb file to mail_eng.rb, and changing the line above
to be :

    basename=File.basename($0,File.extname($0))
    mail_to = settings[basename]['mail_to']

Which sets basename to be name of the script called (mail_eng, in this
case.)

#1 : Check Def:
{
  "checks": {
    "check_mem": {
      "command": "/etc/sensu/plugins/check_mem.sh -w 90 -c 95",
      "occurrences": 2,
      "handlers": [
        “mailer",
        "mail_eng"
      ],
      "standalone": false,
      "subscribers": [
        “allhosts"
      ],
      "interval": 60
    }
  }
}

#2: mailer Handler:
{
  “mailer": {
    "mail_to": "mbarr@example.com",
    "mail_from": "sensu@example.com",
  },
  "handlers": {
    "mailer": {
      "severities": [
        "warning",
        "critical",
        "unknown"
      ],
      "command": "/etc/sensu/handlers/mailer.rb",
      "type": "pipe"
    }
  }
}

#3: mail_eng Handler:
{
  "mail_eng": {
    "mail_to": "eng-alerts@example.com",
    "mail_from": "sensu@example.com",
  },
  "handlers": {
    "mail_eng": {
      "severities": [
        "critical"
      ],
      "type": "pipe",
      "command": "/etc/sensu/handlers/mail_eng.rb"
    }
  }
}

[Greg_Poirier]

If it is useful, here is what we use for mailer.rb:

https://gist.github.com/grepory/11409074

It allows you to define mail_to, mail_from, and mail_subject at the check level as well as create a default.

We’re moving away from this to something a little easier to maintain, but it has served us well until now.

···

On Tue, Apr 29, 2014 at 8:12 AM, Kyle Anderson kyle@xkyle.com wrote:

Hmm, crap

I probably shouldn’t have recommended something I hadn’t done myself

personally

The only other suggestion I have is the same one from the previous

email, where the check itself emits its email recipients, and the

handler is modified to read that out of the hash.

I could see the argument for the handler “knowing” what its name is,

or what it’s specific settings hash is, but maybe not.

All the settings are there, the only thing we are missing is the

“name” to lookup the specific settings, instead of hardcoding it in

the handler itself, assuming it is the only instance. (like you

said)

I don’t know what to say. This seems like a pretty common use case,

and I don’t think the that everyone who deploys Sensu should have to

re-invent this wheel with their own custom handlers.

I vote with you that the handler invocation:

https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L316

Should pass event_data merged with { “calling_handler” => handler }

and each community plugins handler be modified to look up

settings[calling_handler][‘mail_to’], to allow multiple instances of a

handler be configured.

In the end, it does seem like handlers themselves are almost config

All of my handlers in my deployment ended up needing to be modified

to fit our business needs. But this kind of change would allow them to

be slightly more re-usable.

On Mon, Apr 28, 2014 at 8:41 AM, Matthew Barr mbarr@kensho.com wrote:

tl;dr:

Handlers have no idea what sensu handler “name” they were called as. They

can’t lookup their settings based on how they were called. Can’t have 2 of

the same handler command with different settings with most community

plugins, nor is there a way to have a setting change based on which handler

name was called. Should we add a handler hash to the event output?

(All code below main body, for readability.)

Let’s assume I have a memory check( #1) :

I’d like to notify Ops (mailer, #2) if there’s any systems !OK state, and

Eng (mail_eng, #3) if there’s any critical. No problem - just use a

different handler, w/ different to:’s.

However, if you want to use most of the community plugins, you can’t

actually use 2 different sets of config settings, because they hard code the

setting key they use to typically their name.

From

https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/mailer.rb#L41

:

mail_to = settings[‘mailer’]['mail_to’]

If you don’t define the handler name as mailer.rb, it won’t use the

settings.

This is an issue for pretty much every community handler plugin, as far as I

can see.

Sensu doesn’t emit the name of the handler being called as part of the json

event output to pipe commands.

I’d like to propose that we add a handler section to an event, as it’s

emitted.

( Probably best in:

https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L296 )

We could put the handler settings there, so that handlers could be

simplified, and not have to look up their values from the settings

I’ve heard suggestions to use a custom field on the check, but that still

doesn’t let me change which group gets alerted per severity.

Open to other ideas as to how to deal w/ this, obviously… but this just

seems like a nice programatic way to deal w/ these kinds of issues.

I’ve been able to hack around this in a fashion for right now by using a

(sym)link for the mailer.rb file to mail_eng.rb, and changing the line above

to be :

basename=File.basename($0,File.extname($0))

mail_to = settings[basename]['mail_to']

Which sets basename to be name of the script called (mail_eng, in this

case.)

#1 : Check Def:

{

“checks”: {

"check_mem": {

  "command": "/etc/sensu/plugins/check_mem.sh -w 90 -c 95",

  "occurrences": 2,

  "handlers": [

    “mailer",

    "mail_eng"

  ],

  "standalone": false,

  "subscribers": [

    “allhosts"

  ],

  "interval": 60

}

}

}

#2: mailer Handler:

{

“mailer": {

"mail_to": "mbarr@example.com",

"mail_from": "sensu@example.com",

},

“handlers”: {

"mailer": {

  "severities": [

    "warning",

    "critical",

    "unknown"

  ],

  "command": "/etc/sensu/handlers/mailer.rb",

  "type": "pipe"

}

}

}

#3: mail_eng Handler:

{

“mail_eng”: {

"mail_to": "eng-alerts@example.com",

"mail_from": "sensu@example.com",

},

“handlers”: {

"mail_eng": {

  "severities": [

    "critical"

  ],

  "type": "pipe",

  "command": "/etc/sensu/handlers/mail_eng.rb"

}

}

}

[Matthew_Barr]

That would help Frédéric Médery, but not us. We don’t actually want to pivot on a per check basis, but on a handler basis

In truth, the handler edit that I noted in my original post would be much easier to maintain, with less duplication of settings between checks.

As a general rule, i’d much prefer to not pass secrets to checks ( on all the hosts) only to get them back, and use them.

···

On Tue, Apr 29, 2014 at 8:12 AM, Kyle Anderson kyle@xkyle.com wrote:

Hmm, crap

I probably shouldn’t have recommended something I hadn’t done myself

personally

The only other suggestion I have is the same one from the previous

email, where the check itself emits its email recipients, and the

handler is modified to read that out of the hash.

I could see the argument for the handler “knowing” what its name is,

or what it’s specific settings hash is, but maybe not.

All the settings are there, the only thing we are missing is the

“name” to lookup the specific settings, instead of hardcoding it in

the handler itself, assuming it is the only instance. (like you

said)

I don’t know what to say. This seems like a pretty common use case,

and I don’t think the that everyone who deploys Sensu should have to

re-invent this wheel with their own custom handlers.

I vote with you that the handler invocation:

https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L316

Should pass event_data merged with { “calling_handler” => handler }

and each community plugins handler be modified to look up

settings[calling_handler][‘mail_to’], to allow multiple instances of a

handler be configured.

In the end, it does seem like handlers themselves are almost config

All of my handlers in my deployment ended up needing to be modified

to fit our business needs. But this kind of change would allow them to

be slightly more re-usable.

On Mon, Apr 28, 2014 at 8:41 AM, Matthew Barr mbarr@kensho.com wrote:

tl;dr:

Handlers have no idea what sensu handler “name” they were called as. They

can’t lookup their settings based on how they were called. Can’t have 2 of

the same handler command with different settings with most community

plugins, nor is there a way to have a setting change based on which handler

name was called. Should we add a handler hash to the event output?

(All code below main body, for readability.)

Let’s assume I have a memory check( #1) :

I’d like to notify Ops (mailer, #2) if there’s any systems !OK state, and

Eng (mail_eng, #3) if there’s any critical. No problem - just use a

different handler, w/ different to:’s.

However, if you want to use most of the community plugins, you can’t

actually use 2 different sets of config settings, because they hard code the

setting key they use to typically their name.

From

https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/mailer.rb#L41

:

mail_to = settings[‘mailer’]['mail_to’]

If you don’t define the handler name as mailer.rb, it won’t use the

settings.

This is an issue for pretty much every community handler plugin, as far as I

can see.

Sensu doesn’t emit the name of the handler being called as part of the json

event output to pipe commands.

I’d like to propose that we add a handler section to an event, as it’s

emitted.

( Probably best in:

https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L296 )

We could put the handler settings there, so that handlers could be

simplified, and not have to look up their values from the settings

I’ve heard suggestions to use a custom field on the check, but that still

doesn’t let me change which group gets alerted per severity.

Open to other ideas as to how to deal w/ this, obviously… but this just

seems like a nice programatic way to deal w/ these kinds of issues.

I’ve been able to hack around this in a fashion for right now by using a

(sym)link for the mailer.rb file to mail_eng.rb, and changing the line above

to be :

basename=File.basename($0,File.extname($0))

mail_to = settings[basename]['mail_to']

Which sets basename to be name of the script called (mail_eng, in this

case.)

#1 : Check Def:

{

“checks”: {

"check_mem": {

  "command": "/etc/sensu/plugins/check_mem.sh -w 90 -c 95",

  "occurrences": 2,

  "handlers": [

    “mailer",

    "mail_eng"

  ],

  "standalone": false,

  "subscribers": [

    “allhosts"

  ],

  "interval": 60

}

}

}

#2: mailer Handler:

{

“mailer": {

"mail_to": "mbarr@example.com",

"mail_from": "sensu@example.com",

},

“handlers”: {

"mailer": {

  "severities": [

    "warning",

    "critical",

    "unknown"

  ],

  "command": "/etc/sensu/handlers/mailer.rb",

  "type": "pipe"

}

}

}

#3: mail_eng Handler:

{

“mail_eng”: {

"mail_to": "eng-alerts@example.com",

"mail_from": "sensu@example.com",

},

“handlers”: {

"mail_eng": {

  "severities": [

    "critical"

  ],

  "type": "pipe",

  "command": "/etc/sensu/handlers/mail_eng.rb"

}

}

}

[Greg_Poirier]

What if you used a combination of handlers and filters?

Define two handlers: mailer_eng, mailer_ops

In mailer_eng, use a filter that limits the status to critical

In mailer_ops, don’t filter

Then have a “mailer” handler that is just a set containing both mailer_eng and mailer_ops?

···

On Tue, Apr 29, 2014 at 12:16 PM, Matthew Barr mbarr@kensho.com wrote:

That would help Frédéric Médery, but not us. We don’t actually want to pivot on a per check basis, but on a handler basis

In truth, the handler edit that I noted in my original post would be much easier to maintain, with less duplication of settings between checks.

As a general rule, i’d much prefer to not pass secrets to checks ( on all the hosts) only to get them back, and use them.

On Apr 29, 2014, at 3:08 PM, Greg Poirier greg.poirier@opower.com wrote:

If it is useful, here is what we use for mailer.rb:

https://gist.github.com/grepory/11409074

It allows you to define mail_to, mail_from, and mail_subject at the check level as well as create a default.

We’re moving away from this to something a little easier to maintain, but it has served us well until now.

On Tue, Apr 29, 2014 at 8:12 AM, Kyle Anderson kyle@xkyle.com wrote:

Hmm, crap

I probably shouldn’t have recommended something I hadn’t done myself

personally

The only other suggestion I have is the same one from the previous

email, where the check itself emits its email recipients, and the

handler is modified to read that out of the hash.

I could see the argument for the handler “knowing” what its name is,

or what it’s specific settings hash is, but maybe not.

All the settings are there, the only thing we are missing is the

“name” to lookup the specific settings, instead of hardcoding it in

the handler itself, assuming it is the only instance. (like you

said)

I don’t know what to say. This seems like a pretty common use case,

and I don’t think the that everyone who deploys Sensu should have to

re-invent this wheel with their own custom handlers.

I vote with you that the handler invocation:

https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L316

Should pass event_data merged with { “calling_handler” => handler }

and each community plugins handler be modified to look up

settings[calling_handler][‘mail_to’], to allow multiple instances of a

handler be configured.

In the end, it does seem like handlers themselves are almost config

All of my handlers in my deployment ended up needing to be modified

to fit our business needs. But this kind of change would allow them to

be slightly more re-usable.

On Mon, Apr 28, 2014 at 8:41 AM, Matthew Barr mbarr@kensho.com wrote:

tl;dr:

Handlers have no idea what sensu handler “name” they were called as. They

can’t lookup their settings based on how they were called. Can’t have 2 of

the same handler command with different settings with most community

plugins, nor is there a way to have a setting change based on which handler

name was called. Should we add a handler hash to the event output?

(All code below main body, for readability.)

Let’s assume I have a memory check( #1) :

I’d like to notify Ops (mailer, #2) if there’s any systems !OK state, and

Eng (mail_eng, #3) if there’s any critical. No problem - just use a

different handler, w/ different to:’s.

However, if you want to use most of the community plugins, you can’t

actually use 2 different sets of config settings, because they hard code the

setting key they use to typically their name.

From

https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/mailer.rb#L41

:

mail_to = settings[‘mailer’]['mail_to’]

If you don’t define the handler name as mailer.rb, it won’t use the

settings.

This is an issue for pretty much every community handler plugin, as far as I

can see.

Sensu doesn’t emit the name of the handler being called as part of the json

event output to pipe commands.

I’d like to propose that we add a handler section to an event, as it’s

emitted.

( Probably best in:

https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L296 )

We could put the handler settings there, so that handlers could be

simplified, and not have to look up their values from the settings

I’ve heard suggestions to use a custom field on the check, but that still

doesn’t let me change which group gets alerted per severity.

Open to other ideas as to how to deal w/ this, obviously… but this just

seems like a nice programatic way to deal w/ these kinds of issues.

I’ve been able to hack around this in a fashion for right now by using a

(sym)link for the mailer.rb file to mail_eng.rb, and changing the line above

to be :

basename=File.basename($0,File.extname($0))

mail_to = settings[basename]['mail_to']

Which sets basename to be name of the script called (mail_eng, in this

case.)

#1 : Check Def:

{

“checks”: {

"check_mem": {

  "command": "/etc/sensu/plugins/check_mem.sh -w 90 -c 95",

  "occurrences": 2,

  "handlers": [

    “mailer",

    "mail_eng"

  ],

  "standalone": false,

  "subscribers": [

    “allhosts"

  ],

  "interval": 60

}

}

}

#2: mailer Handler:

{

“mailer": {

"mail_to": "mbarr@example.com",

"mail_from": "sensu@example.com",

},

“handlers”: {

"mailer": {

  "severities": [

    "warning",

    "critical",

    "unknown"

  ],

  "command": "/etc/sensu/handlers/mailer.rb",

  "type": "pipe"

}

}

}

#3: mail_eng Handler:

{

“mail_eng”: {

"mail_to": "eng-alerts@example.com",

"mail_from": "sensu@example.com",

},

“handlers”: {

"mail_eng": {

  "severities": [

    "critical"

  ],

  "type": "pipe",

  "command": "/etc/sensu/handlers/mail_eng.rb"

}

}

}