Hipchat handler sends alert in check but not from keep alive


#1

Hi all,

I’m having an issue where a handler will work if it used as part of a check but will not fire if it is part of a keep-alive. Not really sure where/how to debug this further.

The check that is working has the following configuration on the server:

{

“checks”: {

“ntp_check”: {

“handlers”: [“hipchat”],

“command”: "/etc/sensu/plugins/check-procs.rb -p ntp -C 1 ",

“interval”: 60,

“subscribers”: [ “all” ]

}

}

}

This is client setting:

{

“client”: {

“address”: “127.0.0.1”,

“handlers”: [

“hipchat”

],

“keepalive”: {

“thresholds”: {

“critical”: 300,

“warning”: 90

}

},

“name”: “debian.ospreytest.com”,

“refresh”: 1800,

“subscriptions”: [

“all”

]

}

}

And the handler is defined:

{

“handlers”: {

“hipchat” : {

“type”: “pipe”,

“command”: “/etc/sensu/plugins/hipchat_handler.rb”

}

}

If I stop the ntp service on the client machine I get an alert in hipchat as expected which is resolved on when I restart ntp. However if I stop the sensu-client service on the client machine, uchiwa will show a warning at 90 sec and an alert at 300 sec in the events but nothing is posted to hipchat. I don’t say anything indicating an error connecting to hipchat in the server logs.

Any suggestions as to where to begin troubleshooting this further would be greatly appreciated.

Cheers,

Tyler


#2

This is a bit hard to troubleshoot because the flexibility of sensu's
configuration makes your typo not explode.

The issue here is that your handlers stanza is in the wrong place:
http://sensuapp.org/docs/latest/clients#keepalives

One way to determine this would be to *very* closely inspect the
keepalive events coming out of the logs on the server and see that
they do not have the hipchat handler on them, only "default". Pretty
tricky.

···

On Mon, Dec 15, 2014 at 5:19 PM, <tyler@ospreyinformatics.com> wrote:

Hi all,

I'm having an issue where a handler will work if it used as part of a check
but will not fire if it is part of a keep-alive. Not really sure where/how
to debug this further.

The check that is working has the following configuration on the server:

{
  "checks": {
    "ntp_check": {
      "handlers": ["hipchat"],
      "command": "/etc/sensu/plugins/check-procs.rb -p ntp -C 1 ",
      "interval": 60,
      "subscribers": [ "all" ]
    }
  }
}

This is client setting:

{
  "client": {
    "address": "127.0.0.1",
    "handlers": [
      "hipchat"
    ],
    "keepalive": {
      "thresholds": {
        "critical": 300,
        "warning": 90
      }
    },
    "name": "debian.ospreytest.com",
    "refresh": 1800,
    "subscriptions": [
      "all"
    ]
  }
}

And the handler is defined:

{
        "handlers": {
                "hipchat" : {
                        "type": "pipe",
                        "command": "/etc/sensu/plugins/hipchat_handler.rb"
                }
}

If I stop the ntp service on the client machine I get an alert in hipchat as
expected which is resolved on when I restart ntp. However if I stop the
sensu-client service on the client machine, uchiwa will show a warning at 90
sec and an alert at 300 sec in the events but nothing is posted to hipchat.
I don't say anything indicating an error connecting to hipchat in the server
logs.

Any suggestions as to where to begin troubleshooting this further would be
greatly appreciated.

Cheers,
Tyler


#3

Hi Kyle,

Thanks for the link. Moving the handlers array inside the keepalive object has resolved the issue.

Cheers,

Tyler

···

On Monday, December 15, 2014 8:14:18 PM UTC-7, Kyle Anderson wrote:

This is a bit hard to troubleshoot because the flexibility of sensu’s

configuration makes your typo not explode.

The issue here is that your handlers stanza is in the wrong place:

http://sensuapp.org/docs/latest/clients#keepalives

One way to determine this would be to very closely inspect the

keepalive events coming out of the logs on the server and see that

they do not have the hipchat handler on them, only “default”. Pretty

tricky.

On Mon, Dec 15, 2014 at 5:19 PM, ty...@ospreyinformatics.com wrote:

Hi all,

I’m having an issue where a handler will work if it used as part of a check

but will not fire if it is part of a keep-alive. Not really sure where/how

to debug this further.

The check that is working has the following configuration on the server:

{

“checks”: {

"ntp_check": {
  "handlers": ["hipchat"],
  "command": "/etc/sensu/plugins/check-procs.rb -p ntp -C 1 ",
  "interval": 60,
  "subscribers": [ "all" ]
}

}

}

This is client setting:

{

“client”: {

"address": "127.0.0.1",
"handlers": [
  "hipchat"
],
"keepalive": {
  "thresholds": {
    "critical": 300,
    "warning": 90
  }
},
"name": "[debian.ospreytest.com](http://debian.ospreytest.com)",
"refresh": 1800,
"subscriptions": [
  "all"
]

}

}

And the handler is defined:

{

    "handlers": {
            "hipchat" : {
                    "type": "pipe",
                    "command": "/etc/sensu/plugins/hipchat_handler.rb"
            }

}

If I stop the ntp service on the client machine I get an alert in hipchat as

expected which is resolved on when I restart ntp. However if I stop the

sensu-client service on the client machine, uchiwa will show a warning at 90

sec and an alert at 300 sec in the events but nothing is posted to hipchat.

I don’t say anything indicating an error connecting to hipchat in the server

logs.

Any suggestions as to where to begin troubleshooting this further would be

greatly appreciated.

Cheers,

Tyler