Changing handlers for a check not honored


#1

Hi all,

I have recently changed all of our checks to use a different set of handlers, removing an older “hipchat” handler, and adding a new one. After applying the configuration and restarting the server, any new alerts seem to continue using the old set of handlers. The json listed here is the log entry on the master showing what it’s going to do with this check. The second block is the configuration block output on server startup, limited to this check.

{

“timestamp”: “2015-08-04T20:09:18.961191+0000”,

“level”: “info”,

“message”: “processing event”,

“event”: {

“id”: “ba5ba045-abf2-405b-acd2-7c5b84edb091”,

“client”: {

“name”: “host.erchinger.org”,

“address”: “10.208.22.113”,

“subscriptions”: [

“atd”

],

“version”: “0.19.2”,

“timestamp”: 1438718944

},

“check”: {

“command”: “/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1”,

“dependencies”: [

sensu-server.erchinger.org/alerts-on_check

],

“handlers”: [

“mailer”,

“hipchat”

],

“interval”: 60,

“occurrences”: 2,

“refresh”: 86400,

“subscribers”: [

“atd”

],

“name”: “atd_check”,

“issued”: 1438718958,

“executed”: 1438718958,

“duration”: 0.181,

“output”: “CheckProcs OK: Found 1 matching processes; cmd /atd/\n”,

“status”: 0,

“history”: [

“2”,

“2”,

“2”,

“0”

],

“total_state_change”: 11

},

“occurrences”: 3,

“action”: “resolve”,

“timestamp”: 1438718958

}

}

{ “atd_check”: [

null,

{

“command”: “/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1”,

“dependencies”: [

sensu-server.erchinger.org/alerts-on_check

],

“handlers”: [

“pagerduty”

],

“interval”: 60,

“occurrences”: 2,

“refresh”: 86400,

“subscribers”: [

“atd”

]

}

]}

Why wouldn’t the alert log entry shown first not adhere to the configuration read on startup? Do I have to clear something out of rabbitmq, or am I missing something else? For what it’s worth, I don’t even have anything with the word hipchat in the /etc/sensu directory anymore.

Thoughts?

Thanks,

Ethan


#2

Clearly the formatting into a code block didn’t work, for a reason I don’t understand. Anyway, reposted here.

{

“timestamp”: “2015-08-04T20:09:18.961191+0000”,

“level”: “info”,

“message”: “processing event”,

“event”: {

"id": "ba5ba045-abf2-405b-acd2-7c5b84edb091",

"client": {

  "name": "[host.erchinger.org](http://host.erchinger.org/)",

  "address": "10.208.22.113",

  "subscriptions": [

    "atd"

  ],

  "version": "0.19.2",

  "timestamp": 1438718944

},

"check": {

  "command": "/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1",

  "dependencies": [

    "[sensu-server.erchinger.org/alerts-on_check](http://sensu-server.erchinger.org/alerts-on_check)"

  ],

  "handlers": [

    "mailer",

    "hipchat"

  ],

  "interval": 60,

  "occurrences": 2,

  "refresh": 86400,

  "subscribers": [

    "atd"

  ],

  "name": "atd_check",

  "issued": 1438718958,

  "executed": 1438718958,

  "duration": 0.181,

  "output": "CheckProcs OK: Found 1 matching processes; cmd /atd/\n",

  "status": 0,

  "history": [

    "2",

    "2",

    "2",

    "0"

  ],

  "total_state_change": 11

},

"occurrences": 3,

"action": "resolve",

"timestamp": 1438718958

}

}

``

{“atd_check”: [

null,

{

“command”: “/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1”,

“dependencies”: [

sensu-server.ord.1debit.com/alerts-on_check

],

“handlers”: [

“pagerduty”

],

“high_flap_threshold”: 25,

“interval”: 60,

“low_flap_threshold”: 10,

“occurrences”: 2,

“refresh”: 86400,

“subscribers”: [

“atd”

]

}

]}

``


#3

Is atd_check defined on host.erchinger.org at all? (the client)

If so, local configuration takes precedence:

If so, just adjust the local configuration and restart the sensu client.
If not, can you paste client logs from host.erchinger.org for more context?

···

On Tue, Aug 4, 2015 at 1:36 PM, Ethan Erchinger <ethan@erchinger.org> wrote:

Clearly the formatting into a code block didn't work, for a reason I don't
understand. Anyway, reposted here.

{
"timestamp": "2015-08-04T20:09:18.961191+0000",
"level": "info",
"message": "processing event",
"event": {
   "id": "ba5ba045-abf2-405b-acd2-7c5b84edb091",
   "client": {
     "name": "host.erchinger.org",
     "address": "10.208.22.113",
     "subscriptions": [
       "atd"
      ],
      "version": "0.19.2",
     "timestamp": 1438718944
   },
   "check": {
     "command": "/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1",
     "dependencies": [
       "sensu-server.erchinger.org/alerts-on_check"
     ],
     "handlers": [
       "mailer",
       "hipchat"
     ],
     "interval": 60,
      "occurrences": 2,
      "refresh": 86400,
     "subscribers": [
       "atd"
     ],
     "name": "atd_check",
     "issued": 1438718958,
     "executed": 1438718958,
     "duration": 0.181,
     "output": "CheckProcs OK: Found 1 matching processes; cmd /atd/\n",
     "status": 0,
     "history": [
       "2",
       "2",
       "2",
       "0"
     ],
     "total_state_change": 11
   },
   "occurrences": 3,
   "action": "resolve",
   "timestamp": 1438718958
}
}

      {"atd_check": [
        null,
        {
          "command": "/etc/sensu/plugins/processes/check-procs.rb -p atd -C
1",
          "dependencies": [
            "sensu-server.ord.1debit.com/alerts-on_check"
          ],
          "handlers": [
            "pagerduty"
          ],
          "high_flap_threshold": 25,
          "interval": 60,
          "low_flap_threshold": 10,
          "occurrences": 2,
          "refresh": 86400,
          "subscribers": [
            "atd"
          ]
        }
      ]}


#4

Just to wrap this up, I wasn’t using local configuration, only published checks. Things did finally start working as expected, it just took a couple hours for some reason.

Unfortunately one of the effected clients was destroyed by provisioning, so I’m not able to share the logs.

Ethan

···

On Tuesday, August 4, 2015 at 7:43:06 PM UTC-7, Kyle Anderson wrote:

Is atd_check defined on host.erchinger.org at all? (the client)

If so, local configuration takes precedence:
https://github.com/sensu/sensu/blob/9a783db96ed350d35eb521875e467be01fc0d084/lib/sensu/client/process.rb#L196-L214

If so, just adjust the local configuration and restart the sensu client.
If not, can you paste client logs from host.erchinger.org for more context?

On Tue, Aug 4, 2015 at 1:36 PM, Ethan Erchinger et...@erchinger.org wrote:

Clearly the formatting into a code block didn’t work, for a reason I don’t
understand. Anyway, reposted here.

{
“timestamp”: “2015-08-04T20:09:18.961191+0000”,
“level”: “info”,
“message”: “processing event”,
“event”: {
“id”: “ba5ba045-abf2-405b-acd2-7c5b84edb091”,
“client”: {
“name”: “host.erchinger.org”,
“address”: “10.208.22.113”,
“subscriptions”: [
“atd”
],
“version”: “0.19.2”,
“timestamp”: 1438718944
},
“check”: {
“command”: “/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1”,
“dependencies”: [
sensu-server.erchinger.org/alerts-on_check
],
“handlers”: [
“mailer”,
“hipchat”
],
“interval”: 60,
“occurrences”: 2,
“refresh”: 86400,
“subscribers”: [
“atd”
],
“name”: “atd_check”,
“issued”: 1438718958,
“executed”: 1438718958,
“duration”: 0.181,
“output”: “CheckProcs OK: Found 1 matching processes; cmd /atd/\n”,
“status”: 0,
“history”: [
“2”,
“2”,
“2”,
“0”
],
“total_state_change”: 11
},
“occurrences”: 3,
“action”: “resolve”,
“timestamp”: 1438718958
}
}

  {"atd_check": [
    null,
    {
      "command": "/etc/sensu/plugins/processes/check-procs.rb -p atd -C

1",
“dependencies”: [
sensu-server.ord.1debit.com/alerts-on_check
],
“handlers”: [
“pagerduty”
],
“high_flap_threshold”: 25,
“interval”: 60,
“low_flap_threshold”: 10,
“occurrences”: 2,
“refresh”: 86400,
“subscribers”: [
“atd”
]
}
]}