Changing handlers for a check not honored

Ethan_Erchinger · August 4, 2015, 8:34pm

Hi all,

I have recently changed all of our checks to use a different set of handlers, removing an older “hipchat” handler, and adding a new one. After applying the configuration and restarting the server, any new alerts seem to continue using the old set of handlers. The json listed here is the log entry on the master showing what it’s going to do with this check. The second block is the configuration block output on server startup, limited to this check.


{
“timestamp”: “2015-08-04T20:09:18.961191+0000”,
“level”: “info”,
“message”: “processing event”,
“event”: {
“id”: “ba5ba045-abf2-405b-acd2-7c5b84edb091”,
“client”: {
“name”: “host.erchinger.org”,
“address”: “10.208.22.113”,
“subscriptions”: [
“atd”
],
“version”: “0.19.2”,
“timestamp”: 1438718944
},
“check”: {
“command”: “/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1”,
“dependencies”: [
“sensu-server.erchinger.org/alerts-on_check”
],
“handlers”: [
“mailer”,
“hipchat”
],
“interval”: 60,
“occurrences”: 2,
“refresh”: 86400,
“subscribers”: [
“atd”
],
“name”: “atd_check”,
“issued”: 1438718958,
“executed”: 1438718958,
“duration”: 0.181,
“output”: “CheckProcs OK: Found 1 matching processes; cmd /atd/\n”,
“status”: 0,
“history”: [
“2”,
“2”,
“2”,
“0”
],
“total_state_change”: 11
},
“occurrences”: 3,
“action”: “resolve”,
“timestamp”: 1438718958
}
}


{ “atd_check”: [
null,
{
“command”: “/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1”,
“dependencies”: [
“sensu-server.erchinger.org/alerts-on_check”
],
“handlers”: [
“pagerduty”
],
“interval”: 60,
“occurrences”: 2,
“refresh”: 86400,
“subscribers”: [
“atd”
]
}
]}

Why wouldn’t the alert log entry shown first not adhere to the configuration read on startup? Do I have to clear something out of rabbitmq, or am I missing something else? For what it’s worth, I don’t even have anything with the word hipchat in the /etc/sensu directory anymore.

Thoughts?

Thanks,

Ethan

Ethan_Erchinger · August 4, 2015, 8:36pm

Clearly the formatting into a code block didn’t work, for a reason I don’t understand. Anyway, reposted here.

{

“timestamp”: “2015-08-04T20:09:18.961191+0000”,

“level”: “info”,

“message”: “processing event”,

“event”: {

"id": "ba5ba045-abf2-405b-acd2-7c5b84edb091",

"client": {

  "name": "[host.erchinger.org](http://host.erchinger.org/)",

  "address": "10.208.22.113",

  "subscriptions": [

    "atd"

  ],

  "version": "0.19.2",

  "timestamp": 1438718944

},

"check": {

  "command": "/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1",

  "dependencies": [

    "[sensu-server.erchinger.org/alerts-on_check](http://sensu-server.erchinger.org/alerts-on_check)"

  ],

  "handlers": [

    "mailer",

    "hipchat"

  ],

  "interval": 60,

  "occurrences": 2,

  "refresh": 86400,

  "subscribers": [

    "atd"

  ],

  "name": "atd_check",

  "issued": 1438718958,

  "executed": 1438718958,

  "duration": 0.181,

  "output": "CheckProcs OK: Found 1 matching processes; cmd /atd/\n",

  "status": 0,

  "history": [

    "2",

    "2",

    "2",

    "0"

  ],

  "total_state_change": 11

},

"occurrences": 3,

"action": "resolve",

"timestamp": 1438718958

}

``

{“atd_check”: [

null,

{

“command”: “/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1”,

“dependencies”: [

“sensu-server.ord.1debit.com/alerts-on_check”

],

“handlers”: [

“pagerduty”

],

“high_flap_threshold”: 25,

“interval”: 60,

“low_flap_threshold”: 10,

“occurrences”: 2,

“refresh”: 86400,

“subscribers”: [

“atd”

]

}

]}

``

Kyle_Anderson · August 5, 2015, 2:43am

Is atd_check defined on host.erchinger.org at all? (the client)

If so, local configuration takes precedence:

github.com

sensu/sensu/blob/9a783db96ed350d35eb521875e467be01fc0d084/lib/sensu/client/process.rb#L196-L214


      
          # command, it will be executed. A standard check request will be
          # merged with a local check definition, if present. Client safe
          # mode is enforced in this method, requiring a local check
          # definition in order to execute the check command. If a local
          # check definition does not exist when operating with client
          # safe mode, a check result will be published to report the
          # missing check definition. A check request without a
          # command indicates a check extension run. The check request may
          # contain `:extension`, the name of the extension to run. If
          # `:extension` is not present, the check name is used for the
          # extension name. If a check extension does not exist for a
          # name, a check result will be published to report the unknown
          # check extension.
          #
          # @param check [Hash]
          def process_check_request(check)
            @logger.debug("processing check", :check => check)
            if @settings.check_exists?(check[:name])
              check.merge!(@settings[:checks][check[:name]])

If so, just adjust the local configuration and restart the sensu client.
If not, can you paste client logs from host.erchinger.org for more context?

···

On Tue, Aug 4, 2015 at 1:36 PM, Ethan Erchinger <ethan@erchinger.org> wrote:

Clearly the formatting into a code block didn't work, for a reason I don't
understand. Anyway, reposted here.

{
"timestamp": "2015-08-04T20:09:18.961191+0000",
"level": "info",
"message": "processing event",
"event": {
   "id": "ba5ba045-abf2-405b-acd2-7c5b84edb091",
   "client": {
     "name": "host.erchinger.org",
     "address": "10.208.22.113",
     "subscriptions": [
       "atd"
      ],
      "version": "0.19.2",
     "timestamp": 1438718944
   },
   "check": {
     "command": "/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1",
     "dependencies": [
       "sensu-server.erchinger.org/alerts-on_check"
     ],
     "handlers": [
       "mailer",
       "hipchat"
     ],
     "interval": 60,
      "occurrences": 2,
      "refresh": 86400,
     "subscribers": [
       "atd"
     ],
     "name": "atd_check",
     "issued": 1438718958,
     "executed": 1438718958,
     "duration": 0.181,
     "output": "CheckProcs OK: Found 1 matching processes; cmd /atd/\n",
     "status": 0,
     "history": [
       "2",
       "2",
       "2",
       "0"
     ],
     "total_state_change": 11
   },
   "occurrences": 3,
   "action": "resolve",
   "timestamp": 1438718958
}
}

      {"atd_check": [
        null,
        {
          "command": "/etc/sensu/plugins/processes/check-procs.rb -p atd -C
1",
          "dependencies": [
            "sensu-server.ord.1debit.com/alerts-on_check"
          ],
          "handlers": [
            "pagerduty"
          ],
          "high_flap_threshold": 25,
          "interval": 60,
          "low_flap_threshold": 10,
          "occurrences": 2,
          "refresh": 86400,
          "subscribers": [
            "atd"
          ]
        }
      ]}

Ethan_Erchinger · August 11, 2015, 8:25pm

Just to wrap this up, I wasn’t using local configuration, only published checks. Things did finally start working as expected, it just took a couple hours for some reason.

Unfortunately one of the effected clients was destroyed by provisioning, so I’m not able to share the logs.

Ethan

···

On Tuesday, August 4, 2015 at 7:43:06 PM UTC-7, Kyle Anderson wrote:

Is atd_check defined on host.erchinger.org at all? (the client)

If so, local configuration takes precedence:
https://github.com/sensu/sensu/blob/9a783db96ed350d35eb521875e467be01fc0d084/lib/sensu/client/process.rb#L196-L214

If so, just adjust the local configuration and restart the sensu client.
If not, can you paste client logs from host.erchinger.org for more context?

On Tue, Aug 4, 2015 at 1:36 PM, Ethan Erchinger et...@erchinger.org wrote:
Clearly the formatting into a code block didn’t work, for a reason I don’t
understand. Anyway, reposted here.

{
“timestamp”: “2015-08-04T20:09:18.961191+0000”,
“level”: “info”,
“message”: “processing event”,
“event”: {
“id”: “ba5ba045-abf2-405b-acd2-7c5b84edb091”,
“client”: {
“name”: “host.erchinger.org”,
“address”: “10.208.22.113”,
“subscriptions”: [
“atd”
],
“version”: “0.19.2”,
“timestamp”: 1438718944
},
“check”: {
“command”: “/etc/sensu/plugins/processes/check-procs.rb -p atd -C 1”,
“dependencies”: [
“sensu-server.erchinger.org/alerts-on_check”
],
“handlers”: [
“mailer”,
“hipchat”
],
“interval”: 60,
“occurrences”: 2,
“refresh”: 86400,
“subscribers”: [
“atd”
],
“name”: “atd_check”,
“issued”: 1438718958,
“executed”: 1438718958,
“duration”: 0.181,
“output”: “CheckProcs OK: Found 1 matching processes; cmd /atd/\n”,
“status”: 0,
“history”: [
“2”,
“2”,
“2”,
“0”
],
“total_state_change”: 11
},
“occurrences”: 3,
“action”: “resolve”,
“timestamp”: 1438718958
}
}
  {"atd_check": [
    null,
    {
      "command": "/etc/sensu/plugins/processes/check-procs.rb -p atd -C
1",
“dependencies”: [
“sensu-server.ord.1debit.com/alerts-on_check”
],
“handlers”: [
“pagerduty”
],
“high_flap_threshold”: 25,
“interval”: 60,
“low_flap_threshold”: 10,
“occurrences”: 2,
“refresh”: 86400,
“subscribers”: [
“atd”
]
}
]}

Topic		Replies	Views
Hipchat handler sends alert in check but not from keep alive Sensu Classic (EOL)	2	482	December 16, 2014
how to send alerts to different Sensu Classic (EOL)	4	485	November 7, 2016
Problem with handlers that inherit from Sensu::Handler Sensu Classic (EOL)	6	562	July 13, 2015
Sensu server not picking up handlers? Sensu Classic (EOL)	5	538	July 8, 2016
local sensu check Sensu Classic (EOL)	4	415	November 22, 2018

Changing handlers for a check not honored

Related topics