Sensu Handler

Jeferson_Martins · February 10, 2016, 6:26pm

Hello,

Whats happens when a sensu handler fails?

There’s somekind of retry in Sensu?

Kyle_Anderson · February 12, 2016, 3:26am

Sensu does not retry failing handlers, but it does log the fact that
they failed and their output

sensu/sensu/blob/4d696b23f800a68ea1f3e6ea7cb94a7437fa94c9/lib/sensu/server/handle.rb#L157


      
                  end
                end
          
                # Handle an event, providing event data to an event handler.
                # This method logs event data and the handler definition at the
                # debug log level, then calls the `handler_type_router()`
                # method.
                #
                # @param handler [Hash] definition.
                # @param event_data [Object] to pass to an event handler.
                def handle_event(handler, event_data)
                  definition = handler.is_a?(Hash) ? handler : handler.definition
                  @logger.debug("handling event", {
                    :event_data => event_data,
                    :handler => definition
                  })
                  handler_type_router(handler, event_data)
                end
              end
            end
          end

Of course, if the check is still failing and sensu is scheduling the
check, you will get another try after the interval.

But, this is sometimes not enough, especially on a "resolve" event,
where you only get 1 shot to resolve it.
At Yelp we do the retries *in* the handler, but only on those that
really need it and those that are idempotent, like pagerduty:

github.com

Yelp/sensu_handlers/blob/9664831a110b6a0c3b75fa6994cae37032a7ce4c/files/pagerduty.rb#L56


      
            log "pagerduty -- Ignoring incident #{incident_key} as it is not set to page."
            return
          end
          begin
            action = case @event['check']['status'].to_i
              when 2
                'trigger'
              when 0,1
              'resolve'
            end
            response = timeout_and_retry do
              case @event['check']['status'].to_i
              when 2
                trigger_incident
              when 0,1
                resolve_incident
              end
            end
            if response
              log 'pagerduty -- ' + action.capitalize + 'd incident -- ' + incident_key
            else

···

On Wed, Feb 10, 2016 at 10:26 AM, Jeferson Martins <jmf2004@gmail.com> wrote:

Hello,

Whats happens when a sensu handler fails?
There's somekind of retry in Sensu?

Topic		Replies	Views
Handler not always sending resolve events Sensu Classic (EOL)	3	1455	December 3, 2019
Handler behavior when check goes from warning to critical Sensu Classic (EOL)	2	692	September 10, 2014
Sensu server losing events Sensu Classic (EOL)	3	442	November 22, 2018
Handlers and checks Sensu Classic (EOL)	6	442	February 25, 2015
external check result and handlers Sensu Classic (EOL)	3	516	March 1, 2016

Sensu Handler

Related topics