Sensu Handler

Jeferson_Martins · February 10, 2016, 6:26pm

Hello,

Whats happens when a sensu handler fails?

There’s somekind of retry in Sensu?

Kyle_Anderson · February 12, 2016, 3:26am

Sensu does not retry failing handlers, but it does log the fact that
they failed and their output

sensu/sensu/blob/4d696b23f800a68ea1f3e6ea7cb94a7437fa94c9/lib/sensu/server/handle.rb#L157


      
                  end
                end
          
                # Handle an event, providing event data to an event handler.
                # This method logs event data and the handler definition at the
                # debug log level, then calls the `handler_type_router()`
                # method.
                #
                # @param handler [Hash] definition.
                # @param event_data [Object] to pass to an event handler.
                def handle_event(handler, event_data)
                  definition = handler.is_a?(Hash) ? handler : handler.definition
                  @logger.debug("handling event", {
                    :event_data => event_data,
                    :handler => definition
                  })
                  handler_type_router(handler, event_data)
                end
              end
            end
          end

Of course, if the check is still failing and sensu is scheduling the
check, you will get another try after the interval.

But, this is sometimes not enough, especially on a "resolve" event,
where you only get 1 shot to resolve it.
At Yelp we do the retries *in* the handler, but only on those that
really need it and those that are idempotent, like pagerduty:

github.com

Yelp/sensu_handlers/blob/9664831a110b6a0c3b75fa6994cae37032a7ce4c/files/pagerduty.rb#L56


      
            log "pagerduty -- Ignoring incident #{incident_key} as it is not set to page."
            return
          end
          begin
            action = case @event['check']['status'].to_i
              when 2
                'trigger'
              when 0,1
              'resolve'
            end
            response = timeout_and_retry do
              case @event['check']['status'].to_i
              when 2
                trigger_incident
              when 0,1
                resolve_incident
              end
            end
            if response
              log 'pagerduty -- ' + action.capitalize + 'd incident -- ' + incident_key
            else

···

On Wed, Feb 10, 2016 at 10:26 AM, Jeferson Martins <jmf2004@gmail.com> wrote:

Hello,

Whats happens when a sensu handler fails?
There's somekind of retry in Sensu?

Topic		Replies	Views
Handler not always sending resolve events Sensu Classic (EOL)	3	1395	December 3, 2019
Handlers and checks Sensu Classic (EOL)	6	409	February 25, 2015
Client handler? Sensu Classic (EOL)	1	481	October 16, 2016
Trigger a handler on status 0 Sensu Classic (EOL)	1	436	March 20, 2015
Handler Extension & severities Sensu Classic (EOL)	1	572	February 14, 2019

Sensu Handler

Related Topics