Problem with handlers that inherit from Sensu::Handler

Justin_Menga · December 16, 2014, 2:42am

Hi

I have been troubleshooting problems with the HipChat handler not working. Everything looks good config wise but I keep on getting the following output response:

{“timestamp”:“2014-12-16T15:17:45.576765+1300”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/hipchat.rb",“name”:“hipchat”},“output”:“only handling every 60 occurrences: my.host.com/i.dont.exist_HTTP\n”}

I have setup similar handlers for:

Pagerduty

Slack

Mailer

And all of them give the same response, which leads me to believe there is some problem (likely configuration or perhaps Ruby environment related) in relation to sensu-handler, from which all of the above handlers extend.

Note I have testing the Mail handler OK and this works fine (which rules out this issue is isolated to pipe type handlers).

Any ideas?

Regards

Justin

Kyle_Anderson · December 16, 2014, 3:11am

This is a FAQ:
http://sensuapp.org/docs/latest/faq#occurences-refresh

Only 1 out of 60 events is getting through as configured.

···

On Mon, Dec 15, 2014 at 6:42 PM, Justin Menga <justin.menga@gmail.com> wrote:

Hi

I have been troubleshooting problems with the HipChat handler not working.
Everything looks good config wise but I keep on getting the following output
response:

{"timestamp":"2014-12-16T15:17:45.576765+1300","level":"info","message":"handler
output","handler":{"type":"pipe","command":"/etc/sensu/handlers/hipchat.rb","name":"hipchat"},"output":"only
handling every 60 occurrences: my.host.com/i.dont.exist_HTTP\n"}

I have setup similar handlers for:

Pagerduty
Slack
Mailer

And all of them give the same response, which leads me to believe there is
some problem (likely configuration or perhaps Ruby environment related) in
relation to sensu-handler, from which all of the above handlers extend.

Note I have testing the Mail handler OK and this works fine (which rules out
this issue is isolated to pipe type handlers).

Any ideas?

Regards
Justin

Justin_Menga · December 16, 2014, 6:46am

Hi

I set the refresh to 60s and this shows the real error for all HTTP based handlers:

/opt/sensu/embedded/lib/ruby/2.0.0/net/http.rb:878:in `initialize’: Connection refused - connect(2) (Errno::ECONNREFUSED)

The only thing I could find for this error was here - https://groups.google.com/forum/#!topic/sensu-users/BtbR4lAZuPA - but I don’t have a /var/log/event folder ??

Here is the error for the mailer handler:

/opt/sensu/embedded/lib/ruby/2.0.0/rubygems/dependency.rb:296:in `to_specs’: Could not find ‘mail’ (~> 2.5.4) - did find: [mail-2.6.3] (Gem::LoadError)

The mailer issue seems to be this - https://github.com/sensu/sensu/issues/761 - I fixed by commenting the gem version restriction, after which I get the HTTP error as per above.

Regards

Justin

Justin_Menga · December 16, 2014, 7:42am

OK I managed to figure out the issue. I found that the failure was actually on an internal API call that the sense-handler makes to filter on silenced alerts.

After running a tcpdump on the loopback interface, I noted that every time the failure occurred, there was a TCP connection reset happening. It was due to localhost being resolved to IPv6 localhost address (::1) instead of the IPv4 localhost address (127.0.0.1). Setting the API host to 127.0.0.1 (instead of localhost) fixes the problem:

“api”: {

“host”: “127.0.0.1”,

“port”: 4567,

“user”: “admin”,

“password”: “xxxxxxxx”

}

Very obscure issue and difficult to pinpoint (I spent half a day scratching my head on this one), hope this helps anybody else who stumbles on this…

Kyle_Anderson · December 16, 2014, 5:38pm

That seems surprising, because the sensu::handler api_request function
uses exactly what is in the api configuration file to establish what
to connect to?

github.com

sensu-plugins/sensu-plugin/blob/f8541e116c8d5e7fecb0aeeeb1268d7faf6e6adb/lib/sensu-handler.rb#L79


      
            end
            summary
          end
          
          def bail(msg)
            puts msg + ': ' + @event['client']['name'] + '/' + @event['check']['name']
            exit 0
          end
          
          def api_request(method, path, &blk)
            http = Net::HTTP.new(settings['api']['host'], settings['api']['port'])
            req = net_http_req_class(method).new(path)
            if settings['api']['user'] && settings['api']['password']
              req.basic_auth(settings['api']['user'], settings['api']['password'])
            end
            yield(req) if block_given?
            http.request(req)
          end
          
          def filter_disabled
            if @event['check']['alert'] == false

But in this case, localhost resolves to ::1, (probably also to
127.0.0.1, in /etc/hosts right?)
But for binding purposes, ruby only binds on 127.0.0.1?

Can you tell me more about the platform you are on? Also what does this return?
/opt/sensu/embedded/bin/ruby -d -rsocket -e "p
IPSocket.getaddress('localhost')"

···

On Mon, Dec 15, 2014 at 11:42 PM, Justin Menga <justin.menga@gmail.com> wrote:

OK I managed to figure out the issue. I found that the failure was actually
on an internal API call that the sense-handler makes to filter on silenced
alerts.

After running a tcpdump on the loopback interface, I noted that every time
the failure occurred, there was a TCP connection reset happening. It was
due to localhost being resolved to IPv6 localhost address (::1) instead of
the IPv4 localhost address (127.0.0.1). Setting the API host to 127.0.0.1
(instead of localhost) fixes the problem:

"api": {
  "host": "127.0.0.1",
  "port": 4567,
  "user": "admin",
  "password": "xxxxxxxx"
}

Very obscure issue and difficult to pinpoint (I spent half a day scratching
my head on this one), hope this helps anybody else who stumbles on this...

Justin_Menga · December 17, 2014, 5:44am

Hi

The platform is Ubuntu 14.04.

Checking the /etc/hosts file, it was a bit different from the default /etc/hosts and had the 127.0.0.1 entry removed (but not the ::! entry), hence the output of the command below was “::1”. This also explains the behaviour observed.

Not sure why the /etc/hosts file was in this state, the build is vanilla and nothing else on that server.

Regards
Justin

···

On 17/12/2014, at 6:38 am, Kyle Anderson <kyle@xkyle.com> wrote:

That seems surprising, because the sensu::handler api_request function
uses exactly what is in the api configuration file to establish what
to connect to?
https://github.com/sensu/sensu-plugin/blob/f8541e116c8d5e7fecb0aeeeb1268d7faf6e6adb/lib/sensu-handler.rb#L79

But in this case, localhost resolves to ::1, (probably also to
127.0.0.1, in /etc/hosts right?)
But for binding purposes, ruby only binds on 127.0.0.1?

Can you tell me more about the platform you are on? Also what does this return?
/opt/sensu/embedded/bin/ruby -d -rsocket -e "p
IPSocket.getaddress('localhost')"

On Mon, Dec 15, 2014 at 11:42 PM, Justin Menga <justin.menga@gmail.com> wrote:

OK I managed to figure out the issue. I found that the failure was actually
on an internal API call that the sense-handler makes to filter on silenced
alerts.

After running a tcpdump on the loopback interface, I noted that every time
the failure occurred, there was a TCP connection reset happening. It was
due to localhost being resolved to IPv6 localhost address (::1) instead of
the IPv4 localhost address (127.0.0.1). Setting the API host to 127.0.0.1
(instead of localhost) fixes the problem:

"api": {
"host": "127.0.0.1",
"port": 4567,
"user": "admin",
"password": "xxxxxxxx"
}

Very obscure issue and difficult to pinpoint (I spent half a day scratching
my head on this one), hope this helps anybody else who stumbles on this...

Jack_Hollow · July 13, 2015, 1:58pm

@Justin Menga - Thanks for this - I had this exact issue with my slack handler - Your fix solved it. Peace

···

On Tuesday, 16 December 2014 09:42:48 UTC+2, Justin Menga wrote: