sensu client fails to start: "transport connection error", only affecting versions after 0.27.0

This is pretty odd. I’ve been upgrading sensu to the latest version across the board, from 0.26.5-2 to 0.28.2-1. Hundreds of hosts have worked, but a persistently refuse to start, no matter what I’ve tried. They throw this error in the sensu-client.log:

{“timestamp”:“2017-03-10T11:33:10.043503-0700”,“level”:“error”,“message”:“transport connection error”,“error”:“connection closed”}
/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:530:in start_tcp_server': no acceptor (port is in use or requires root privileges) (RuntimeError) from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:530:instart_server’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:394:in setup_http_socket' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:406:insetup_sockets’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:482:in block in start' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/daemon.rb:273:inblock in setup_transport’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-7.0.2/lib/sensu/transport.rb:26:in block in connect' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:151:inset_deferred_status’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:191:in succeed' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-7.0.2/lib/sensu/transport/redis.rb:26:inblock (2 levels) in connect’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:151:in set_deferred_status' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:191:insucceed’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:273:in block (2 levels) in connection_completed' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:258:inblock in verify_version’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:334:in dispatch_response' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:356:inparse_response_line’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:384:in receive_data' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:194:inrun_machine’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:194:in run' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:19:inrun’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/exe/sensu-client:10:in <top (required)>' from /opt/sensu/bin/sensu-client:22:inload’
from /opt/sensu/bin/sensu-client:22:in `’

Aside from checks and plugins these hosts have identical sensu configurations (transport = redis, and the redis settings are identical on all hosts). I can confirm that nothing is listening to on the client’s port:3030. The only way I can get these hosts to start a sensu-client is to backtrack to 0.26.5-2. It starts immediately. None of the 0.27 or 0.28 versions will start (I’ve tried almost all of them). I’m stumped for now. Anyone have any clues here?

`start_tcp_server’: no acceptor (port is in use or requires root privileges) (RuntimeError)

Something is using some port that Sensu Uses.

···

2017-03-10 15:54 GMT-03:00 Paul Rafferty paulr@qualtrics.com:

This is pretty odd. I’ve been upgrading sensu to the latest version across the board, from 0.26.5-2 to 0.28.2-1. Hundreds of hosts have worked, but a persistently refuse to start, no matter what I’ve tried. They throw this error in the sensu-client.log:

{“timestamp”:“2017-03-10T11:33:10.043503-0700”,“level”:“error”,“message”:“transport connection error”,“error”:“connection closed”}
/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:530:in start_tcp_server': no acceptor (port is in use or requires root privileges) (RuntimeError) from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:530:instart_server’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:394:in setup_http_socket' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:406:insetup_sockets’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:482:in block in start' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/daemon.rb:273:inblock in setup_transport’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-7.0.2/lib/sensu/transport.rb:26:in block in connect' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:151:inset_deferred_status’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:191:in succeed' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-7.0.2/lib/sensu/transport/redis.rb:26:inblock (2 levels) in connect’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:151:in set_deferred_status' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:191:insucceed’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:273:in block (2 levels) in connection_completed' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:258:inblock in verify_version’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:334:in dispatch_response' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:356:inparse_response_line’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:384:in receive_data' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:194:inrun_machine’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:194:in run' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:19:inrun’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/exe/sensu-client:10:in <top (required)>' from /opt/sensu/bin/sensu-client:22:inload’
from /opt/sensu/bin/sensu-client:22:in `’

Aside from checks and plugins these hosts have identical sensu configurations (transport = redis, and the redis settings are identical on all hosts). I can confirm that nothing is listening to on the client’s port:3030. The only way I can get these hosts to start a sensu-client is to backtrack to 0.26.5-2. It starts immediately. None of the 0.27 or 0.28 versions will start (I’ve tried almost all of them). I’m stumped for now. Anyone have any clues here?

If it is, then why don’t sensu 0.26 and lower have the same issue? It’s not the local port 3030,for sure .

···

On 10 Mar 2017 19:10, “Alex Baule” alexwbaule@gmail.com wrote:

`start_tcp_server’: no acceptor (port is in use or requires root privileges) (RuntimeError)

Something is using some port that Sensu Uses.

2017-03-10 15:54 GMT-03:00 Paul Rafferty paulr@qualtrics.com:

This is pretty odd. I’ve been upgrading sensu to the latest version across the board, from 0.26.5-2 to 0.28.2-1. Hundreds of hosts have worked, but a persistently refuse to start, no matter what I’ve tried. They throw this error in the sensu-client.log:

{“timestamp”:“2017-03-10T11:33:10.043503-0700”,“level”:“error”,“message”:“transport connection error”,“error”:“connection closed”}
/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:530:in start_tcp_server': no acceptor (port is in use or requires root privileges) (RuntimeError) from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:530:instart_server’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:394:in setup_http_socket' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:406:insetup_sockets’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:482:in block in start' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/daemon.rb:273:inblock in setup_transport’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-7.0.2/lib/sensu/transport.rb:26:in block in connect' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:151:inset_deferred_status’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:191:in succeed' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-7.0.2/lib/sensu/transport/redis.rb:26:inblock (2 levels) in connect’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:151:in set_deferred_status' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/em/deferrable.rb:191:insucceed’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:273:in block (2 levels) in connection_completed' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:258:inblock in verify_version’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:334:in dispatch_response' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:356:inparse_response_line’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-redis-2.1.0/lib/sensu/redis/client.rb:384:in receive_data' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:194:inrun_machine’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.1/lib/eventmachine.rb:194:in run' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/lib/sensu/client/process.rb:19:inrun’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.27.0/exe/sensu-client:10:in <top (required)>' from /opt/sensu/bin/sensu-client:22:inload’
from /opt/sensu/bin/sensu-client:22:in `’

Aside from checks and plugins these hosts have identical sensu configurations (transport = redis, and the redis settings are identical on all hosts). I can confirm that nothing is listening to on the client’s port:3030. The only way I can get these hosts to start a sensu-client is to backtrack to 0.26.5-2. It starts immediately. None of the 0.27 or 0.28 versions will start (I’ve tried almost all of them). I’m stumped for now. Anyone have any clues here?