Monitoring/Restarting sensu-client

Sensu Classic (EOL)
1

I’m just starting to set up a sensu monitoring environment. One thing that I’m seeing is that on Centos 7.1 sensu-client will fail to start sometimes. This of course shows that the host is down.

Is there a way to have sensu-server to (re)start a sensu-client if there is no heartbeat received? How do you insure that the sensu-client is always running? I know there are ways to delay the start of the client until the network is up, but I’d like to ensure that the client is always up from the server perspective.

{“timestamp”:“2016-11-14T23:34:07.056437-0600”,“level”:“info”,“message”:“configuring sensu spawn”,“settings”:{“limit”:12}}

/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:677:in `connect_server’: unable to resolve server address: Inappropriate ioctl for device (EventMachine::ConnectionError)

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:677:in `bind_connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:653:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp/session.rb:457:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp.rb:242:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:188:in `setup_connection’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:218:in `connect_with_eligible_options’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:20:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport.rb:24:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/daemon.rb:253:in `setup_transport’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:451:in `start’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:19:in `block in run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in `run_machine’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in `run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:18:in `run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/exe/sensu-client:10:in `<top (required)>’

from /opt/sensu/bin/sensu-client:22:in `load’

from /opt/sensu/bin/sensu-client:22:in `’

Thanks

2

Anyone have a solution for keeping sensu-client running?

Thanks

3

Comments inline.

Anyone have a solution for keeping sensu-client running?

Thanks

Looking at your error message below:

/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:677:in `connect_server’: unable to resolve server address: Inappropriate ioctl for device (EventMachine::ConnectionError)

It could be that you are experiencing DNS resolution issues. You can try to identify and fix the actual DNS problem

or alternatively, use IP addresses and see if that helps.

Personally, I’d investigate and fix the "unable to resolve server address” error.

Regards.

@shankerbalan

···

On 23-Nov-2016, at 6:26 PM, jemiller@kcg.com wrote:

4

I think this calls for writing a Systemd unit file for sensu-client, where you can prioritise sensu-client to start up after all dependencies like network is up.

···

On Tuesday, 15 November 2016 20:41:07 UTC+5:30, jemi...@kcg.com wrote:

I’m just starting to set up a sensu monitoring environment. One thing that I’m seeing is that on Centos 7.1 sensu-client will fail to start sometimes. This of course shows that the host is down.

Is there a way to have sensu-server to (re)start a sensu-client if there is no heartbeat received? How do you insure that the sensu-client is always running? I know there are ways to delay the start of the client until the network is up, but I’d like to ensure that the client is always up from the server perspective.

{“timestamp”:“2016-11-14T23:34:07.056437-0600”,“level”:“info”,“message”:“configuring sensu spawn”,“settings”:{“limit”:12}}

/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:677:in `connect_server’: unable to resolve server address: Inappropriate ioctl for device (EventMachine::ConnectionError)

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:677:in `bind_connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:653:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp/session.rb:457:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp.rb:242:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:188:in `setup_connection’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:218:in `connect_with_eligible_options’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:20:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport.rb:24:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/daemon.rb:253:in `setup_transport’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:451:in `start’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:19:in `block in run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in `run_machine’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in `run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:18:in `run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/exe/sensu-client:10:in `<top (required)>’

from /opt/sensu/bin/sensu-client:22:in `load’

from /opt/sensu/bin/sensu-client:22:in `’

Thanks

5

Thanks, has anyone created the systemd service yet? Also can systemd watchdog the service and restart if it dies?

···

On Wednesday, November 23, 2016 at 11:20:01 AM UTC-5, Jay Kumar wrote:

I think this calls for writing a Systemd unit file for sensu-client, where you can prioritise sensu-client to start up after all dependencies like network is up.

On Tuesday, 15 November 2016 20:41:07 UTC+5:30, jemi...@kcg.com wrote:

I’m just starting to set up a sensu monitoring environment. One thing that I’m seeing is that on Centos 7.1 sensu-client will fail to start sometimes. This of course shows that the host is down.

Is there a way to have sensu-server to (re)start a sensu-client if there is no heartbeat received? How do you insure that the sensu-client is always running? I know there are ways to delay the start of the client until the network is up, but I’d like to ensure that the client is always up from the server perspective.

{“timestamp”:“2016-11-14T23:34:07.056437-0600”,“level”:“info”,“message”:“configuring sensu spawn”,“settings”:{“limit”:12}}

/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:677:in `connect_server’: unable to resolve server address: Inappropriate ioctl for device (EventMachine::ConnectionError)

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:677:in `bind_connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:653:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp/session.rb:457:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp.rb:242:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:188:in `setup_connection’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:218:in `connect_with_eligible_options’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:20:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport.rb:24:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/daemon.rb:253:in `setup_transport’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:451:in `start’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:19:in `block in run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in `run_machine’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in `run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:18:in `run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/exe/sensu-client:10:in `<top (required)>’

from /opt/sensu/bin/sensu-client:22:in `load’

from /opt/sensu/bin/sensu-client:22:in `’

Thanks

6

https://github.com/sensu/sensu-build/blob/master/sensu_configs/systemd/sensu-client.service

Hope this helps.

···

On Tuesday, 15 November 2016 20:41:07 UTC+5:30, jemi...@kcg.com wrote:

I’m just starting to set up a sensu monitoring environment. One thing that I’m seeing is that on Centos 7.1 sensu-client will fail to start sometimes. This of course shows that the host is down.

Is there a way to have sensu-server to (re)start a sensu-client if there is no heartbeat received? How do you insure that the sensu-client is always running? I know there are ways to delay the start of the client until the network is up, but I’d like to ensure that the client is always up from the server perspective.

{“timestamp”:“2016-11-14T23:34:07.056437-0600”,“level”:“info”,“message”:“configuring sensu spawn”,“settings”:{“limit”:12}}

/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:677:in `connect_server’: unable to resolve server address: Inappropriate ioctl for device (EventMachine::ConnectionError)

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:677:in `bind_connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:653:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp/session.rb:457:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp.rb:242:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:188:in `setup_connection’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:218:in `connect_with_eligible_options’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:20:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport.rb:24:in `connect’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/daemon.rb:253:in `setup_transport’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:451:in `start’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:19:in `block in run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in `run_machine’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in `run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/lib/sensu/client/process.rb:18:in `run’

from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.5/exe/sensu-client:10:in `<top (required)>’

from /opt/sensu/bin/sensu-client:22:in `load’

from /opt/sensu/bin/sensu-client:22:in `’

Thanks