Sensu client on VMware vCenter Server Appliance 6

Sensu Classic (EOL)
1

Hi,

I have been trying to get sensu client running on VMware vCenter Server Appliance 6.0.0.20001 with no luck.

These are the details of the server.

atvpocvcen56-v6:/var/run/sensu # uname -a
Linux atvpocvcen56-v6 3.0.101-0.47.71-default #1 SMP Thu Nov 12 12:22:22 UTC 2015 (b5b212e) x86_64 x86_64 x86_64 GNU/Linux

atvpocvcen56-v6:/var/run/sensu # cat /etc/*release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3

See below for output showing the problem…

atvpocvcen56-v6:/var/run/sensu # service sensu-client status
/etc/init.d/sensu-service: line 301: kill: (2667) - No such process
sensu-client dead but pid file exists

atvpocvcen56-v6:/var/run/sensu # cat /var/run/sensu/sensu-client.pid
2667

atvpocvcen56-v6:/var/run/sensu # ps -ef | grep 2667
root 22001 6217 0 14:12 pts/1 00:00:00 grep 2667

atvpocvcen56-v6:/var/run/sensu # rm /var/run/sensu/sensu-client.pid

atvpocvcen56-v6:/var/run/sensu # service sensu-client restart
Stopping sensu-client done
Starting sensu-client
failed
atvpocvcen56-v6:/var/run/sensu # service sensu-client status
/etc/init.d/sensu-service: line 301: kill: (27901) - No such process
sensu-client dead but pid file exists

Has anyone managed to get this working ?

Regards,

NJ

2

What do the logs say? /var/log/sensu/sensu-client.log

···

On Thursday, 22 September 2016 16:05:15 UTC+1, NJOM wrote:

Hi,

I have been trying to get sensu client running on VMware vCenter Server Appliance 6.0.0.20001 with no luck.

These are the details of the server.

atvpocvcen56-v6:/var/run/sensu # uname -a
Linux atvpocvcen56-v6 3.0.101-0.47.71-default #1 SMP Thu Nov 12 12:22:22 UTC 2015 (b5b212e) x86_64 x86_64 x86_64 GNU/Linux

atvpocvcen56-v6:/var/run/sensu # cat /etc/*release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3

See below for output showing the problem…

atvpocvcen56-v6:/var/run/sensu # service sensu-client status
/etc/init.d/sensu-service: line 301: kill: (2667) - No such process
sensu-client dead but pid file exists

atvpocvcen56-v6:/var/run/sensu # cat /var/run/sensu/sensu-client.pid
2667

atvpocvcen56-v6:/var/run/sensu # ps -ef | grep 2667
root 22001 6217 0 14:12 pts/1 00:00:00 grep 2667

atvpocvcen56-v6:/var/run/sensu # rm /var/run/sensu/sensu-client.pid

atvpocvcen56-v6:/var/run/sensu # service sensu-client restart
Stopping sensu-client done
Starting sensu-client
failed
atvpocvcen56-v6:/var/run/sensu # service sensu-client status
/etc/init.d/sensu-service: line 301: kill: (27901) - No such process
sensu-client dead but pid file exists

Has anyone managed to get this working ?

Regards,

NJ

3

to tidy up logs first …

atvpocvcen56-v6:/tmp # cat /dev/null > /var/log/sensu/sensu-client.log

atvpocvcen56-v6:/tmp # service sensu-client restart

atvpocvcen56-v6:/tmp # tail -100 /var/log/sensu/sensu-client.log
{“timestamp”:“2016-09-22T15:36:26.320043+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:"/etc/sensu/config.json"}
{“timestamp”:“2016-09-22T15:36:26.320197+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:"/etc/sensu/config.json"}
{“timestamp”:“2016-09-22T15:36:26.320267+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:"/etc/sensu/conf.d"}
{“timestamp”:“2016-09-22T15:36:26.320329+0000”,“level”:“warn”,“message”:“loading config file”,“file”:"/etc/sensu/conf.d/client.json"}
{“timestamp”:“2016-09-22T15:36:26.320390+0000”,“level”:“warn”,“message”:“loading config file”,“file”:"/etc/sensu/conf.d/rabbitmq.json"}
{“timestamp”:“2016-09-22T15:36:26.320493+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:"/etc/sensu/conf.d/rabbitmq.json",“changes”:{“rabbitmq”:[null,{“ssl”:{“cert_chain_file”:"/etc/sensu/ssl/cert.pem",“private_key_file”:"/etc/sensu/ssl/key.pem"},“host”:“ieatsensu02”,“port”:5671,“vhost”:"/sensu",“user”:“sensu”,“password”:“REDACTED”}]}}
{“timestamp”:“2016-09-22T15:36:26.320569+0000”,“level”:“warn”,“message”:“loading config file”,“file”:"/etc/sensu/conf.d/transport.json"}
{“timestamp”:“2016-09-22T15:36:26.320688+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:"/etc/sensu/conf.d/transport.json",“changes”:{}}
{“timestamp”:“2016-09-22T15:36:26.320768+0000”,“level”:“warn”,“message”:“applied sensu client overrides”,“client”:{“name”:“atvpocvcen56-v6”,“address”:“10.45.200.33”,“subscriptions”:[“test_servers”,“client:atvpocvcen56-v6”]}}
{“timestamp”:“2016-09-22T15:36:26.329497+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:"/etc/sensu/extensions"}
{“timestamp”:“2016-09-22T15:36:26.329622+0000”,“level”:“info”,“message”:“configuring sensu spawn”,“settings”:{“limit”:12}}
/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:430:in block in start_tls': Could not find /etc/sensu/ssl/key.pem for start_tls (EventMachine::FileNotFoundException) from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:428:ineach’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:428:in start_tls' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp/session.rb:1180:inupgrade_to_tls_if_necessary’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp/session.rb:583:in post_init' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:58:inblock in new’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:49:in instance_eval' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:49:innew’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:683:in bind_connect' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:653:inconnect’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp/session.rb:457:in connect' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp.rb:242:inconnect’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:188:in setup_connection' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:218:inconnect_with_eligible_options’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:20:in connect' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport.rb:24:inconnect’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.1/lib/sensu/daemon.rb:253:in setup_transport' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.1/lib/sensu/client/process.rb:441:instart’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.1/lib/sensu/client/process.rb:19:in block in run' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:inrun_machine’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in run' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.1/lib/sensu/client/process.rb:18:inrun’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.1/exe/sensu-client:10:in <top (required)>' from /opt/sensu/bin/sensu-client:22:inload’
from /opt/sensu/bin/sensu-client:22:in `’
atvpocvcen56-v6:/tmp #

SSL files are present and have the same permissions as another server i am testing the sensu client on.

atvpocvcen56-v6:/tmp # ll /etc/sensu/ssl/
total 8
-rw-r–r-- 1 root root 1054 Sep 22 13:53 cert.pem
-rw-r–r-- 1 root root 1675 Sep 22 13:53 key.pem

a list of the installed gems …

atvpocvcen56-v6:/tmp # /opt/sensu/embedded/bin/gem list

*** LOCAL GEMS ***

amq-protocol (2.0.1)
amqp (1.6.0)
bigdecimal (default: 1.2.8)
childprocess (0.5.8)
did_you_mean (1.0.0)
em-http-server (0.1.8)
em-worker (0.0.2)
eventmachine (1.2.0.1)
ffi (1.9.14)
io-console (default: 0.4.5)
ipaddress (0.8.3)
json (default: 1.8.3)
minitest (5.8.3)
mixlib-cli (1.7.0)
mixlib-config (2.2.4)
mixlib-log (1.7.1)
mixlib-shellout (2.2.7)
net-telnet (0.1.1)
ohai (6.16.0)
oj (2.14.6)
power_assert (0.2.6)
psych (default: 2.0.17)
rake (10.4.2)
rdoc (default: 4.2.1)
rubygems-update (2.6.6)
sensu (0.26.1)
sensu-extension (1.5.0)
sensu-extensions (1.7.0)
sensu-extensions-debug (1.0.0)
sensu-extensions-json (1.0.0)
sensu-extensions-occurrences (1.1.0)
sensu-extensions-only-check-output (1.0.0)
sensu-extensions-ruby-hash (1.0.0)
sensu-json (2.0.0)
sensu-logger (1.2.0)
sensu-plugin (1.4.0, 1.2.0)
sensu-redis (1.6.0)
sensu-settings (9.1.1)
sensu-spawn (2.2.0)
sensu-transport (6.0.0)
systemu (2.6.5, 2.5.2)
test-unit (3.1.5)
yajl-ruby (1.2.1)
atvpocvcen56-v6:/tmp #

4

If Sensu has been set up automatically, the permissions are like this:

user@host:/etc/sensu/ssl# ls -lah
total 16K
drwxr-xr-x 2 sensu sensu 4.0K Feb 15 2016 .
drwxr-xr-x 8 sensu sensu 4.0K Sep 23 06:45 …
-r–r----- 1 sensu sensu 1.1K Feb 15 2016 cert.pem
-r–r----- 1 sensu sensu 1.7K Feb 15 2016 key.pem

``

You could also try making sure that sensu can read the file and it at least looks like the correct file.

What it should look like:

user@host:/etc/sensu/ssl# sudo -u sensu file /etc/sensu/ssl/key.pem
/etc/sensu/ssl/key.pem: PEM RSA private key

``

If there are permission issues it will look similar to this:

user@host:/etc/sensu/ssl# sudo -u sensu file /root/.ssh/known_hosts
/root/.ssh/known_hosts: ERROR: cannot open `/root/.ssh/known_hosts’ (Permission denied)

``

···

On Thursday, September 22, 2016 at 5:50:09 PM UTC+2, NJOM wrote:

/opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:430:in block in start_tls': Could not find /etc/sensu/ssl/key.pem for start_tls (EventMachine::FileNotFoundException) from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:428:in each’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:428:in start_tls' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp/session.rb:1180:in upgrade_to_tls_if_necessary’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp/session.rb:583:in post_init' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:58:in block in new’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:49:in instance_eval' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/em/connection.rb:49:in new’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:683:in bind_connect' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:653:in connect’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp/session.rb:457:in connect' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/amqp-1.6.0/lib/amqp.rb:242:in connect’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:188:in setup_connection' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:218:in connect_with_eligible_options’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport/rabbitmq.rb:20:in connect' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-transport-6.0.0/lib/sensu/transport.rb:24:in connect’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.1/lib/sensu/daemon.rb:253:in setup_transport' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.1/lib/sensu/client/process.rb:441:in start’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.1/lib/sensu/client/process.rb:19:in block in run' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in run_machine’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in run' from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.1/lib/sensu/client/process.rb:18:in run’
from /opt/sensu/embedded/lib/ruby/gems/2.3.0/gems/sensu-0.26.1/exe/sensu-client:10:in <top (required)>' from /opt/sensu/bin/sensu-client:22:in load’
from /opt/sensu/bin/sensu-client:22:in `’
atvpocvcen56-v6:/tmp #

SSL files are present and have the same permissions as another server i am testing the sensu client on.

atvpocvcen56-v6:/tmp # ll /etc/sensu/ssl/
total 8
-rw-r–r-- 1 root root 1054 Sep 22 13:53 cert.pem
-rw-r–r-- 1 root root 1675 Sep 22 13:53 key.pem

5

Thanks Alexander,

i removed the key.pem and cert.pem and recreated them as Sensu user with the same permissions you provided. Client is working now.

I am deploying the config using ansible. Other clients worked fine so not sure why there is a difference when using the vsphere appliance.

atvpocvcen56-v6:/etc/sensu/ssl # ls -lah

total 16K

drwx------ 2 sensu sensu 4.0K Sep 27 11:51 .

drwxr-xr-x 8 sensu sensu 4.0K Sep 27 11:48 …

-r–r----- 1 sensu sensu 1.1K Sep 27 11:51 cert.pem

-r–r----- 1 sensu sensu 1.7K Sep 27 11:51 key.pem

Regards,

NJ