Sensu client installed on a remote IP is not able to connect to rabbitmq

Since I am running my servers in Amazon AWS VPC, I don’t need to use ssl to connect to rabbitmq.

############## sensu-client config file on a remote IP ##############

[root@ip-10-1-0-24 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “10.1.0.120”,

“port”: “5673”,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on remote IP##############

[root@ip-10-1-0-24 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:47:57.220872+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:"/etc/sensu/config.json"}

{“timestamp”:“2015-01-20T15:47:57.221000+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:"/etc/sensu/config.json"}

{“timestamp”:“2015-01-20T15:47:57.221056+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:"/etc/sensu/conf.d"}

{“timestamp”:“2015-01-20T15:47:57.221125+0000”,“level”:“warn”,“message”:“loading config file”,“file”:"/etc/sensu/conf.d/rabbitmq.json"}

{“timestamp”:“2015-01-20T15:47:57.221177+0000”,“level”:“warn”,“message”:“loading config file”,“file”:"/etc/sensu/conf.d/client.json"}

{“timestamp”:“2015-01-20T15:47:57.221252+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:"/etc/sensu/conf.d/client.json",“changes”:{“client”:[null,{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:47:57.223738+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:"/etc/sensu/extensions"}

{“timestamp”:“2015-01-20T15:47:57.240916+0000”,“level”:“debug”,“message”:“connecting to transport”,“name”:“rabbitmq”,“settings”:{“host”:“10.1.0.120”,“port”:“5673”,“vhost”:"/sensu",“user”:“sensu”,“password”:“MYPASS”}}

{“timestamp”:“2015-01-20T15:47:57.307724+0000”,“level”:“debug”,“message”:“binding client tcp and udp sockets”,“options”:{“bind”:“127.0.0.1”,“port”:3030}}

{“timestamp”:“2015-01-20T15:47:57.307911+0000”,“level”:“debug”,“message”:“scheduling keepalives”}

{“timestamp”:“2015-01-20T15:47:57.308000+0000”,“level”:“debug”,“message”:“publishing keepalive”,“payload”:{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”],“version”:“0.15.0.beta”,“timestamp”:1421768877}}

{“timestamp”:“2015-01-20T15:47:57.308174+0000”,“level”:“debug”,“message”:“subscribing to client subscriptions”}

{“timestamp”:“2015-01-20T15:47:57.308224+0000”,“level”:“debug”,“message”:“subscribing to a subscription”,“subscription”:“all”}

{“timestamp”:“2015-01-20T15:47:57.308344+0000”,“level”:“debug”,“message”:“scheduling standalone checks”}

{“timestamp”:“2015-01-20T15:47:57.308574+0000”,“level”:“error”,“message”:"[amqp] Detected TCP connection failure"}

{“timestamp”:“2015-01-20T15:47:57.308634+0000”,“level”:“warn”,“message”:“reconnecting to transport”}

{“timestamp”:“2015-01-20T15:48:02.310492+0000”,“level”:“error”,“message”:"[amqp] Detected TCP connection failure"}

{“timestamp”:“2015-01-20T15:48:07.312180+0000”,“level”:“error”,“message”:"[amqp] Detected TCP connection failure"}

{“timestamp”:“2015-01-20T15:48:12.313845+0000”,“level”:“error”,“message”:"[amqp] Detected TCP connection failure"}

{“timestamp”:“2015-01-20T15:48:17.314603+0000”,“level”:“error”,“message”:"[amqp] Detected TCP connection failure"}

{“timestamp”:“2015-01-20T15:48:22.316323+0000”,“level”:“error”,“message”:"[amqp] Detected TCP connection failure"}

############## rabbitmq config file on server ##############

[root@ip-10-1-0-120 ~]# cat /etc/rabbitmq/rabbitmq.config

[

{rabbit, [

{tcp_listeners, [5673]}

]}

].

############## sensu-client config file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “localhost”,

“port”: 5673,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:50:31.715960+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:"/etc/sensu/config.json"}

{“timestamp”:“2015-01-20T15:50:31.716094+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:"/etc/sensu/config.json"}

{“timestamp”:“2015-01-20T15:50:31.716218+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:"/etc/sensu/conf.d"}

{“timestamp”:“2015-01-20T15:50:31.716283+0000”,“level”:“warn”,“message”:“loading config file”,“file”:"/etc/sensu/conf.d/api.json"}

{“timestamp”:“2015-01-20T15:50:31.716344+0000”,“level”:“warn”,“message”:“loading config file”,“file”:"/etc/sensu/conf.d/rabbitmq.json"}

{“timestamp”:“2015-01-20T15:50:31.716438+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:"/etc/sensu/conf.d/rabbitmq.json",“changes”:{“rabbitmq”:[null,{“host”:“localhost”,“port”:5673,“vhost”:"/sensu",“user”:“sensu”,“password”:“REDACTED”}]}}

{“timestamp”:“2015-01-20T15:50:31.716513+0000”,“level”:“warn”,“message”:“loading config file”,“file”:"/etc/sensu/conf.d/redis.json"}

{“timestamp”:“2015-01-20T15:50:31.716595+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:"/etc/sensu/conf.d/redis.json",“changes”:{“redis”:[null,{“host”:“localhost”,“port”:6379}]}}

{“timestamp”:“2015-01-20T15:50:31.716681+0000”,“level”:“warn”,“message”:“loading config file”,“file”:"/etc/sensu/conf.d/client.json"}

{“timestamp”:“2015-01-20T15:50:31.716764+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:"/etc/sensu/conf.d/client.json",“changes”:{“client”:[null,{“name”:“sensu-server”,“address”:“10.1.0.120”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:50:31.719598+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:"/etc/sensu/extensions"}

############## rabbitmq log file on server ##############

[root@ip-10-1-0-120 ~]# vim /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

[root@ip-10-1-0-120 ~]#

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

node : rabbit@ip-10-1-0-120

home dir : /var/lib/rabbitmq

config file(s) : /etc/rabbitmq/rabbitmq.config

cookie hash : A5sqFbNca8sDbE7JoSYtlg==

log : /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

sasl log : /var/log/rabbitmq/rabbit@ip-10-1-0-120-sasl.log

database dir : /var/lib/rabbitmq/mnesia/rabbit@ip-10-1-0-120

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Memory limit set to 801MB of 2004MB total.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Disk free limit set to 50MB

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

started TCP Listener on [::]:5673

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Management plugin started. Port: 15672

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Statistics database started.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Server startup complete; 6 plugins started.

  • rabbitmq_management

  • rabbitmq_web_dispatch

  • webmachine

  • mochiweb

  • rabbitmq_management_agent

  • amqp_client

=INFO REPORT==== 20-Jan-2015::15:32:04 ===

accepting AMQP connection <0.285.0> (127.0.0.1:35151 -> 127.0.0.1:5673)

···

===============================================================================================

Looks like the local sensu client (running on 10.1.0.120, i.e. the same IP where rabbitmq is running) is able to connect to rabbitmq. But the sensu client running on a remote IP is not able to connect to rabbitmq

Though I am to able to telnet from the remote client to rabbitmq:

[root@ip-10-1-0-24 ~]# telnet 10.1.0.120 5673

Trying 10.1.0.120…

Connected to 10.1.0.120.

Escape character is ‘^]’.

asdf

Connection closed by foreign host.

Please help.

Need a tuple for the RabbitMQ listener, {“0.0.0.0”, 5673}. Also, ensure you allow traffic in the VPC security group(s).

···

On Jan 20, 2015 7:57 AM, “Vrn” shankarvarun1@gmail.com wrote:

Since I am running my servers in Amazon AWS VPC, I don’t need to use ssl to connect to rabbitmq.

############## sensu-client config file on a remote IP ##############

[root@ip-10-1-0-24 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “10.1.0.120”,

“port”: “5673”,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on remote IP##############

[root@ip-10-1-0-24 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:47:57.220872+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:47:57.221000+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:47:57.221056+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:“/etc/sensu/conf.d”}

{“timestamp”:“2015-01-20T15:47:57.221125+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/rabbitmq.json”}

{“timestamp”:“2015-01-20T15:47:57.221177+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/client.json”}

{“timestamp”:“2015-01-20T15:47:57.221252+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/client.json”,“changes”:{“client”:[null,{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:47:57.223738+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:“/etc/sensu/extensions”}

{“timestamp”:“2015-01-20T15:47:57.240916+0000”,“level”:“debug”,“message”:“connecting to transport”,“name”:“rabbitmq”,“settings”:{“host”:“10.1.0.120”,“port”:“5673”,“vhost”:“/sensu”,“user”:“sensu”,“password”:“MYPASS”}}

{“timestamp”:“2015-01-20T15:47:57.307724+0000”,“level”:“debug”,“message”:“binding client tcp and udp sockets”,“options”:{“bind”:“127.0.0.1”,“port”:3030}}

{“timestamp”:“2015-01-20T15:47:57.307911+0000”,“level”:“debug”,“message”:“scheduling keepalives”}

{“timestamp”:“2015-01-20T15:47:57.308000+0000”,“level”:“debug”,“message”:“publishing keepalive”,“payload”:{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”],“version”:“0.15.0.beta”,“timestamp”:1421768877}}

{“timestamp”:“2015-01-20T15:47:57.308174+0000”,“level”:“debug”,“message”:“subscribing to client subscriptions”}

{“timestamp”:“2015-01-20T15:47:57.308224+0000”,“level”:“debug”,“message”:“subscribing to a subscription”,“subscription”:“all”}

{“timestamp”:“2015-01-20T15:47:57.308344+0000”,“level”:“debug”,“message”:“scheduling standalone checks”}

{“timestamp”:“2015-01-20T15:47:57.308574+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:47:57.308634+0000”,“level”:“warn”,“message”:“reconnecting to transport”}

{“timestamp”:“2015-01-20T15:48:02.310492+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:07.312180+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:12.313845+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:17.314603+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:22.316323+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

############## rabbitmq config file on server ##############

[root@ip-10-1-0-120 ~]# cat /etc/rabbitmq/rabbitmq.config

[

{rabbit, [

{tcp_listeners, [5673]}

]}

].

############## sensu-client config file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “localhost”,

“port”: 5673,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:50:31.715960+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:50:31.716094+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:50:31.716218+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:“/etc/sensu/conf.d”}

{“timestamp”:“2015-01-20T15:50:31.716283+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/api.json”}

{“timestamp”:“2015-01-20T15:50:31.716344+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/rabbitmq.json”}

{“timestamp”:“2015-01-20T15:50:31.716438+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/rabbitmq.json”,“changes”:{“rabbitmq”:[null,{“host”:“localhost”,“port”:5673,“vhost”:“/sensu”,“user”:“sensu”,“password”:“REDACTED”}]}}

{“timestamp”:“2015-01-20T15:50:31.716513+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/redis.json”}

{“timestamp”:“2015-01-20T15:50:31.716595+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/redis.json”,“changes”:{“redis”:[null,{“host”:“localhost”,“port”:6379}]}}

{“timestamp”:“2015-01-20T15:50:31.716681+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/client.json”}

{“timestamp”:“2015-01-20T15:50:31.716764+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/client.json”,“changes”:{“client”:[null,{“name”:“sensu-server”,“address”:“10.1.0.120”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:50:31.719598+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:“/etc/sensu/extensions”}

############## rabbitmq log file on server ##############

[root@ip-10-1-0-120 ~]# vim /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

[root@ip-10-1-0-120 ~]#

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

node : rabbit@ip-10-1-0-120

home dir : /var/lib/rabbitmq

config file(s) : /etc/rabbitmq/rabbitmq.config

cookie hash : A5sqFbNca8sDbE7JoSYtlg==

log : /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

sasl log : /var/log/rabbitmq/rabbit@ip-10-1-0-120-sasl.log

database dir : /var/lib/rabbitmq/mnesia/rabbit@ip-10-1-0-120

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Memory limit set to 801MB of 2004MB total.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Disk free limit set to 50MB

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

started TCP Listener on [::]:5673

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Management plugin started. Port: 15672

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Statistics database started.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Server startup complete; 6 plugins started.

  • rabbitmq_management
  • rabbitmq_web_dispatch
  • webmachine
  • mochiweb
  • rabbitmq_management_agent
  • amqp_client

=INFO REPORT==== 20-Jan-2015::15:32:04 ===

accepting AMQP connection <0.285.0> (127.0.0.1:35151127.0.0.1:5673)

===============================================================================================

Looks like the local sensu client (running on 10.1.0.120, i.e. the same IP where rabbitmq is running) is able to connect to rabbitmq. But the sensu client running on a remote IP is not able to connect to rabbitmq

Though I am to able to telnet from the remote client to rabbitmq:

[root@ip-10-1-0-24 ~]# telnet 10.1.0.120 5673

Trying 10.1.0.120…

Connected to 10.1.0.120.

Escape character is ‘^]’.

asdf

Connection closed by foreign host.

Please help.

Thanks for replying Sean. I made the change suggested by you. That doesn’t solve the issue for me. I restarted both rabbitmq and sensu-client but I am still getting the same error in sensu-client log.

{“timestamp”:“2015-01-22T06:36:19.747621+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:24.749413+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:29.751060+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:34.752710+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

I am sure VPC security group is not a problem because telnet works fine on the given IP and port. What else should I try? Do I need any change in the sensu-client configuration? I am running the client in debug mode but that too is not giving any useful information.

···

On Tuesday, January 20, 2015 at 10:51:30 PM UTC+5:30, Sean Porter wrote:

Need a tuple for the RabbitMQ listener, {“0.0.0.0”, 5673}. Also, ensure you allow traffic in the VPC security group(s).

On Jan 20, 2015 7:57 AM, “Vrn” shanka...@gmail.com wrote:

Since I am running my servers in Amazon AWS VPC, I don’t need to use ssl to connect to rabbitmq.

############## sensu-client config file on a remote IP ##############

[root@ip-10-1-0-24 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “10.1.0.120”,

“port”: “5673”,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on remote IP##############

[root@ip-10-1-0-24 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:47:57.220872+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:47:57.221000+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:47:57.221056+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:“/etc/sensu/conf.d”}

{“timestamp”:“2015-01-20T15:47:57.221125+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/rabbitmq.json”}

{“timestamp”:“2015-01-20T15:47:57.221177+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/client.json”}

{“timestamp”:“2015-01-20T15:47:57.221252+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/client.json”,“changes”:{“client”:[null,{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:47:57.223738+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:“/etc/sensu/extensions”}

{“timestamp”:“2015-01-20T15:47:57.240916+0000”,“level”:“debug”,“message”:“connecting to transport”,“name”:“rabbitmq”,“settings”:{“host”:“10.1.0.120”,“port”:“5673”,“vhost”:“/sensu”,“user”:“sensu”,“password”:“MYPASS”}}

{“timestamp”:“2015-01-20T15:47:57.307724+0000”,“level”:“debug”,“message”:“binding client tcp and udp sockets”,“options”:{“bind”:“127.0.0.1”,“port”:3030}}

{“timestamp”:“2015-01-20T15:47:57.307911+0000”,“level”:“debug”,“message”:“scheduling keepalives”}

{“timestamp”:“2015-01-20T15:47:57.308000+0000”,“level”:“debug”,“message”:“publishing keepalive”,“payload”:{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”],“version”:“0.15.0.beta”,“timestamp”:1421768877}}

{“timestamp”:“2015-01-20T15:47:57.308174+0000”,“level”:“debug”,“message”:“subscribing to client subscriptions”}

{“timestamp”:“2015-01-20T15:47:57.308224+0000”,“level”:“debug”,“message”:“subscribing to a subscription”,“subscription”:“all”}

{“timestamp”:“2015-01-20T15:47:57.308344+0000”,“level”:“debug”,“message”:“scheduling standalone checks”}

{“timestamp”:“2015-01-20T15:47:57.308574+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:47:57.308634+0000”,“level”:“warn”,“message”:“reconnecting to transport”}

{“timestamp”:“2015-01-20T15:48:02.310492+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:07.312180+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:12.313845+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:17.314603+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:22.316323+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

############## rabbitmq config file on server ##############

[root@ip-10-1-0-120 ~]# cat /etc/rabbitmq/rabbitmq.config

[

{rabbit, [

{tcp_listeners, [5673]}

]}

].

############## sensu-client config file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “localhost”,

“port”: 5673,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:50:31.715960+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:50:31.716094+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:50:31.716218+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:“/etc/sensu/conf.d”}

{“timestamp”:“2015-01-20T15:50:31.716283+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/api.json”}

{“timestamp”:“2015-01-20T15:50:31.716344+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/rabbitmq.json”}

{“timestamp”:“2015-01-20T15:50:31.716438+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/rabbitmq.json”,“changes”:{“rabbitmq”:[null,{“host”:“localhost”,“port”:5673,“vhost”:“/sensu”,“user”:“sensu”,“password”:“REDACTED”}]}}

{“timestamp”:“2015-01-20T15:50:31.716513+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/redis.json”}

{“timestamp”:“2015-01-20T15:50:31.716595+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/redis.json”,“changes”:{“redis”:[null,{“host”:“localhost”,“port”:6379}]}}

{“timestamp”:“2015-01-20T15:50:31.716681+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/client.json”}

{“timestamp”:“2015-01-20T15:50:31.716764+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/client.json”,“changes”:{“client”:[null,{“name”:“sensu-server”,“address”:“10.1.0.120”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:50:31.719598+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:“/etc/sensu/extensions”}

############## rabbitmq log file on server ##############

[root@ip-10-1-0-120 ~]# vim /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

[root@ip-10-1-0-120 ~]#

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

node : rabbit@ip-10-1-0-120

home dir : /var/lib/rabbitmq

config file(s) : /etc/rabbitmq/rabbitmq.config

cookie hash : A5sqFbNca8sDbE7JoSYtlg==

log : /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

sasl log : /var/log/rabbitmq/rabbit@ip-10-1-0-120-sasl.log

database dir : /var/lib/rabbitmq/mnesia/rabbit@ip-10-1-0-120

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Memory limit set to 801MB of 2004MB total.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Disk free limit set to 50MB

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

started TCP Listener on [::]:5673

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Management plugin started. Port: 15672

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Statistics database started.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Server startup complete; 6 plugins started.

  • rabbitmq_management
  • rabbitmq_web_dispatch
  • webmachine
  • mochiweb
  • rabbitmq_management_agent
  • amqp_client

=INFO REPORT==== 20-Jan-2015::15:32:04 ===

accepting AMQP connection <0.285.0> (127.0.0.1:35151127.0.0.1:5673)

===============================================================================================

Looks like the local sensu client (running on 10.1.0.120, i.e. the same IP where rabbitmq is running) is able to connect to rabbitmq. But the sensu client running on a remote IP is not able to connect to rabbitmq

Though I am to able to telnet from the remote client to rabbitmq:

[root@ip-10-1-0-24 ~]# telnet 10.1.0.120 5673

Trying 10.1.0.120…

Connected to 10.1.0.120.

Escape character is ‘^]’.

asdf

Connection closed by foreign host.

Please help.

I am posting some more debug info:

making sure that the user and vhost exist and the user has permission on the given vhost

root@ip-10-1-0-120 ~]# rabbitmqctl list_users

Listing users …

guest [administrator]

sensu [administrator]

[root@ip-10-1-0-120 ~]# rabbitmqctl list_user_permissions sensu

Listing permissions for user “sensu” …

/sensu .* .* .*

Making sure rabbitmq is listening on the given port

[root@ip-10-1-0-120 ~]# lsof -i :5673

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

sensu-api 2308 sensu 11u IPv4 13648 0t0 TCP localhost:34608->localhost:jms (ESTABLISHED)

sensu-cli 2334 sensu 9u IPv4 13651 0t0 TCP localhost:34609->localhost:jms (ESTABLISHED)

sensu-ser 2360 sensu 10u IPv4 13657 0t0 TCP localhost:34610->localhost:jms (ESTABLISHED)

beam 3432 rabbitmq 9u IPv4 13649 0t0 TCP localhost:jms->localhost:34608 (ESTABLISHED)

beam 3432 rabbitmq 16u IPv4 13639 0t0 TCP *:jms (LISTEN)

beam 3432 rabbitmq 19u IPv4 13653 0t0 TCP localhost:jms->localhost:34609 (ESTABLISHED)

beam 3432 rabbitmq 20u IPv4 13658 0t0 TCP localhost:jms->localhost:34610 (ESTABLISHED)

···

On Thursday, January 22, 2015 at 12:14:09 PM UTC+5:30, Vrn wrote:

Thanks for replying Sean. I made the change suggested by you. That doesn’t solve the issue for me. I restarted both rabbitmq and sensu-client but I am still getting the same error in sensu-client log.

{“timestamp”:“2015-01-22T06:36:19.747621+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:24.749413+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:29.751060+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:34.752710+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

I am sure VPC security group is not a problem because telnet works fine on the given IP and port. What else should I try? Do I need any change in the sensu-client configuration? I am running the client in debug mode but that too is not giving any useful information.

On Tuesday, January 20, 2015 at 10:51:30 PM UTC+5:30, Sean Porter wrote:

Need a tuple for the RabbitMQ listener, {“0.0.0.0”, 5673}. Also, ensure you allow traffic in the VPC security group(s).

On Jan 20, 2015 7:57 AM, “Vrn” shanka...@gmail.com wrote:

Since I am running my servers in Amazon AWS VPC, I don’t need to use ssl to connect to rabbitmq.

############## sensu-client config file on a remote IP ##############

[root@ip-10-1-0-24 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “10.1.0.120”,

“port”: “5673”,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on remote IP##############

[root@ip-10-1-0-24 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:47:57.220872+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:47:57.221000+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:47:57.221056+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:“/etc/sensu/conf.d”}

{“timestamp”:“2015-01-20T15:47:57.221125+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/rabbitmq.json”}

{“timestamp”:“2015-01-20T15:47:57.221177+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/client.json”}

{“timestamp”:“2015-01-20T15:47:57.221252+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/client.json”,“changes”:{“client”:[null,{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:47:57.223738+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:“/etc/sensu/extensions”}

{“timestamp”:“2015-01-20T15:47:57.240916+0000”,“level”:“debug”,“message”:“connecting to transport”,“name”:“rabbitmq”,“settings”:{“host”:“10.1.0.120”,“port”:“5673”,“vhost”:“/sensu”,“user”:“sensu”,“password”:“MYPASS”}}

{“timestamp”:“2015-01-20T15:47:57.307724+0000”,“level”:“debug”,“message”:“binding client tcp and udp sockets”,“options”:{“bind”:“127.0.0.1”,“port”:3030}}

{“timestamp”:“2015-01-20T15:47:57.307911+0000”,“level”:“debug”,“message”:“scheduling keepalives”}

{“timestamp”:“2015-01-20T15:47:57.308000+0000”,“level”:“debug”,“message”:“publishing keepalive”,“payload”:{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”],“version”:“0.15.0.beta”,“timestamp”:1421768877}}

{“timestamp”:“2015-01-20T15:47:57.308174+0000”,“level”:“debug”,“message”:“subscribing to client subscriptions”}

{“timestamp”:“2015-01-20T15:47:57.308224+0000”,“level”:“debug”,“message”:“subscribing to a subscription”,“subscription”:“all”}

{“timestamp”:“2015-01-20T15:47:57.308344+0000”,“level”:“debug”,“message”:“scheduling standalone checks”}

{“timestamp”:“2015-01-20T15:47:57.308574+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:47:57.308634+0000”,“level”:“warn”,“message”:“reconnecting to transport”}

{“timestamp”:“2015-01-20T15:48:02.310492+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:07.312180+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:12.313845+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:17.314603+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:22.316323+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

############## rabbitmq config file on server ##############

[root@ip-10-1-0-120 ~]# cat /etc/rabbitmq/rabbitmq.config

[

{rabbit, [

{tcp_listeners, [5673]}

]}

].

############## sensu-client config file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “localhost”,

“port”: 5673,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:50:31.715960+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:50:31.716094+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:50:31.716218+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:“/etc/sensu/conf.d”}

{“timestamp”:“2015-01-20T15:50:31.716283+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/api.json”}

{“timestamp”:“2015-01-20T15:50:31.716344+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/rabbitmq.json”}

{“timestamp”:“2015-01-20T15:50:31.716438+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/rabbitmq.json”,“changes”:{“rabbitmq”:[null,{“host”:“localhost”,“port”:5673,“vhost”:“/sensu”,“user”:“sensu”,“password”:“REDACTED”}]}}

{“timestamp”:“2015-01-20T15:50:31.716513+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/redis.json”}

{“timestamp”:“2015-01-20T15:50:31.716595+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/redis.json”,“changes”:{“redis”:[null,{“host”:“localhost”,“port”:6379}]}}

{“timestamp”:“2015-01-20T15:50:31.716681+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/client.json”}

{“timestamp”:“2015-01-20T15:50:31.716764+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/client.json”,“changes”:{“client”:[null,{“name”:“sensu-server”,“address”:“10.1.0.120”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:50:31.719598+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:“/etc/sensu/extensions”}

############## rabbitmq log file on server ##############

[root@ip-10-1-0-120 ~]# vim /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

[root@ip-10-1-0-120 ~]#

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

node : rabbit@ip-10-1-0-120

home dir : /var/lib/rabbitmq

config file(s) : /etc/rabbitmq/rabbitmq.config

cookie hash : A5sqFbNca8sDbE7JoSYtlg==

log : /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

sasl log : /var/log/rabbitmq/rabbit@ip-10-1-0-120-sasl.log

database dir : /var/lib/rabbitmq/mnesia/rabbit@ip-10-1-0-120

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Memory limit set to 801MB of 2004MB total.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Disk free limit set to 50MB

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

started TCP Listener on [::]:5673

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Management plugin started. Port: 15672

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Statistics database started.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Server startup complete; 6 plugins started.

  • rabbitmq_management
  • rabbitmq_web_dispatch
  • webmachine
  • mochiweb
  • rabbitmq_management_agent
  • amqp_client

=INFO REPORT==== 20-Jan-2015::15:32:04 ===

accepting AMQP connection <0.285.0> (127.0.0.1:35151127.0.0.1:5673)

===============================================================================================

Looks like the local sensu client (running on 10.1.0.120, i.e. the same IP where rabbitmq is running) is able to connect to rabbitmq. But the sensu client running on a remote IP is not able to connect to rabbitmq

Though I am to able to telnet from the remote client to rabbitmq:

[root@ip-10-1-0-24 ~]# telnet 10.1.0.120 5673

Trying 10.1.0.120…

Connected to 10.1.0.120.

Escape character is ‘^]’.

asdf

Connection closed by foreign host.

Please help.

Can I do anything to log more information about the situation? Any help is appreciated.

···

On Thursday, January 22, 2015 at 12:27:14 PM UTC+5:30, Vrn wrote:

I am posting some more debug info:

making sure that the user and vhost exist and the user has permission on the given vhost

root@ip-10-1-0-120 ~]# rabbitmqctl list_users

Listing users …

guest [administrator]

sensu [administrator]

[root@ip-10-1-0-120 ~]# rabbitmqctl list_user_permissions sensu

Listing permissions for user “sensu” …

/sensu .* .* .*

Making sure rabbitmq is listening on the given port

[root@ip-10-1-0-120 ~]# lsof -i :5673

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

sensu-api 2308 sensu 11u IPv4 13648 0t0 TCP localhost:34608->localhost:jms (ESTABLISHED)

sensu-cli 2334 sensu 9u IPv4 13651 0t0 TCP localhost:34609->localhost:jms (ESTABLISHED)

sensu-ser 2360 sensu 10u IPv4 13657 0t0 TCP localhost:34610->localhost:jms (ESTABLISHED)

beam 3432 rabbitmq 9u IPv4 13649 0t0 TCP localhost:jms->localhost:34608 (ESTABLISHED)

beam 3432 rabbitmq 16u IPv4 13639 0t0 TCP *:jms (LISTEN)

beam 3432 rabbitmq 19u IPv4 13653 0t0 TCP localhost:jms->localhost:34609 (ESTABLISHED)

beam 3432 rabbitmq 20u IPv4 13658 0t0 TCP localhost:jms->localhost:34610 (ESTABLISHED)

On Thursday, January 22, 2015 at 12:14:09 PM UTC+5:30, Vrn wrote:

Thanks for replying Sean. I made the change suggested by you. That doesn’t solve the issue for me. I restarted both rabbitmq and sensu-client but I am still getting the same error in sensu-client log.

{“timestamp”:“2015-01-22T06:36:19.747621+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:24.749413+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:29.751060+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:34.752710+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

I am sure VPC security group is not a problem because telnet works fine on the given IP and port. What else should I try? Do I need any change in the sensu-client configuration? I am running the client in debug mode but that too is not giving any useful information.

On Tuesday, January 20, 2015 at 10:51:30 PM UTC+5:30, Sean Porter wrote:

Need a tuple for the RabbitMQ listener, {“0.0.0.0”, 5673}. Also, ensure you allow traffic in the VPC security group(s).

On Jan 20, 2015 7:57 AM, “Vrn” shanka...@gmail.com wrote:

Since I am running my servers in Amazon AWS VPC, I don’t need to use ssl to connect to rabbitmq.

############## sensu-client config file on a remote IP ##############

[root@ip-10-1-0-24 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “10.1.0.120”,

“port”: “5673”,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on remote IP##############

[root@ip-10-1-0-24 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:47:57.220872+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:47:57.221000+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:47:57.221056+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:“/etc/sensu/conf.d”}

{“timestamp”:“2015-01-20T15:47:57.221125+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/rabbitmq.json”}

{“timestamp”:“2015-01-20T15:47:57.221177+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/client.json”}

{“timestamp”:“2015-01-20T15:47:57.221252+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/client.json”,“changes”:{“client”:[null,{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:47:57.223738+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:“/etc/sensu/extensions”}

{“timestamp”:“2015-01-20T15:47:57.240916+0000”,“level”:“debug”,“message”:“connecting to transport”,“name”:“rabbitmq”,“settings”:{“host”:“10.1.0.120”,“port”:“5673”,“vhost”:“/sensu”,“user”:“sensu”,“password”:“MYPASS”}}

{“timestamp”:“2015-01-20T15:47:57.307724+0000”,“level”:“debug”,“message”:“binding client tcp and udp sockets”,“options”:{“bind”:“127.0.0.1”,“port”:3030}}

{“timestamp”:“2015-01-20T15:47:57.307911+0000”,“level”:“debug”,“message”:“scheduling keepalives”}

{“timestamp”:“2015-01-20T15:47:57.308000+0000”,“level”:“debug”,“message”:“publishing keepalive”,“payload”:{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”],“version”:“0.15.0.beta”,“timestamp”:1421768877}}

{“timestamp”:“2015-01-20T15:47:57.308174+0000”,“level”:“debug”,“message”:“subscribing to client subscriptions”}

{“timestamp”:“2015-01-20T15:47:57.308224+0000”,“level”:“debug”,“message”:“subscribing to a subscription”,“subscription”:“all”}

{“timestamp”:“2015-01-20T15:47:57.308344+0000”,“level”:“debug”,“message”:“scheduling standalone checks”}

{“timestamp”:“2015-01-20T15:47:57.308574+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:47:57.308634+0000”,“level”:“warn”,“message”:“reconnecting to transport”}

{“timestamp”:“2015-01-20T15:48:02.310492+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:07.312180+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:12.313845+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:17.314603+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:22.316323+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

############## rabbitmq config file on server ##############

[root@ip-10-1-0-120 ~]# cat /etc/rabbitmq/rabbitmq.config

[

{rabbit, [

{tcp_listeners, [5673]}

]}

].

############## sensu-client config file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “localhost”,

“port”: 5673,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:50:31.715960+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:50:31.716094+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:50:31.716218+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:“/etc/sensu/conf.d”}

{“timestamp”:“2015-01-20T15:50:31.716283+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/api.json”}

{“timestamp”:“2015-01-20T15:50:31.716344+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/rabbitmq.json”}

{“timestamp”:“2015-01-20T15:50:31.716438+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/rabbitmq.json”,“changes”:{“rabbitmq”:[null,{“host”:“localhost”,“port”:5673,“vhost”:“/sensu”,“user”:“sensu”,“password”:“REDACTED”}]}}

{“timestamp”:“2015-01-20T15:50:31.716513+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/redis.json”}

{“timestamp”:“2015-01-20T15:50:31.716595+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/redis.json”,“changes”:{“redis”:[null,{“host”:“localhost”,“port”:6379}]}}

{“timestamp”:“2015-01-20T15:50:31.716681+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/client.json”}

{“timestamp”:“2015-01-20T15:50:31.716764+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/client.json”,“changes”:{“client”:[null,{“name”:“sensu-server”,“address”:“10.1.0.120”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:50:31.719598+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:“/etc/sensu/extensions”}

############## rabbitmq log file on server ##############

[root@ip-10-1-0-120 ~]# vim /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

[root@ip-10-1-0-120 ~]#

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

node : rabbit@ip-10-1-0-120

home dir : /var/lib/rabbitmq

config file(s) : /etc/rabbitmq/rabbitmq.config

cookie hash : A5sqFbNca8sDbE7JoSYtlg==

log : /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

sasl log : /var/log/rabbitmq/rabbit@ip-10-1-0-120-sasl.log

database dir : /var/lib/rabbitmq/mnesia/rabbit@ip-10-1-0-120

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Memory limit set to 801MB of 2004MB total.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Disk free limit set to 50MB

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

started TCP Listener on [::]:5673

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Management plugin started. Port: 15672

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Statistics database started.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Server startup complete; 6 plugins started.

  • rabbitmq_management
  • rabbitmq_web_dispatch
  • webmachine
  • mochiweb
  • rabbitmq_management_agent
  • amqp_client

=INFO REPORT==== 20-Jan-2015::15:32:04 ===

accepting AMQP connection <0.285.0> (127.0.0.1:35151127.0.0.1:5673)

===============================================================================================

Looks like the local sensu client (running on 10.1.0.120, i.e. the same IP where rabbitmq is running) is able to connect to rabbitmq. But the sensu client running on a remote IP is not able to connect to rabbitmq

Though I am to able to telnet from the remote client to rabbitmq:

[root@ip-10-1-0-24 ~]# telnet 10.1.0.120 5673

Trying 10.1.0.120…

Connected to 10.1.0.120.

Escape character is ‘^]’.

asdf

Connection closed by foreign host.

Please help.

{“timestamp”:“2015-01-23T12:05:15.956576+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-23T12:05:15.956634+0000”,“level”:“warn”,“message”:“reconnecting to transport”}

{“timestamp”:“2015-01-23T12:05:20.958452+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-23T12:05:25.960121+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

I expect sensu client to say more than “Detected TCP connection failure”. It should say which IP:port etc., it is trying to connect.

Since telnet works for me, I believe the sensu client is somehow picking wrong IP or port. Is there any setting I can enable in the client so that it starts logging more info.

···

On Friday, January 23, 2015 at 5:17:59 PM UTC+5:30, Vrn wrote:

Can I do anything to log more information about the situation? Any help is appreciated.

On Thursday, January 22, 2015 at 12:27:14 PM UTC+5:30, Vrn wrote:

I am posting some more debug info:

making sure that the user and vhost exist and the user has permission on the given vhost

root@ip-10-1-0-120 ~]# rabbitmqctl list_users

Listing users …

guest [administrator]

sensu [administrator]

[root@ip-10-1-0-120 ~]# rabbitmqctl list_user_permissions sensu

Listing permissions for user “sensu” …

/sensu .* .* .*

Making sure rabbitmq is listening on the given port

[root@ip-10-1-0-120 ~]# lsof -i :5673

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

sensu-api 2308 sensu 11u IPv4 13648 0t0 TCP localhost:34608->localhost:jms (ESTABLISHED)

sensu-cli 2334 sensu 9u IPv4 13651 0t0 TCP localhost:34609->localhost:jms (ESTABLISHED)

sensu-ser 2360 sensu 10u IPv4 13657 0t0 TCP localhost:34610->localhost:jms (ESTABLISHED)

beam 3432 rabbitmq 9u IPv4 13649 0t0 TCP localhost:jms->localhost:34608 (ESTABLISHED)

beam 3432 rabbitmq 16u IPv4 13639 0t0 TCP *:jms (LISTEN)

beam 3432 rabbitmq 19u IPv4 13653 0t0 TCP localhost:jms->localhost:34609 (ESTABLISHED)

beam 3432 rabbitmq 20u IPv4 13658 0t0 TCP localhost:jms->localhost:34610 (ESTABLISHED)

On Thursday, January 22, 2015 at 12:14:09 PM UTC+5:30, Vrn wrote:

Thanks for replying Sean. I made the change suggested by you. That doesn’t solve the issue for me. I restarted both rabbitmq and sensu-client but I am still getting the same error in sensu-client log.

{“timestamp”:“2015-01-22T06:36:19.747621+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:24.749413+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:29.751060+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-22T06:36:34.752710+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

I am sure VPC security group is not a problem because telnet works fine on the given IP and port. What else should I try? Do I need any change in the sensu-client configuration? I am running the client in debug mode but that too is not giving any useful information.

On Tuesday, January 20, 2015 at 10:51:30 PM UTC+5:30, Sean Porter wrote:

Need a tuple for the RabbitMQ listener, {“0.0.0.0”, 5673}. Also, ensure you allow traffic in the VPC security group(s).

On Jan 20, 2015 7:57 AM, “Vrn” shanka...@gmail.com wrote:

Since I am running my servers in Amazon AWS VPC, I don’t need to use ssl to connect to rabbitmq.

############## sensu-client config file on a remote IP ##############

[root@ip-10-1-0-24 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “10.1.0.120”,

“port”: “5673”,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on remote IP##############

[root@ip-10-1-0-24 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:47:57.220872+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:47:57.221000+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:47:57.221056+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:“/etc/sensu/conf.d”}

{“timestamp”:“2015-01-20T15:47:57.221125+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/rabbitmq.json”}

{“timestamp”:“2015-01-20T15:47:57.221177+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/client.json”}

{“timestamp”:“2015-01-20T15:47:57.221252+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/client.json”,“changes”:{“client”:[null,{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:47:57.223738+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:“/etc/sensu/extensions”}

{“timestamp”:“2015-01-20T15:47:57.240916+0000”,“level”:“debug”,“message”:“connecting to transport”,“name”:“rabbitmq”,“settings”:{“host”:“10.1.0.120”,“port”:“5673”,“vhost”:“/sensu”,“user”:“sensu”,“password”:“MYPASS”}}

{“timestamp”:“2015-01-20T15:47:57.307724+0000”,“level”:“debug”,“message”:“binding client tcp and udp sockets”,“options”:{“bind”:“127.0.0.1”,“port”:3030}}

{“timestamp”:“2015-01-20T15:47:57.307911+0000”,“level”:“debug”,“message”:“scheduling keepalives”}

{“timestamp”:“2015-01-20T15:47:57.308000+0000”,“level”:“debug”,“message”:“publishing keepalive”,“payload”:{“name”:“sensu-client-ip-10-1-0-24”,“address”:“10.1.0.24”,“subscriptions”:[“all”],“version”:“0.15.0.beta”,“timestamp”:1421768877}}

{“timestamp”:“2015-01-20T15:47:57.308174+0000”,“level”:“debug”,“message”:“subscribing to client subscriptions”}

{“timestamp”:“2015-01-20T15:47:57.308224+0000”,“level”:“debug”,“message”:“subscribing to a subscription”,“subscription”:“all”}

{“timestamp”:“2015-01-20T15:47:57.308344+0000”,“level”:“debug”,“message”:“scheduling standalone checks”}

{“timestamp”:“2015-01-20T15:47:57.308574+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:47:57.308634+0000”,“level”:“warn”,“message”:“reconnecting to transport”}

{“timestamp”:“2015-01-20T15:48:02.310492+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:07.312180+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:12.313845+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:17.314603+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

{“timestamp”:“2015-01-20T15:48:22.316323+0000”,“level”:“error”,“message”:“[amqp] Detected TCP connection failure”}

############## rabbitmq config file on server ##############

[root@ip-10-1-0-120 ~]# cat /etc/rabbitmq/rabbitmq.config

[

{rabbit, [

{tcp_listeners, [5673]}

]}

].

############## sensu-client config file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

“host”: “localhost”,

“port”: 5673,

“vhost”: “/sensu”,

“user”: “sensu”,

“password”: “MYPASS”

}

}

############## sensu-client log file on the IP where rabbitmq is installed ##############

[root@ip-10-1-0-120 ~]# cat /var/log/sensu/sensu-client.log

{“timestamp”:“2015-01-20T15:50:31.715960+0000”,“level”:“warn”,“message”:“config file does not exist or is not readable”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:50:31.716094+0000”,“level”:“warn”,“message”:“ignoring config file”,“file”:“/etc/sensu/config.json”}

{“timestamp”:“2015-01-20T15:50:31.716218+0000”,“level”:“warn”,“message”:“loading config files from directory”,“directory”:“/etc/sensu/conf.d”}

{“timestamp”:“2015-01-20T15:50:31.716283+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/api.json”}

{“timestamp”:“2015-01-20T15:50:31.716344+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/rabbitmq.json”}

{“timestamp”:“2015-01-20T15:50:31.716438+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/rabbitmq.json”,“changes”:{“rabbitmq”:[null,{“host”:“localhost”,“port”:5673,“vhost”:“/sensu”,“user”:“sensu”,“password”:“REDACTED”}]}}

{“timestamp”:“2015-01-20T15:50:31.716513+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/redis.json”}

{“timestamp”:“2015-01-20T15:50:31.716595+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/redis.json”,“changes”:{“redis”:[null,{“host”:“localhost”,“port”:6379}]}}

{“timestamp”:“2015-01-20T15:50:31.716681+0000”,“level”:“warn”,“message”:“loading config file”,“file”:“/etc/sensu/conf.d/client.json”}

{“timestamp”:“2015-01-20T15:50:31.716764+0000”,“level”:“warn”,“message”:“config file applied changes”,“file”:“/etc/sensu/conf.d/client.json”,“changes”:{“client”:[null,{“name”:“sensu-server”,“address”:“10.1.0.120”,“subscriptions”:[“all”]}]}}

{“timestamp”:“2015-01-20T15:50:31.719598+0000”,“level”:“warn”,“message”:“loading extension files from directory”,“directory”:“/etc/sensu/extensions”}

############## rabbitmq log file on server ##############

[root@ip-10-1-0-120 ~]# vim /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

[root@ip-10-1-0-120 ~]#

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

node : rabbit@ip-10-1-0-120

home dir : /var/lib/rabbitmq

config file(s) : /etc/rabbitmq/rabbitmq.config

cookie hash : A5sqFbNca8sDbE7JoSYtlg==

log : /var/log/rabbitmq/rabbit@ip-10-1-0-120.log

sasl log : /var/log/rabbitmq/rabbit@ip-10-1-0-120-sasl.log

database dir : /var/lib/rabbitmq/mnesia/rabbit@ip-10-1-0-120

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Memory limit set to 801MB of 2004MB total.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Disk free limit set to 50MB

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

started TCP Listener on [::]:5673

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Management plugin started. Port: 15672

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Statistics database started.

=INFO REPORT==== 20-Jan-2015::15:32:03 ===

Server startup complete; 6 plugins started.

  • rabbitmq_management
  • rabbitmq_web_dispatch
  • webmachine
  • mochiweb
  • rabbitmq_management_agent
  • amqp_client

=INFO REPORT==== 20-Jan-2015::15:32:04 ===

accepting AMQP connection <0.285.0> (127.0.0.1:35151127.0.0.1:5673)

===============================================================================================

Looks like the local sensu client (running on 10.1.0.120, i.e. the same IP where rabbitmq is running) is able to connect to rabbitmq. But the sensu client running on a remote IP is not able to connect to rabbitmq

Though I am to able to telnet from the remote client to rabbitmq:

[root@ip-10-1-0-24 ~]# telnet 10.1.0.120 5673

Trying 10.1.0.120…

Connected to 10.1.0.120.

Escape character is ‘^]’.

asdf

Connection closed by foreign host.

Please help.

I am also facing same Issue not able to connect from remote client.Did you find any solution.Can help me here.

Same issue here. All of my settings look correct and when I do a rabbitmqctl -n rabbit@hostname status I return good data, but get this same message.
“level”:“error”,“message”:“[amqp] Detected TCP connection failure”

···

On Saturday, April 30, 2016 at 8:38:52 AM UTC-4, kumar....@olacabs.com wrote:

I am also facing same Issue not able to connect from remote client.Did you find any solution.Can help me here.