Rabbitmq authentication failure. wrong credentials?

Sensu Classic (EOL)
1

Hi All,

We are currently deploying Sensu into Production. We have RabbitMQ version 3.6.15 and Erlang 20.3.6 in our server -running CentOS.

Configurations are supposed to be correct as the same configurations run Sensu smoothly on our Development Environment - Ubuntu Server. We have looked for different solutions online but there aren’t enough resources available. We have also tried to upgrade the versions of RabbitMQ and Erlang to latest but it did not resolve the issue. Hoping you could provide suggestions. Thanks!

There are no error displayed when we tail the rabbitmq log but sensu-enterprise.log shows an error message related to amqp credentials

{“timestamp”:“2018-06-10T16:39:15.989000+0200”,“level”:“warn”,“message”:“transport connection error”,“reason”:“possible authentication failure. wrong credentials?”,“user”:“admin”}

/etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

"host": "xxx@localhost",

"port": 5671,

"user": "admin",

"password": "admin",

"vhost": "/sensu",

"ssl": {

"cert_chain_file": "/etc/sensu/ssl/cert.pem",

"private_key_file": "/etc/sensu/ssl/key.pem"}

}

}

``

/etc/rabbitmq/rabbitmq.config

[{rabbit, [

 {ssl_listeners, [5671]},

 {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},

                {certfile,"/etc/rabbitmq/ssl/cert.pem"},

                {keyfile,"/etc/rabbitmq/ssl/key.pem"},

                {versions, ['tlsv1.2']},

                {verify,verify_peer},

                {fail_if_no_peer_cert,true}]}

]},

{rabbitmq_management, [{listener, [{port, 15672}]}]}

].

``

2

Hey Kaye,

Try resetting the password to the admin user, maybe the password isn’t what you think it is.

rabbitmqctl change_password admin New-Password

If that doesn’t work tail the rabbitmq logs and look for any issues. And debug just as you would an authentication issue outside of sensu.

Thanks,

Josh

···

On Jun 10, 2018, at 9:27 AM, Kaye sanpascual.keeshia@gmail.com wrote:

Hi All,

We are currently deploying Sensu into Production. We have RabbitMQ version 3.6.15 and Erlang 20.3.6 in our server -running CentOS.

Configurations are supposed to be correct as the same configurations run Sensu smoothly on our Development Environment - Ubuntu Server. We have looked for different solutions online but there aren’t enough resources available. We have also tried to upgrade the versions of RabbitMQ and Erlang to latest but it did not resolve the issue. Hoping you could provide suggestions. Thanks!

There are no error displayed when we tail the rabbitmq log but sensu-enterprise.log shows an error message related to amqp credentials

{“timestamp”:“2018-06-10T16:39:15.989000+0200”,“level”:“warn”,“message”:“transport connection error”,“reason”:“possible authentication failure. wrong credentials?”,“user”:“admin”}

/etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

"host": "xxx@localhost",

"port": 5671,

"user": "admin",

"password": "admin",

"vhost": "/sensu",

"ssl": {

"cert_chain_file": "/etc/sensu/ssl/cert.pem",

"private_key_file": "/etc/sensu/ssl/key.pem"}

}

}

``

/etc/rabbitmq/rabbitmq.config

[{rabbit, [

 {ssl_listeners, [5671]},

 {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},

                {certfile,"/etc/rabbitmq/ssl/cert.pem"},

                {keyfile,"/etc/rabbitmq/ssl/key.pem"},

                {versions, ['tlsv1.2']},

                {verify,verify_peer},

                {fail_if_no_peer_cert,true}]}

]},

{rabbitmq_management, [{listener, [{port, 15672}]}]}

].

``

3

Hi Josh,

Thanks for your response! :slight_smile: I have already tried changing the admin password and also creating a new user but the issue persists.

rabbitmqctl add_user sensu changeme rabbitmqctl set_user_tags sensu administrator rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

I don't see any warning or error message when I tail rabbitmq logs:

[root@xxlocalhost rabbitmq]# tail -f rabbit@localhost.log

Statistics database started.

=INFO REPORT==== 11-Jun-2018::06:21:01 ===

Server startup complete; 6 plugins started.

 * rabbitmq_management

 * rabbitmq_management_agent

 * rabbitmq_web_dispatch

 * cowboy

 * amqp_client

 * cowlib

I kept on seeing the credential issue on Sensu Enterprise and Sensu client logs.

Regards,

Kaye

<details class='elided'>
<summary title='Show trimmed content'>&#183;&#183;&#183;</summary>

On Mon, Jun 11, 2018 at 12:17 AM, Josh Pasqualetto <joshua.pasqualetto@gmail.com> wrote:

> Hey Kaye,

> 

> Try resetting the password to the admin user, maybe the password isn’t what you think it is.

> 

> ```
> rabbitmqctl change_password admin New-Password
> ```

> 

> If that doesn’t work tail the rabbitmq logs and look for any issues. And debug just as you would an authentication issue outside of sensu.

> 

> Thanks,

> Josh

> On Jun 10, 2018, at 9:27 AM, Kaye <sanpascual.keeshia@gmail.com> wrote:
> 

> > Hi All,
> > 

> > We are currently deploying Sensu into Production. We have RabbitMQ version 3.6.15 and Erlang 20.3.6 in our server -running CentOS.

> > Configurations are supposed to be correct as the same configurations run Sensu smoothly on our Development Environment - Ubuntu Server. We have looked for different solutions online but there aren't enough resources available. We have also tried to upgrade the versions of RabbitMQ and Erlang to latest but it did not resolve the issue. Hoping you could provide suggestions. Thanks!

> > 

> > There are no error displayed when we tail the rabbitmq log but sensu-enterprise.log shows an error message related to amqp credentials

> > 

> > {"timestamp":"2018-06-10T16:39:15.989000+0200","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"admin"}

> > 

> > **/etc/sensu/conf.d/rabbitmq.json**

> > {

> > 
> >   "rabbitmq": {

> > 
> >     "host": "xxx@localhost",

> > 
> >     "port": 5671,

> > 
> >     "user": "admin",

> > 
> >     "password": "admin",

> > 
> >     "vhost": "/sensu",

> > 
> >     "ssl": {

> > 
> >     "cert_chain_file": "/etc/sensu/ssl/cert.pem",

> > 
> >     "private_key_file": "/etc/sensu/ssl/key.pem"}

> > 
> >   }

> > 
> > }

> > ``

> > 
> > **/etc/rabbitmq/rabbitmq.config**

> > [{rabbit, [

> > 
> >      {ssl_listeners, [5671]},

> > 
> >      {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},

> > 
> >                     {certfile,"/etc/rabbitmq/ssl/cert.pem"},

> > 
> >                     {keyfile,"/etc/rabbitmq/ssl/key.pem"},

> > 
> >                     {versions, ['tlsv1.2']},

> > 
> >                     {verify,verify_peer},

> > 
> >                     {fail_if_no_peer_cert,true}]}

> > 
> >     ]},

> > 
> >     {rabbitmq_management, [{listener, [{port, 15672}]}]}

> > 
> > ].

> > ``

> > 
> > 

> > 
> >

</details>
4

Sorry, this was the commands I used:

rabbitmqctl add_user sensu changeme

rabbitmqctl set_user_tags sensu administrator

rabbitmqctl set_permissions -p /sensu sensu “." ".” “.*”

···

On Mon, Jun 11, 2018 at 12:27 PM, Keeshia San Pascual sanpascual.keeshia@gmail.com wrote:

Hi Josh,

Thanks for your response! :slight_smile: I have already tried changing the admin password and also creating a new user but the issue persists.

rabbitmqctl add_user sensu changeme rabbitmqctl set_user_tags sensu administrator rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"


I don't see any warning or error message when I tail rabbitmq logs:

[root@xxlocalhost rabbitmq]# tail -f rabbit@localhost.log

Statistics database started.

=INFO REPORT==== 11-Jun-2018::06:21:01 ===

Server startup complete; 6 plugins started.

  • rabbitmq_management
  • rabbitmq_management_agent
  • rabbitmq_web_dispatch
  • cowboy
  • amqp_client
  • cowlib

I kept on seeing the credential issue on Sensu Enterprise and Sensu client logs.

Regards,

Kaye

On Mon, Jun 11, 2018 at 12:17 AM, Josh Pasqualetto joshua.pasqualetto@gmail.com wrote:

Hey Kaye,

Try resetting the password to the admin user, maybe the password isn’t what you think it is.

rabbitmqctl change_password admin New-Password

If that doesn’t work tail the rabbitmq logs and look for any issues. And debug just as you would an authentication issue outside of sensu.

Thanks,

Josh

On Jun 10, 2018, at 9:27 AM, Kaye sanpascual.keeshia@gmail.com wrote:

Hi All,

We are currently deploying Sensu into Production. We have RabbitMQ version 3.6.15 and Erlang 20.3.6 in our server -running CentOS.

Configurations are supposed to be correct as the same configurations run Sensu smoothly on our Development Environment - Ubuntu Server. We have looked for different solutions online but there aren’t enough resources available. We have also tried to upgrade the versions of RabbitMQ and Erlang to latest but it did not resolve the issue. Hoping you could provide suggestions. Thanks!

There are no error displayed when we tail the rabbitmq log but sensu-enterprise.log shows an error message related to amqp credentials

{“timestamp”:“2018-06-10T16:39:15.989000+0200”,“level”:“warn”,“message”:“transport connection error”,“reason”:“possible authentication failure. wrong credentials?”,“user”:“admin”}

/etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

"host": "xxx@localhost",

"port": 5671,

"user": "admin",

"password": "admin",

"vhost": "/sensu",

"ssl": {

"cert_chain_file": "/etc/sensu/ssl/cert.pem",

"private_key_file": "/etc/sensu/ssl/key.pem"}

}

}

``

/etc/rabbitmq/rabbitmq.config

[{rabbit, [

 {ssl_listeners, [5671]},

 {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},

                {certfile,"/etc/rabbitmq/ssl/cert.pem"},

                {keyfile,"/etc/rabbitmq/ssl/key.pem"},

                {versions, ['tlsv1.2']},

                {verify,verify_peer},

                {fail_if_no_peer_cert,true}]}

]},

{rabbitmq_management, [{listener, [{port, 15672}]}]}

].

``

5

Hey Keeshia,

Can you authenticate to rabbitmq outside of sensu?

E.g. curl -i -u sensu:password http://localhost:15672/api/whoami

I remember rabbitmq having an access log or perhaps you have to up the debug level. At any rate we need to deduce if it’s an auth problem with rabbitmq or a configuration/permissions issue.

Try to auth to rabbitmq outside sensu and go from there.

Thanks,

Josh

···

On Mon, Jun 11, 2018 at 12:27 PM, Keeshia San Pascual sanpascual.keeshia@gmail.com wrote:

Hi Josh,

Thanks for your response! :slight_smile: I have already tried changing the admin password and also creating a new user but the issue persists.

rabbitmqctl add_user sensu changeme rabbitmqctl set_user_tags sensu administrator rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"


I don't see any warning or error message when I tail rabbitmq logs:

[root@xxlocalhost rabbitmq]# tail -f rabbit@localhost.log

Statistics database started.

=INFO REPORT==== 11-Jun-2018::06:21:01 ===

Server startup complete; 6 plugins started.

  • rabbitmq_management
  • rabbitmq_management_agent
  • rabbitmq_web_dispatch
  • cowboy
  • amqp_client
  • cowlib

I kept on seeing the credential issue on Sensu Enterprise and Sensu client logs.

Regards,

Kaye

On Mon, Jun 11, 2018 at 12:17 AM, Josh Pasqualetto joshua.pasqualetto@gmail.com wrote:

Hey Kaye,

Try resetting the password to the admin user, maybe the password isn’t what you think it is.

rabbitmqctl change_password admin New-Password

If that doesn’t work tail the rabbitmq logs and look for any issues. And debug just as you would an authentication issue outside of sensu.

Thanks,

Josh

On Jun 10, 2018, at 9:27 AM, Kaye sanpascual.keeshia@gmail.com wrote:

Hi All,

We are currently deploying Sensu into Production. We have RabbitMQ version 3.6.15 and Erlang 20.3.6 in our server -running CentOS.

Configurations are supposed to be correct as the same configurations run Sensu smoothly on our Development Environment - Ubuntu Server. We have looked for different solutions online but there aren’t enough resources available. We have also tried to upgrade the versions of RabbitMQ and Erlang to latest but it did not resolve the issue. Hoping you could provide suggestions. Thanks!

There are no error displayed when we tail the rabbitmq log but sensu-enterprise.log shows an error message related to amqp credentials

{“timestamp”:“2018-06-10T16:39:15.989000+0200”,“level”:“warn”,“message”:“transport connection error”,“reason”:“possible authentication failure. wrong credentials?”,“user”:“admin”}

/etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

"host": "xxx@localhost",

"port": 5671,

"user": "admin",

"password": "admin",

"vhost": "/sensu",

"ssl": {

"cert_chain_file": "/etc/sensu/ssl/cert.pem",

"private_key_file": "/etc/sensu/ssl/key.pem"}

}

}

``

/etc/rabbitmq/rabbitmq.config

[{rabbit, [

 {ssl_listeners, [5671]},

 {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},

                {certfile,"/etc/rabbitmq/ssl/cert.pem"},

                {keyfile,"/etc/rabbitmq/ssl/key.pem"},

                {versions, ['tlsv1.2']},

                {verify,verify_peer},

                {fail_if_no_peer_cert,true}]}

]},

{rabbitmq_management, [{listener, [{port, 15672}]}]}

].

``

6

Hello Josh,

Thanks again for your help!

I can authenticate from the client side ~

curl -i -u sensu:sensu http://localhost:15672/api/whoami

HTTP/1.1 200 OK

server: Cowboy

date: Mon, 11 Jun 2018 07:25:23 GMT

content-length: 39

content-type: application/json

vary: accept, accept-encoding, origin

cache-control: no-cache

{“name”:“sensu”,“tags”:“administrator”}

But the issue from the Sensu logs are the same ~

“{“timestamp”:“2018-06-08T15:10:41.715941+0800”,“level”:“warn”,“message”:“transport connection error”,“reason”:“possible authentication failure. wrong credentials?”,“user”:“sensu”}”

Rabbitmq logs suggest Handshake failure ~

2018-06-11 15:31:03.515 [info] <0.1540.0> TLS server: In state certify at ssl_handshake.erl:1289 generated SERVER ALERT: Fatal - Handshake Failure - {bad_cert,invalid_ext_key_usage}

2018-06-11 15:31:23.475 [info] <0.1566.0> TLS server: In state certify at ssl_handshake.erl:1289 generated SERVER ALERT: Fatal - Handshake Failure - {bad_cert,invalid_ext_key_usage}

2018-06-11 15:31:23.514 [info] <0.1571.0> TLS server: In state certify at ssl_handshake.erl:1289 generated SERVER ALERT: Fatal - Handshake Failure - {bad_cert,invalid_ext_key_usage}

Could it be related to the SSL certificates? I haven’t got any issue with the SSL certificates which I have downloaded from Sensu (sensu_ssl_tool.tar) before so I wonder why this time it’s different.

Thanks,

Kaye

···

On Mon, Jun 11, 2018 at 2:25 PM, Josh Pasqualetto joshua.pasqualetto@gmail.com wrote:

Hey Keeshia,

Can you authenticate to rabbitmq outside of sensu?

E.g. curl -i -u sensu:password http://localhost:15672/api/whoami

I remember rabbitmq having an access log or perhaps you have to up the debug level. At any rate we need to deduce if it’s an auth problem with rabbitmq or a configuration/permissions issue.

Try to auth to rabbitmq outside sensu and go from there.

Thanks,

Josh

On Jun 10, 2018, at 10:30 PM, Keeshia San Pascual sanpascual.keeshia@gmail.com wrote:

Sorry, this was the commands I used:

rabbitmqctl add_user sensu changeme

rabbitmqctl set_user_tags sensu administrator

rabbitmqctl set_permissions -p /sensu sensu “." ".” “.*”

On Mon, Jun 11, 2018 at 12:27 PM, Keeshia San Pascual sanpascual.keeshia@gmail.com wrote:

Hi Josh,

Thanks for your response! :slight_smile: I have already tried changing the admin password and also creating a new user but the issue persists.

rabbitmqctl add_user sensu changeme rabbitmqctl set_user_tags sensu administrator rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"


I don't see any warning or error message when I tail rabbitmq logs:

[root@xxlocalhost rabbitmq]# tail -f rabbit@localhost.log

Statistics database started.

=INFO REPORT==== 11-Jun-2018::06:21:01 ===

Server startup complete; 6 plugins started.

  • rabbitmq_management
  • rabbitmq_management_agent
  • rabbitmq_web_dispatch
  • cowboy
  • amqp_client
  • cowlib

I kept on seeing the credential issue on Sensu Enterprise and Sensu client logs.

Regards,

Kaye

On Mon, Jun 11, 2018 at 12:17 AM, Josh Pasqualetto joshua.pasqualetto@gmail.com wrote:

Hey Kaye,

Try resetting the password to the admin user, maybe the password isn’t what you think it is.

rabbitmqctl change_password admin New-Password

If that doesn’t work tail the rabbitmq logs and look for any issues. And debug just as you would an authentication issue outside of sensu.

Thanks,

Josh

On Jun 10, 2018, at 9:27 AM, Kaye sanpascual.keeshia@gmail.com wrote:

Hi All,

We are currently deploying Sensu into Production. We have RabbitMQ version 3.6.15 and Erlang 20.3.6 in our server -running CentOS.

Configurations are supposed to be correct as the same configurations run Sensu smoothly on our Development Environment - Ubuntu Server. We have looked for different solutions online but there aren’t enough resources available. We have also tried to upgrade the versions of RabbitMQ and Erlang to latest but it did not resolve the issue. Hoping you could provide suggestions. Thanks!

There are no error displayed when we tail the rabbitmq log but sensu-enterprise.log shows an error message related to amqp credentials

{“timestamp”:“2018-06-10T16:39:15.989000+0200”,“level”:“warn”,“message”:“transport connection error”,“reason”:“possible authentication failure. wrong credentials?”,“user”:“admin”}

/etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

"host": "xxx@localhost",

"port": 5671,

"user": "admin",

"password": "admin",

"vhost": "/sensu",

"ssl": {

"cert_chain_file": "/etc/sensu/ssl/cert.pem",

"private_key_file": "/etc/sensu/ssl/key.pem"}

}

}

``

/etc/rabbitmq/rabbitmq.config

[{rabbit, [

 {ssl_listeners, [5671]},

 {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},

                {certfile,"/etc/rabbitmq/ssl/cert.pem"},

                {keyfile,"/etc/rabbitmq/ssl/key.pem"},

                {versions, ['tlsv1.2']},

                {verify,verify_peer},

                {fail_if_no_peer_cert,true}]}

]},

{rabbitmq_management, [{listener, [{port, 15672}]}]}

].

``

7

Yeah it is likely a certificate issue, unfortunately I’m not a big rabbitmq expert but I would double check your cents and see if you can auth using ssl at the same domain name you have configured in your sensu conf.

I’m not exactly sure about what sensu enterprise entails but do you have support through them?

Thanks,

Josh

···

On Mon, Jun 11, 2018 at 2:25 PM, Josh Pasqualetto joshua.pasqualetto@gmail.com wrote:

Hey Keeshia,

Can you authenticate to rabbitmq outside of sensu?

E.g. curl -i -u sensu:password http://localhost:15672/api/whoami

I remember rabbitmq having an access log or perhaps you have to up the debug level. At any rate we need to deduce if it’s an auth problem with rabbitmq or a configuration/permissions issue.

Try to auth to rabbitmq outside sensu and go from there.

Thanks,

Josh

On Jun 10, 2018, at 10:30 PM, Keeshia San Pascual sanpascual.keeshia@gmail.com wrote:

Sorry, this was the commands I used:

rabbitmqctl add_user sensu changeme

rabbitmqctl set_user_tags sensu administrator

rabbitmqctl set_permissions -p /sensu sensu “." ".” “.*”

On Mon, Jun 11, 2018 at 12:27 PM, Keeshia San Pascual sanpascual.keeshia@gmail.com wrote:

Hi Josh,

Thanks for your response! :slight_smile: I have already tried changing the admin password and also creating a new user but the issue persists.

rabbitmqctl add_user sensu changeme rabbitmqctl set_user_tags sensu administrator rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"


I don't see any warning or error message when I tail rabbitmq logs:

[root@xxlocalhost rabbitmq]# tail -f rabbit@localhost.log

Statistics database started.

=INFO REPORT==== 11-Jun-2018::06:21:01 ===

Server startup complete; 6 plugins started.

  • rabbitmq_management
  • rabbitmq_management_agent
  • rabbitmq_web_dispatch
  • cowboy
  • amqp_client
  • cowlib

I kept on seeing the credential issue on Sensu Enterprise and Sensu client logs.

Regards,

Kaye

On Mon, Jun 11, 2018 at 12:17 AM, Josh Pasqualetto joshua.pasqualetto@gmail.com wrote:

Hey Kaye,

Try resetting the password to the admin user, maybe the password isn’t what you think it is.

rabbitmqctl change_password admin New-Password

If that doesn’t work tail the rabbitmq logs and look for any issues. And debug just as you would an authentication issue outside of sensu.

Thanks,

Josh

On Jun 10, 2018, at 9:27 AM, Kaye sanpascual.keeshia@gmail.com wrote:

Hi All,

We are currently deploying Sensu into Production. We have RabbitMQ version 3.6.15 and Erlang 20.3.6 in our server -running CentOS.

Configurations are supposed to be correct as the same configurations run Sensu smoothly on our Development Environment - Ubuntu Server. We have looked for different solutions online but there aren’t enough resources available. We have also tried to upgrade the versions of RabbitMQ and Erlang to latest but it did not resolve the issue. Hoping you could provide suggestions. Thanks!

There are no error displayed when we tail the rabbitmq log but sensu-enterprise.log shows an error message related to amqp credentials

{“timestamp”:“2018-06-10T16:39:15.989000+0200”,“level”:“warn”,“message”:“transport connection error”,“reason”:“possible authentication failure. wrong credentials?”,“user”:“admin”}

/etc/sensu/conf.d/rabbitmq.json

{

“rabbitmq”: {

"host": "xxx@localhost",

"port": 5671,

"user": "admin",

"password": "admin",

"vhost": "/sensu",

"ssl": {

"cert_chain_file": "/etc/sensu/ssl/cert.pem",

"private_key_file": "/etc/sensu/ssl/key.pem"}

}

}

``

/etc/rabbitmq/rabbitmq.config

[{rabbit, [

 {ssl_listeners, [5671]},

 {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},

                {certfile,"/etc/rabbitmq/ssl/cert.pem"},

                {keyfile,"/etc/rabbitmq/ssl/key.pem"},

                {versions, ['tlsv1.2']},

                {verify,verify_peer},

                {fail_if_no_peer_cert,true}]}

]},

{rabbitmq_management, [{listener, [{port, 15672}]}]}

].

``