sensu-server shutting down, "no exchange in vhost"

Sensu Classic (EOL)
#1

I’ve been battling RabbitMQ for the past couple of days to get it to work in a cluster with firewalled hosts. The cluster appears to be up and running okay.

But now my sensu-server is shutting itself down with a “rabbitmq connection error”

In the rabbitmq log I get these:

=ERROR REPORT==== 20-Mar-2014::13:03:00 ===
connection <0.1577.0>, channel 1 - soft error:
{amqp_error,not_found,“no exchange ‘metrics’ in vhost ‘sensu’”,

        'exchange.declare'}

=ERROR REPORT==== 20-Mar-2014::13:03:00 ===
connection <0.1531.0>, channel 1 - soft error:
{amqp_error,not_found,“no queue ‘keepalives’ in vhost ‘sensu’”,

        'queue.declare'}

=ERROR REPORT==== 20-Mar-2014::13:03:01 ===
AMQP connection <0.1577.0> (running), channel 1 - error:
{amqp_error,channel_error,“expected ‘channel.open’”,‘basic.recover’}

=INFO REPORT==== 20-Mar-2014::13:03:09 ===
closing AMQP connection <0.1577.0> (127.0.0.1:35420 -> 127.0.0.1:5672)

I’m not sure where to start.

Mojo

#2

I’ve tried adding a “rabbitmq_exchange” resource in puppet following the user and vhost creation and with permissions set, but I can’t create the exchange:

bash-4.1# /usr/local/bin/rabbitmqadmin declare exchange --vhost=sensu --user=something --password=something name=metrics type=topic

*** Access refused: /api/exchanges/sensu/metrics

The values are all the same as set for sensu.

This all used to work, then we put up a firewall that broke everything, and I’m back to ground zero. :frowning:

Mojo

···

On Thu, Mar 20, 2014 at 10:07 AM, Mojo mojo.la@gmail.com wrote:

I’ve been battling RabbitMQ for the past couple of days to get it to work in a cluster with firewalled hosts. The cluster appears to be up and running okay.

But now my sensu-server is shutting itself down with a “rabbitmq connection error”

In the rabbitmq log I get these:

=ERROR REPORT==== 20-Mar-2014::13:03:00 ===
connection <0.1577.0>, channel 1 - soft error:
{amqp_error,not_found,“no exchange ‘metrics’ in vhost ‘sensu’”,

        'exchange.declare'}

=ERROR REPORT==== 20-Mar-2014::13:03:00 ===
connection <0.1531.0>, channel 1 - soft error:
{amqp_error,not_found,“no queue ‘keepalives’ in vhost ‘sensu’”,

        'queue.declare'}

=ERROR REPORT==== 20-Mar-2014::13:03:01 ===
AMQP connection <0.1577.0> (running), channel 1 - error:
{amqp_error,channel_error,“expected ‘channel.open’”,‘basic.recover’}

=INFO REPORT==== 20-Mar-2014::13:03:09 ===
closing AMQP connection <0.1577.0> (127.0.0.1:35420 -> 127.0.0.1:5672)

I’m not sure where to start.

Mojo

#3

A followup to this. I have it working now, and a theory as to why it was failing.

I’m using AMQP to echo metrics checks to graphite, on the “metrics” exchange. Sensu is configured “exchange”: { “passive” : true }.

What I tried that worked was to shut down carbon and graphite, reset RabbitMQ, startup RabbitMQ and Sensu, then start up carbon and graphite. Everything came up smoothly.

Not sure what the best way to ensure that happening would be.

Mojo

···

On Thu, Mar 20, 2014 at 12:14 PM, Mojo mojo.la@gmail.com wrote:

I’ve tried adding a “rabbitmq_exchange” resource in puppet following the user and vhost creation and with permissions set, but I can’t create the exchange:

bash-4.1# /usr/local/bin/rabbitmqadmin declare exchange --vhost=sensu --user=something --password=something name=metrics type=topic

*** Access refused: /api/exchanges/sensu/metrics

The values are all the same as set for sensu.

This all used to work, then we put up a firewall that broke everything, and I’m back to ground zero. :frowning:

Mojo

On Thu, Mar 20, 2014 at 10:07 AM, Mojo mojo.la@gmail.com wrote:

I’ve been battling RabbitMQ for the past couple of days to get it to work in a cluster with firewalled hosts. The cluster appears to be up and running okay.

But now my sensu-server is shutting itself down with a “rabbitmq connection error”

In the rabbitmq log I get these:

=ERROR REPORT==== 20-Mar-2014::13:03:00 ===
connection <0.1577.0>, channel 1 - soft error:
{amqp_error,not_found,“no exchange ‘metrics’ in vhost ‘sensu’”,

        'exchange.declare'}

=ERROR REPORT==== 20-Mar-2014::13:03:00 ===
connection <0.1531.0>, channel 1 - soft error:
{amqp_error,not_found,“no queue ‘keepalives’ in vhost ‘sensu’”,

        'queue.declare'}

=ERROR REPORT==== 20-Mar-2014::13:03:01 ===
AMQP connection <0.1577.0> (running), channel 1 - error:
{amqp_error,channel_error,“expected ‘channel.open’”,‘basic.recover’}

=INFO REPORT==== 20-Mar-2014::13:03:09 ===
closing AMQP connection <0.1577.0> (127.0.0.1:35420 -> 127.0.0.1:5672)

I’m not sure where to start.

Mojo

#4