RabbitMQ behind haproxy - numerous "connection_closed_abruptly" errors


#1

We are trying to put our Sensu server components (RabbitMQ, Server/API, Uchiwa) behind haproxy. Rabbitmq is v3.3.5. All services come up fine but the RabbitMQ logs are quickly filling up with errors like below:

function_clause

Event crashed log handler:

{warning_msg,<0.136.0>,

{<0.186.0>,“closing AMQP connection ~p (~s):~n~p~n”,

[<0.24078.1>,“172.16.72.200:45653 -> 172.16.75.57:5671”,

connection_closed_abruptly]}}

Any idea what the issue may be and what configs need to be tweaked to resolve this?


#2

HAProxy version is 1.6.9

···

#3

Hi Alex,

AMQP, the protocol Sensu uses for RabbitMQ communication, uses a long-lived connection to the broker which doesn’t lend itself well to load balancing via haproxy or similar. While it is possible to tune your haproxy systems to allow long-lived idle connections, this has historically required low-level TCP stack tuning (see https://deviantony.wordpress.com/2014/10/30/rabbitmq-and-haproxy-a-timeout-issue/).

Instead of relying on haproxy to balance long-lived connections, I encourage you to take advantage of Sensu’s support for defining multiple brokers in your RabbitMQ configuration. See https://sensuapp.org/docs/latest/reference/rabbitmq.html#sensu-rabbitmq-configuration-examples-high-availability for an example.

Cameron

···

On Wednesday, November 2, 2016 at 10:48:46 AM UTC-6, Alex Rodrigues wrote:

We are trying to put our Sensu server components (RabbitMQ, Server/API, Uchiwa) behind haproxy. Rabbitmq is v3.3.5. All services come up fine but the RabbitMQ logs are quickly filling up with errors like below:

function_clause

Event crashed log handler:

{warning_msg,<0.136.0>,

{<0.186.0>,“closing AMQP connection ~p (~s):~n~p~n”,

[<0.24078.1>,“172.16.72.200:45653 -> 172.16.75.57:5671”,

connection_closed_abruptly]}}

Any idea what the issue may be and what configs need to be tweaked to resolve this?


#4

Thanks Cameron!

···

On Thursday, November 3, 2016 at 4:50:47 PM UTC-4, Cameron Johnston wrote:

Hi Alex,

AMQP, the protocol Sensu uses for RabbitMQ communication, uses a long-lived connection to the broker which doesn’t lend itself well to load balancing via haproxy or similar. While it is possible to tune your haproxy systems to allow long-lived idle connections, this has historically required low-level TCP stack tuning (see https://deviantony.wordpress.com/2014/10/30/rabbitmq-and-haproxy-a-timeout-issue/).

Instead of relying on haproxy to balance long-lived connections, I encourage you to take advantage of Sensu’s support for defining multiple brokers in your RabbitMQ configuration. See https://sensuapp.org/docs/latest/reference/rabbitmq.html#sensu-rabbitmq-configuration-examples-high-availability for an example.

Cameron

On Wednesday, November 2, 2016 at 10:48:46 AM UTC-6, Alex Rodrigues wrote:

We are trying to put our Sensu server components (RabbitMQ, Server/API, Uchiwa) behind haproxy. Rabbitmq is v3.3.5. All services come up fine but the RabbitMQ logs are quickly filling up with errors like below:

function_clause

Event crashed log handler:

{warning_msg,<0.136.0>,

{<0.186.0>,“closing AMQP connection ~p (~s):~n~p~n”,

[<0.24078.1>,“172.16.72.200:45653 -> 172.16.75.57:5671”,

connection_closed_abruptly]}}

Any idea what the issue may be and what configs need to be tweaked to resolve this?