RabbitMQ behind haproxy - numerous "connection_closed_abruptly" errors

We are trying to put our Sensu server components (RabbitMQ, Server/API, Uchiwa) behind haproxy. Rabbitmq is v3.3.5. All services come up fine but the RabbitMQ logs are quickly filling up with errors like below:

function_clause

Event crashed log handler:

{warning_msg,<0.136.0>,

{<0.186.0>,“closing AMQP connection ~p (~s):~n~p~n”,

[<0.24078.1>,“172.16.72.200:45653 -> 172.16.75.57:5671”,

connection_closed_abruptly]}}

Any idea what the issue may be and what configs need to be tweaked to resolve this?

HAProxy version is 1.6.9

···

Hi Alex,

AMQP, the protocol Sensu uses for RabbitMQ communication, uses a long-lived connection to the broker which doesn’t lend itself well to load balancing via haproxy or similar. While it is possible to tune your haproxy systems to allow long-lived idle connections, this has historically required low-level TCP stack tuning (see RabbitMQ and HAProxy: a timeout issue – Tony's press).

Instead of relying on haproxy to balance long-lived connections, I encourage you to take advantage of Sensu’s support for defining multiple brokers in your RabbitMQ configuration. See https://sensuapp.org/docs/latest/reference/rabbitmq.html#sensu-rabbitmq-configuration-examples-high-availability for an example.

Cameron

···

On Wednesday, November 2, 2016 at 10:48:46 AM UTC-6, Alex Rodrigues wrote:

We are trying to put our Sensu server components (RabbitMQ, Server/API, Uchiwa) behind haproxy. Rabbitmq is v3.3.5. All services come up fine but the RabbitMQ logs are quickly filling up with errors like below:

function_clause

Event crashed log handler:

{warning_msg,<0.136.0>,

{<0.186.0>,“closing AMQP connection ~p (~s):~n~p~n”,

[<0.24078.1>,“172.16.72.200:45653172.16.75.57:5671”,

connection_closed_abruptly]}}

Any idea what the issue may be and what configs need to be tweaked to resolve this?

Thanks Cameron!

···

On Thursday, November 3, 2016 at 4:50:47 PM UTC-4, Cameron Johnston wrote:

Hi Alex,

AMQP, the protocol Sensu uses for RabbitMQ communication, uses a long-lived connection to the broker which doesn’t lend itself well to load balancing via haproxy or similar. While it is possible to tune your haproxy systems to allow long-lived idle connections, this has historically required low-level TCP stack tuning (see https://deviantony.wordpress.com/2014/10/30/rabbitmq-and-haproxy-a-timeout-issue/).

Instead of relying on haproxy to balance long-lived connections, I encourage you to take advantage of Sensu’s support for defining multiple brokers in your RabbitMQ configuration. See https://sensuapp.org/docs/latest/reference/rabbitmq.html#sensu-rabbitmq-configuration-examples-high-availability for an example.

Cameron

On Wednesday, November 2, 2016 at 10:48:46 AM UTC-6, Alex Rodrigues wrote:

We are trying to put our Sensu server components (RabbitMQ, Server/API, Uchiwa) behind haproxy. Rabbitmq is v3.3.5. All services come up fine but the RabbitMQ logs are quickly filling up with errors like below:

function_clause

Event crashed log handler:

{warning_msg,<0.136.0>,

{<0.186.0>,“closing AMQP connection ~p (~s):~n~p~n”,

[<0.24078.1>,“172.16.72.200:45653172.16.75.57:5671”,

connection_closed_abruptly]}}

Any idea what the issue may be and what configs need to be tweaked to resolve this?