# Clustered Sensu with RabbitMQ

**URL:** <https://discourse.sensu.io/t/clustered-sensu-with-rabbitmq/52>\
**Category:** Sensu Classic (EOL)\
**Created:** [June 25, 2013, 2:56pm UTC](https://discourse.sensu.io/t/clustered-sensu-with-rabbitmq/52 "2013-06-25T14:56:27Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Quenten\_Griffith](https://avatars.discourse-cdn.com/v4/letter/q/e47774/32.png) [@Quenten\_Griffith](https://discourse.sensu.io/u/Quenten_Griffith)\
**Post date:** [June 25, 2013, 2:56pm UTC](https://discourse.sensu.io/t/clustered-sensu-with-rabbitmq/52/1 "2013-06-25T14:56:27Z")

</div>

Hello all,

I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.

When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?

---

<div class="post-metadata">

**Author:** ![Sean\_Porter](https://avatars.discourse-cdn.com/v4/letter/s/7feea3/32.png) [@Sean\_Porter](https://discourse.sensu.io/u/Sean_Porter)\
**Post date:** [June 25, 2013, 3:05pm UTC](https://discourse.sensu.io/t/clustered-sensu-with-rabbitmq/52/2 "2013-06-25T15:05:28Z")

</div>

Hi Quentin,

It’s fairly standard practice to run a RabbitMQ cluster behind HAProxy, providing some broker redundancy, without needing to update connection information in the event of a failure.

There can be only one instance of Redis, but you may have slaves to fail over to.

You can have one or more Sensu servers, as result and event processing is automatically balanced between them.

Sean.

> **···**
>
> On 25 Jun 2013 07:56, “Quenten Griffith” [qgriffith@gmail.com](mailto:qgriffith@gmail.com) wrote:
> 
> > Hello all,
> 
> > I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
> 
> > 
> 
> > When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
> 
> > 
> 
> >

---

<div class="post-metadata">

**Author:** ![Quenten\_Griffith](https://avatars.discourse-cdn.com/v4/letter/q/e47774/32.png) [@Quenten\_Griffith](https://discourse.sensu.io/u/Quenten_Griffith)\
**Post date:** [June 25, 2013, 7:50pm UTC](https://discourse.sensu.io/t/clustered-sensu-with-rabbitmq/52/3 "2013-06-25T19:50:05Z")

</div>

Hello Sean,

Thanks for the reply. I ended up getting the RabbitMQ cluster working with Sensu. The last trick was to make sure the queues where setup to be mirrored. Before I did that, it would seem that messages would get lost, and the clients would not end up running the checks.

> **···**
>
> On Tuesday, June 25, 2013 11:05:28 AM UTC-4, portertech wrote:
> 
> > Hi Quentin,
> 
> > It’s fairly standard practice to run a RabbitMQ cluster behind HAProxy, providing some broker redundancy, without needing to update connection information in the event of a failure.
> 
> > There can be only one instance of Redis, but you may have slaves to fail over to.
> 
> > You can have one or more Sensu servers, as result and event processing is automatically balanced between them.
> 
> > Sean.
> 
> > On 25 Jun 2013 07:56, “Quenten Griffith” [qgri...@gmail.com](mailto:qgri...@gmail.com) wrote:
> > 
> > > Hello all,
> 
> > > I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
> 
> > >
> 
> > > When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
> 
> > >
> 
> > >

---

<div class="post-metadata">

**Author:** ![Sean\_Porter](https://avatars.discourse-cdn.com/v4/letter/s/7feea3/32.png) [@Sean\_Porter](https://discourse.sensu.io/u/Sean_Porter)\
**Post date:** [June 25, 2013, 8:46pm UTC](https://discourse.sensu.io/t/clustered-sensu-with-rabbitmq/52/4 "2013-06-25T20:46:43Z")

</div>

Shouldn’t need to mirror queues, as messages published to an exchange should reach queues/bindings on all rabbit nodes in the cluster. What version of Sensu are you using?

> **···**
>
> On 25 Jun 2013 12:50, “Quenten Griffith” [qgriffith@gmail.com](mailto:qgriffith@gmail.com) wrote:
> 
> > Hello Sean,
> 
> > Thanks for the reply. I ended up getting the RabbitMQ cluster working with Sensu. The last trick was to make sure the queues where setup to be mirrored. Before I did that, it would seem that messages would get lost, and the clients would not end up running the checks.
> > 
> > On Tuesday, June 25, 2013 11:05:28 AM UTC-4, portertech wrote:
> > 
> > > Hi Quentin,
> 
> > > It’s fairly standard practice to run a RabbitMQ cluster behind HAProxy, providing some broker redundancy, without needing to update connection information in the event of a failure.
> 
> > > There can be only one instance of Redis, but you may have slaves to fail over to.
> 
> > > You can have one or more Sensu servers, as result and event processing is automatically balanced between them.
> 
> > > Sean.
> 
> > > On 25 Jun 2013 07:56, “Quenten Griffith” [qgri...@gmail.com](mailto:qgri...@gmail.com) wrote:
> > > 
> > > > Hello all,
> 
> > > > I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
> 
> > > >
> 
> > > > When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
> 
> > > >
> 
> > > >

---

<div class="post-metadata">

**Author:** ![Joe\_Miller](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/joe_miller/32/160_2.png) [@Joe\_Miller](https://discourse.sensu.io/u/Joe_Miller)\
**Post date:** [June 26, 2013, 2:19pm UTC](https://discourse.sensu.io/t/clustered-sensu-with-rabbitmq/52/5 "2013-06-26T14:19:06Z")

</div>

Quenten, have you tested the cluster? ie: kill one of the rabbit nodes, and do the sensu-clients reconnect on their own?

Perhaps mirrored queues for results + keepalives would be good? If those queues are only on one rabbit node and it goes down, then any pending messages would be lost and you could miss some important results or keepalives … ?

> **···**
>
> On Tue, Jun 25, 2013 at 1:46 PM, Sean Porter [portertech@gmail.com](mailto:portertech@gmail.com) wrote:
> 
> > Shouldn’t need to mirror queues, as messages published to an exchange should reach queues/bindings on all rabbit nodes in the cluster. What version of Sensu are you using?
> 
> > On 25 Jun 2013 12:50, “Quenten Griffith” [qgriffith@gmail.com](mailto:qgriffith@gmail.com) wrote:
> > 
> > > Hello Sean,
> 
> > > Thanks for the reply. I ended up getting the RabbitMQ cluster working with Sensu. The last trick was to make sure the queues where setup to be mirrored. Before I did that, it would seem that messages would get lost, and the clients would not end up running the checks.
> > > 
> > > On Tuesday, June 25, 2013 11:05:28 AM UTC-4, portertech wrote:
> > > 
> > > > Hi Quentin,
> 
> > > > It’s fairly standard practice to run a RabbitMQ cluster behind HAProxy, providing some broker redundancy, without needing to update connection information in the event of a failure.
> 
> > > > There can be only one instance of Redis, but you may have slaves to fail over to.
> 
> > > > You can have one or more Sensu servers, as result and event processing is automatically balanced between them.
> 
> > > > Sean.
> 
> > > > On 25 Jun 2013 07:56, “Quenten Griffith” [qgri...@gmail.com](mailto:qgri...@gmail.com) wrote:
> > > > 
> > > > > Hello all,
> 
> > > > > I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
> 
> > > > >
> 
> > > > > When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
> 
> > > > >
> 
> > > > >

---

<div class="post-metadata">

**Author:** ![Quenten\_Griffith](https://avatars.discourse-cdn.com/v4/letter/q/e47774/32.png) [@Quenten\_Griffith](https://discourse.sensu.io/u/Quenten_Griffith)\
**Post date:** [June 26, 2013, 2:30pm UTC](https://discourse.sensu.io/t/clustered-sensu-with-rabbitmq/52/6 "2013-06-26T14:30:26Z")

</div>

Hello Joe,

I did test the cluster. I caused a check to trigger, then shutdown the primary RabbitMQ server. I made sure the clients connected to the other RabibtMQ server, then caused the correct the issue on the check. I saw that the check was sending the OK status, however the dashboard would never clear the check. Once I mirrored the queues everything worked.

I did make a lot of changes yesterday, adding a password to redis, binding redis on an external IP, setting up a second sensu server, and then clustering RabbitMQ. Today I have un done all that and am going to test each piece to determine if I need to mirror the queues or not. Everything is running as VM’s using vagrant on my local laptop so it is pretty easy to re-configure everything.

> **···**
>
> On Tuesday, June 25, 2013 10:56:27 AM UTC-4, Quenten Griffith wrote:
> 
> > Hello all,
> 
> > I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
> 
> > 
> 
> > When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
> 
> > 
> 
> >

---

<div class="post-metadata">

**Author:** ![Quenten\_Griffith](https://avatars.discourse-cdn.com/v4/letter/q/e47774/32.png) [@Quenten\_Griffith](https://discourse.sensu.io/u/Quenten_Griffith)\
**Post date:** [June 26, 2013, 7:21pm UTC](https://discourse.sensu.io/t/clustered-sensu-with-rabbitmq/52/7 "2013-06-26T19:21:24Z")

</div>

After all my testing, I have determined if the queues are not mirrored, and the RabbitMQ server that contains the queues goes down, then an active alert is never cleared, even if the check returns an OK status. When I bring up the main RabbitMQ server, the alert will clear. Once I mirrored the queues this issue went away. I can stop any of the RabbitMQ servers and sensu works with out any issues.

> **···**
>
> On Wednesday, June 26, 2013 10:30:26 AM UTC-4, Quenten Griffith wrote:
> 
> > Hello Joe,
> 
> > I did test the cluster. I caused a check to trigger, then shutdown the primary RabbitMQ server. I made sure the clients connected to the other RabibtMQ server, then caused the correct the issue on the check. I saw that the check was sending the OK status, however the dashboard would never clear the check. Once I mirrored the queues everything worked.
> 
> > 
> 
> > I did make a lot of changes yesterday, adding a password to redis, binding redis on an external IP, setting up a second sensu server, and then clustering RabbitMQ. Today I have un done all that and am going to test each piece to determine if I need to mirror the queues or not. Everything is running as VM’s using vagrant on my local laptop so it is pretty easy to re-configure everything.
> > 
> > On Tuesday, June 25, 2013 10:56:27 AM UTC-4, Quenten Griffith wrote:
> > 
> > > Hello all,
> 
> > > I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
> 
> > >
> 
> > > When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
> 
> > >
> 
> > >

---

<div class="post-metadata">

**Author:** ![Brandon](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@Brandon](https://discourse.sensu.io/u/Brandon)\
**Post date:** [April 2, 2016, 12:26am UTC](https://discourse.sensu.io/t/clustered-sensu-with-rabbitmq/52/8 "2016-04-02T00:26:09Z")

</div>

Is this still a thing? I’d very much like to get rid of my rabbit cluster/mirroring and move to LB’d standalone queues because I’m getting tired of network partitions. Shouldn’t this work fine without mirroring? Especially since I only run standalone checks? I only have 2 queues from what I can see, keepalives and results, and they’re always empty (as I’d expect since state should be handled by Redis). The docs say that these queues must be mirrored but doesn’t say why. But then there’s also older posts like this one wherein Sean says that mirroring isn’t needed. Which is it?

/B

p.s. Am about to test this myself too to see if I get problems like Quenten did.

> **···**
>
> On Wednesday, June 26, 2013 at 12:21:24 PM UTC-7, Quenten Griffith wrote:
> 
> > After all my testing, I have determined if the queues are not mirrored, and the RabbitMQ server that contains the queues goes down, then an active alert is never cleared, even if the check returns an OK status. When I bring up the main RabbitMQ server, the alert will clear. Once I mirrored the queues this issue went away. I can stop any of the RabbitMQ servers and sensu works with out any issues.
> 
> > On Wednesday, June 26, 2013 10:30:26 AM UTC-4, Quenten Griffith wrote:
> > 
> > > Hello Joe,
> 
> > > I did test the cluster. I caused a check to trigger, then shutdown the primary RabbitMQ server. I made sure the clients connected to the other RabibtMQ server, then caused the correct the issue on the check. I saw that the check was sending the OK status, however the dashboard would never clear the check. Once I mirrored the queues everything worked.
> 
> > >
> 
> > > I did make a lot of changes yesterday, adding a password to redis, binding redis on an external IP, setting up a second sensu server, and then clustering RabbitMQ. Today I have un done all that and am going to test each piece to determine if I need to mirror the queues or not. Everything is running as VM’s using vagrant on my local laptop so it is pretty easy to re-configure everything.
> > > 
> > > On Tuesday, June 25, 2013 10:56:27 AM UTC-4, Quenten Griffith wrote:
> > > 
> > > > Hello all,
> 
> > > > I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
> 
> > > >
> 
> > > > When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
> 
> > > >
> 
> > > >

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [November 22, 2018, 2:04am UTC](https://discourse.sensu.io/t/clustered-sensu-with-rabbitmq/52/9 "2018-11-22T02:04:56Z")

</div>