Hello all,
I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
Hi Quentin,
It’s fairly standard practice to run a RabbitMQ cluster behind HAProxy, providing some broker redundancy, without needing to update connection information in the event of a failure.
There can be only one instance of Redis, but you may have slaves to fail over to.
You can have one or more Sensu servers, as result and event processing is automatically balanced between them.
Sean.
···
On 25 Jun 2013 07:56, “Quenten Griffith” qgriffith@gmail.com wrote:
Hello all,
I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
Hello Sean,
Thanks for the reply. I ended up getting the RabbitMQ cluster working with Sensu. The last trick was to make sure the queues where setup to be mirrored. Before I did that, it would seem that messages would get lost, and the clients would not end up running the checks.
···
On Tuesday, June 25, 2013 11:05:28 AM UTC-4, portertech wrote:
Hi Quentin,
It’s fairly standard practice to run a RabbitMQ cluster behind HAProxy, providing some broker redundancy, without needing to update connection information in the event of a failure.
There can be only one instance of Redis, but you may have slaves to fail over to.
You can have one or more Sensu servers, as result and event processing is automatically balanced between them.
Sean.
On 25 Jun 2013 07:56, “Quenten Griffith” qgri...@gmail.com wrote:
Hello all,
I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
Shouldn’t need to mirror queues, as messages published to an exchange should reach queues/bindings on all rabbit nodes in the cluster. What version of Sensu are you using?
···
On 25 Jun 2013 12:50, “Quenten Griffith” qgriffith@gmail.com wrote:
Hello Sean,
Thanks for the reply. I ended up getting the RabbitMQ cluster working with Sensu. The last trick was to make sure the queues where setup to be mirrored. Before I did that, it would seem that messages would get lost, and the clients would not end up running the checks.
On Tuesday, June 25, 2013 11:05:28 AM UTC-4, portertech wrote:
Hi Quentin,
It’s fairly standard practice to run a RabbitMQ cluster behind HAProxy, providing some broker redundancy, without needing to update connection information in the event of a failure.
There can be only one instance of Redis, but you may have slaves to fail over to.
You can have one or more Sensu servers, as result and event processing is automatically balanced between them.
Sean.
On 25 Jun 2013 07:56, “Quenten Griffith” qgri...@gmail.com wrote:
Hello all,
I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
Quenten, have you tested the cluster? ie: kill one of the rabbit nodes, and do the sensu-clients reconnect on their own?
Perhaps mirrored queues for results + keepalives would be good? If those queues are only on one rabbit node and it goes down, then any pending messages would be lost and you could miss some important results or keepalives … ?
···
On Tue, Jun 25, 2013 at 1:46 PM, Sean Porter portertech@gmail.com wrote:
Shouldn’t need to mirror queues, as messages published to an exchange should reach queues/bindings on all rabbit nodes in the cluster. What version of Sensu are you using?
On 25 Jun 2013 12:50, “Quenten Griffith” qgriffith@gmail.com wrote:
Hello Sean,
Thanks for the reply. I ended up getting the RabbitMQ cluster working with Sensu. The last trick was to make sure the queues where setup to be mirrored. Before I did that, it would seem that messages would get lost, and the clients would not end up running the checks.
On Tuesday, June 25, 2013 11:05:28 AM UTC-4, portertech wrote:
Hi Quentin,
It’s fairly standard practice to run a RabbitMQ cluster behind HAProxy, providing some broker redundancy, without needing to update connection information in the event of a failure.
There can be only one instance of Redis, but you may have slaves to fail over to.
You can have one or more Sensu servers, as result and event processing is automatically balanced between them.
Sean.
On 25 Jun 2013 07:56, “Quenten Griffith” qgri...@gmail.com wrote:
Hello all,
I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
Hello Joe,
I did test the cluster. I caused a check to trigger, then shutdown the primary RabbitMQ server. I made sure the clients connected to the other RabibtMQ server, then caused the correct the issue on the check. I saw that the check was sending the OK status, however the dashboard would never clear the check. Once I mirrored the queues everything worked.
I did make a lot of changes yesterday, adding a password to redis, binding redis on an external IP, setting up a second sensu server, and then clustering RabbitMQ. Today I have un done all that and am going to test each piece to determine if I need to mirror the queues or not. Everything is running as VM’s using vagrant on my local laptop so it is pretty easy to re-configure everything.
···
On Tuesday, June 25, 2013 10:56:27 AM UTC-4, Quenten Griffith wrote:
Hello all,
I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
After all my testing, I have determined if the queues are not mirrored, and the RabbitMQ server that contains the queues goes down, then an active alert is never cleared, even if the check returns an OK status. When I bring up the main RabbitMQ server, the alert will clear. Once I mirrored the queues this issue went away. I can stop any of the RabbitMQ servers and sensu works with out any issues.
···
On Wednesday, June 26, 2013 10:30:26 AM UTC-4, Quenten Griffith wrote:
Hello Joe,
I did test the cluster. I caused a check to trigger, then shutdown the primary RabbitMQ server. I made sure the clients connected to the other RabibtMQ server, then caused the correct the issue on the check. I saw that the check was sending the OK status, however the dashboard would never clear the check. Once I mirrored the queues everything worked.
I did make a lot of changes yesterday, adding a password to redis, binding redis on an external IP, setting up a second sensu server, and then clustering RabbitMQ. Today I have un done all that and am going to test each piece to determine if I need to mirror the queues or not. Everything is running as VM’s using vagrant on my local laptop so it is pretty easy to re-configure everything.
On Tuesday, June 25, 2013 10:56:27 AM UTC-4, Quenten Griffith wrote:
Hello all,
I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?
Is this still a thing? I’d very much like to get rid of my rabbit cluster/mirroring and move to LB’d standalone queues because I’m getting tired of network partitions. Shouldn’t this work fine without mirroring? Especially since I only run standalone checks? I only have 2 queues from what I can see, keepalives and results, and they’re always empty (as I’d expect since state should be handled by Redis). The docs say that these queues must be mirrored but doesn’t say why. But then there’s also older posts like this one wherein Sean says that mirroring isn’t needed. Which is it?
/B
p.s. Am about to test this myself too to see if I get problems like Quenten did.
···
On Wednesday, June 26, 2013 at 12:21:24 PM UTC-7, Quenten Griffith wrote:
After all my testing, I have determined if the queues are not mirrored, and the RabbitMQ server that contains the queues goes down, then an active alert is never cleared, even if the check returns an OK status. When I bring up the main RabbitMQ server, the alert will clear. Once I mirrored the queues this issue went away. I can stop any of the RabbitMQ servers and sensu works with out any issues.
On Wednesday, June 26, 2013 10:30:26 AM UTC-4, Quenten Griffith wrote:
Hello Joe,
I did test the cluster. I caused a check to trigger, then shutdown the primary RabbitMQ server. I made sure the clients connected to the other RabibtMQ server, then caused the correct the issue on the check. I saw that the check was sending the OK status, however the dashboard would never clear the check. Once I mirrored the queues everything worked.
I did make a lot of changes yesterday, adding a password to redis, binding redis on an external IP, setting up a second sensu server, and then clustering RabbitMQ. Today I have un done all that and am going to test each piece to determine if I need to mirror the queues or not. Everything is running as VM’s using vagrant on my local laptop so it is pretty easy to re-configure everything.
On Tuesday, June 25, 2013 10:56:27 AM UTC-4, Quenten Griffith wrote:
Hello all,
I am looking into the various options of setting up Sensu with 2 servers for redundancy. From searching the groups it appears it may be recommend to use RabbitMQ federation. One thing that I am missing the understanding on, is if one RabbitMQ server goes down, do you have to change all the agents to point to the other RabbitMQ server manually? Has anyone used Sensu with a clustered pair of RabbitMQ servers using a load balancer? That way it would avoid having to change a bunch of agent configuration files.
When running more then one Sensu server, are people using a single Reddis database, or installing Reddis on each sensu server?