Multiple Sensu Servers

Sensu Classic (EOL)
1

Hi,

I am looking to improve on our single Sensu server installation. First port of call is resiliency, and most immediate answer there is - multiple servers.

However I am struggling to find information on this topic. I’ve read the FAQs and the docs and various blog posts, I understand that there is some sort of covert master/slave election between servers, but don’t see where that would be configured : how do the servers know about eachother?

Also, is it mandatory to separate the RabbitMQ instances into a separate tier, or can each server run RabbitMQ locally? Similarly, can they use a local Redis instance or does one need to configure a shared Redis backend (or Elasticache, whatever)?

Any assistance would be much appreciated, I’m a big fan of Sensu and just want to take our use of it to the next level.

Thanks,

Louis McCormack

2

I found this blog post informative, illustrating one way to setup
Sensu clusters:
http://failshell.io/sensu/high-availability-sensu/

For the sensu-server part, yes, leader election is automatic and does
not need to be configured. As long as they are using the same redis
master, it will work:
https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L692

RabbitMQ you can do whatever works best for your environment. To KISS
I put RabbitMQ on each sensu server and cluster them in mirror mode
per the failshell article. In my particular setup, it is not required
that a sensu-server us the *local* RMQ instance, but it can. In
practice it doesn't matter as long as they are mirrored:
http://www.rabbitmq.com/ha.html

The Redis part is a little tricky, but the failshell article also
covers how he handles it. However you do it, all sensu servers / api /
dashboards need to talk to *a* redis master. If you use something
fancy to move that around, that is cool.
Again, I'm using the failshell approach with haproxy. Maybe someday
Sensu will be Sentinel-aware?
Not to spread FUD, but if you are going to setup a Redis cluster, you
should be at least aware of the potential failure modes during splits:
http://aphyr.com/posts/283-call-me-maybe-redis

Welcome to the world of distributed systems! :slight_smile:

···

On Sat, Feb 8, 2014 at 1:42 AM, <louis@spaceapegames.com> wrote:

Hi,

I am looking to improve on our single Sensu server installation. First port
of call is resiliency, and most immediate answer there is - multiple
servers.

However I am struggling to find information on this topic. I've read the
FAQs and the docs and various blog posts, I understand that there is some
sort of covert master/slave election between servers, but don't see where
that would be configured : how do the servers know about eachother?

Also, is it mandatory to separate the RabbitMQ instances into a separate
tier, or can each server run RabbitMQ locally? Similarly, can they use a
local Redis instance or does one need to configure a shared Redis backend
(or Elasticache, whatever)?

Any assistance would be much appreciated, I'm a big fan of Sensu and just
want to take our use of it to the next level.

Thanks,
Louis McCormack

3

Excellent, thank you Kyle. You’re a gentleman and a scholar.

I had seen that article, was kind of hoping there’d be a quick and dirty alternative. :slight_smile:

Thanks again.

Louis

···

On Saturday, 8 February 2014 17:05:12 UTC, Kyle Anderson wrote:

I found this blog post informative, illustrating one way to setup

Sensu clusters:

http://failshell.io/sensu/high-availability-sensu/

For the sensu-server part, yes, leader election is automatic and does

not need to be configured. As long as they are using the same redis

master, it will work:

https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L692

RabbitMQ you can do whatever works best for your environment. To KISS

I put RabbitMQ on each sensu server and cluster them in mirror mode

per the failshell article. In my particular setup, it is not required

that a sensu-server us the local RMQ instance, but it can. In

practice it doesn’t matter as long as they are mirrored:

http://www.rabbitmq.com/ha.html

The Redis part is a little tricky, but the failshell article also

covers how he handles it. However you do it, all sensu servers / api /

dashboards need to talk to a redis master. If you use something

fancy to move that around, that is cool.

Again, I’m using the failshell approach with haproxy. Maybe someday

Sensu will be Sentinel-aware?

Not to spread FUD, but if you are going to setup a Redis cluster, you

should be at least aware of the potential failure modes during splits:

http://aphyr.com/posts/283-call-me-maybe-redis

Welcome to the world of distributed systems! :slight_smile:

On Sat, Feb 8, 2014 at 1:42 AM, lo...@spaceapegames.com wrote:

Hi,

I am looking to improve on our single Sensu server installation. First port

of call is resiliency, and most immediate answer there is - multiple

servers.

However I am struggling to find information on this topic. I’ve read the

FAQs and the docs and various blog posts, I understand that there is some

sort of covert master/slave election between servers, but don’t see where

that would be configured : how do the servers know about eachother?

Also, is it mandatory to separate the RabbitMQ instances into a separate

tier, or can each server run RabbitMQ locally? Similarly, can they use a

local Redis instance or does one need to configure a shared Redis backend

(or Elasticache, whatever)?

Any assistance would be much appreciated, I’m a big fan of Sensu and just

want to take our use of it to the next level.

Thanks,

Louis McCormack

4

HI Kyle,

I was seaching for High Availibilty Sensu and found your post.

When I went to http://failshell.io/sensu/high-availability-sensu/ is not going to correct page.Could you please let me know it there is any alternative for this.

Thanks in advance

···

On Saturday, February 8, 2014 at 10:35:12 PM UTC+5:30, Kyle Anderson wrote:

I found this blog post informative, illustrating one way to setup
Sensu clusters:
http://failshell.io/sensu/high-availability-sensu/

For the sensu-server part, yes, leader election is automatic and does
not need to be configured. As long as they are using the same redis
master, it will work:
https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L692

RabbitMQ you can do whatever works best for your environment. To KISS
I put RabbitMQ on each sensu server and cluster them in mirror mode
per the failshell article. In my particular setup, it is not required
that a sensu-server us the local RMQ instance, but it can. In
practice it doesn’t matter as long as they are mirrored:
http://www.rabbitmq.com/ha.html

The Redis part is a little tricky, but the failshell article also
covers how he handles it. However you do it, all sensu servers / api /
dashboards need to talk to a redis master. If you use something
fancy to move that around, that is cool.
Again, I’m using the failshell approach with haproxy. Maybe someday
Sensu will be Sentinel-aware?
Not to spread FUD, but if you are going to setup a Redis cluster, you
should be at least aware of the potential failure modes during splits:
http://aphyr.com/posts/283-call-me-maybe-redis

Welcome to the world of distributed systems! :slight_smile:

On Sat, Feb 8, 2014 at 1:42 AM, lo...@spaceapegames.com wrote:

Hi,

I am looking to improve on our single Sensu server installation. First port
of call is resiliency, and most immediate answer there is - multiple
servers.

However I am struggling to find information on this topic. I’ve read the
FAQs and the docs and various blog posts, I understand that there is some
sort of covert master/slave election between servers, but don’t see where
that would be configured : how do the servers know about eachother?

Also, is it mandatory to separate the RabbitMQ instances into a separate
tier, or can each server run RabbitMQ locally? Similarly, can they use a
local Redis instance or does one need to configure a shared Redis backend
(or Elasticache, whatever)?

Any assistance would be much appreciated, I’m a big fan of Sensu and just
want to take our use of it to the next level.

Thanks,
Louis McCormack

5

I recommend using a service like archive.org to view pages that no longer exist:

···

On Tue, Dec 15, 2015 at 9:45 PM, kumar kittu <dvrkittu@gmail.com> wrote:

HI Kyle,

I was seaching for High Availibilty Sensu and found your post.
When I went to http://failshell.io/sensu/high-availability-sensu/ is not
going to correct page.Could you please let me know it there is any
alternative for this.
Thanks in advance

On Saturday, February 8, 2014 at 10:35:12 PM UTC+5:30, Kyle Anderson wrote:

I found this blog post informative, illustrating one way to setup
Sensu clusters:
http://failshell.io/sensu/high-availability-sensu/

For the sensu-server part, yes, leader election is automatic and does
not need to be configured. As long as they are using the same redis
master, it will work:
https://github.com/sensu/sensu/blob/master/lib/sensu/server.rb#L692

RabbitMQ you can do whatever works best for your environment. To KISS
I put RabbitMQ on each sensu server and cluster them in mirror mode
per the failshell article. In my particular setup, it is not required
that a sensu-server us the *local* RMQ instance, but it can. In
practice it doesn't matter as long as they are mirrored:
Classic Queue Mirroring — RabbitMQ

The Redis part is a little tricky, but the failshell article also
covers how he handles it. However you do it, all sensu servers / api /
dashboards need to talk to *a* redis master. If you use something
fancy to move that around, that is cool.
Again, I'm using the failshell approach with haproxy. Maybe someday
Sensu will be Sentinel-aware?
Not to spread FUD, but if you are going to setup a Redis cluster, you
should be at least aware of the potential failure modes during splits:
Jepsen: Redis

Welcome to the world of distributed systems! :slight_smile:

On Sat, Feb 8, 2014 at 1:42 AM, <lo...@spaceapegames.com> wrote:
> Hi,
>
> I am looking to improve on our single Sensu server installation. First
> port
> of call is resiliency, and most immediate answer there is - multiple
> servers.
>
> However I am struggling to find information on this topic. I've read the
> FAQs and the docs and various blog posts, I understand that there is
> some
> sort of covert master/slave election between servers, but don't see
> where
> that would be configured : how do the servers know about eachother?
>
> Also, is it mandatory to separate the RabbitMQ instances into a separate
> tier, or can each server run RabbitMQ locally? Similarly, can they use a
> local Redis instance or does one need to configure a shared Redis
> backend
> (or Elasticache, whatever)?
>
> Any assistance would be much appreciated, I'm a big fan of Sensu and
> just
> want to take our use of it to the next level.
>
> Thanks,
> Louis McCormack