I have a few servers, running a load of VMs, mainly in 2 datacenters, with a few other VMs in other places.
I’m thinking of setting up a sensu server in each of those 2 DCs, with a 3rd offsite.
I’m thinking they will each be completely separate and standalone - with their own rabbitmq, redis etc.
I should be able to use a single Uchiwa instance to connect to all 3, right?
I’m thinking that all of the machines/VMs will have their sensu_client configured to point to the sensu server in that same DC, the sensu server will have checks to make sure the servers in the same DC respond to pings, the right ports/services respond correctly etc.
The offsite instance will then ping both of the other sensu servers and alert if they go down - and one or both of those instances can ping the offisite one.
Does that sound sane?
I’m thinking the gap here is if something happens to sensu but not the VM it’s running on. Is it possible to have the sensu client on the sensu server connect to 2x sensu servers - both the localhost one and another instance? - it would good to have those connect to localhost (as I use that for outbound checks) and to the offsite sensu server for keepalive and process checks for sensu.
Anyone else do anything similar to this?