I have ~300 fronend servers and it make sense to check HTTP port reachability from remote location , as I have also firewalls and network hardware in front of those frontends and I want to be sure that Users can get to my frontend.
I know that I can define for each server separate check with command definition like “check_http serverX” , but one of the reason that I chose sesu is possibility to automate installation and configuration , so that I could just spun up new servers without considering to edit/adjust my monitoring configuration.
Is it possible to define check which will run from single source , but for all servers included in specific subscription list. Something like :
The state of the art hasn’t changed since that other mailing list post.
subscribers is designed to trigger checks on a set of hosts. It isn’t designed to be used as a kind of service discovery for checks on a different host.
Here are a couple options that I’ve done in the past to simulate this:
Make a “reflector” and still schedule checks on the hosts using the normal sensu subscriber mechanism. (I did this with ipmi interfaces, where I wanted to make sure the ipmi interface could be reached, but the host itself couldn’t reach it) I used xinetd and a bash script. I don’t have an open source example of this.
Use “normal” discovery and make a script that iterates through your servers and emits events on localhost:3030 in their name. (using the “source” attribute). At yelp we call this the “fleet check” https://github.com/Yelp/puppet-monitoring_check/blob/master/files/fleet_check.rb where the code runs centrally, yet checks something about a whole class of servers, and emits something using their hostname. (An example of this is checking for aws maintenance events)
Use configuration management to deploy a check on the single source for each server. This is “easy” if you have a good configuration management setup, and assumes that “the list” of servers you want to check is obtainable programmatically. In puppet at Yelp we like to use the new puppet 4 for-loops. I know you said you “don’t” want to change the monitoring configuration, but I think using configuration management is a solid way to take the pain out of updating a monitoring configuration tool.
I’m sure there are even more creative ways, but those are 3 ways I’ve done this before.
I have ~300 fronend servers and it make sense to check HTTP port reachability from remote location , as I have also firewalls and network hardware in front of those frontends and I want to be sure that Users can get to my frontend.
I know that I can define for each server separate check with command definition like “check_http serverX” , but one of the reason that I chose sesu is possibility to automate installation and configuration , so that I could just spun up new servers without considering to edit/adjust my monitoring configuration.
Is it possible to define check which will run from single source , but for all servers included in specific subscription list. Something like :
On Thu, Apr 6, 2017 at 7:56 AM, Kyle Anderson kyle@xkyle.com wrote:
The state of the art hasn’t changed since that other mailing list post.
subscribers is designed to trigger checks on a set of hosts. It isn’t designed to be used as a kind of service discovery for checks on a different host.
Here are a couple options that I’ve done in the past to simulate this:
Make a “reflector” and still schedule checks on the hosts using the normal sensu subscriber mechanism. (I did this with ipmi interfaces, where I wanted to make sure the ipmi interface could be reached, but the host itself couldn’t reach it) I used xinetd and a bash script. I don’t have an open source example of this.
Use “normal” discovery and make a script that iterates through your servers and emits events on localhost:3030 in their name. (using the “source” attribute). At yelp we call this the “fleet check” https://github.com/Yelp/puppet-monitoring_check/blob/master/files/fleet_check.rb where the code runs centrally, yet checks something about a whole class of servers, and emits something using their hostname. (An example of this is checking for aws maintenance events)
Use configuration management to deploy a check on the single source for each server. This is “easy” if you have a good configuration management setup, and assumes that “the list” of servers you want to check is obtainable programmatically. In puppet at Yelp we like to use the new puppet 4 for-loops. I know you said you “don’t” want to change the monitoring configuration, but I think using configuration management is a solid way to take the pain out of updating a monitoring configuration tool.
I’m sure there are even more creative ways, but those are 3 ways I’ve done this before.
I have ~300 fronend servers and it make sense to check HTTP port reachability from remote location , as I have also firewalls and network hardware in front of those frontends and I want to be sure that Users can get to my frontend.
I know that I can define for each server separate check with command definition like “check_http serverX” , but one of the reason that I chose sesu is possibility to automate installation and configuration , so that I could just spun up new servers without considering to edit/adjust my monitoring configuration.
Is it possible to define check which will run from single source , but for all servers included in specific subscription list. Something like :
Vahagn, I posted yesterday with an example for doing something similar using SSH checks for all hosts which match a certain set of attributes which are in the client. Likely what you’ll want to do is have some defining attribute for your frontend servers, such as having them be subscribed to a subscription called frontend, and use that in the client_attributes as the documentation says. You’ll then want to configure the subscribers for the check to be a roundrobin subscription for some set of servers to check http. The roundrobin is important, because the proxy request will generate a check request per client. If the check gets sent to a non roundrobin subscription, then it will be run on each subscriber of that subscription. For example, if you have 5 servers checking the 300 frontend instances, that would be 1500 checks, with each frontend getting an http request 5 times. Roundrobin ensures that just 1 of the 5 servers checking http will execute the check.
I’m not sure if the firewall rules you mentioned will require you to send the check requests to different hosts. As far as I know, there’s not a way to have one check request to selectively send jobs to different roundrobin subscriptions. For example, if you had 2 availability zones, and wanted to know that http clients in AZ1 could be accessed from hosts in AZ2, then you’d need to make one check to find hosts in AZ1, and push to a subscription for checkers in AZ2. You’d then need another check request for the reverse, and swap the client attributes and the subscribers. It would be nice if it could use some client attribute to send the request to different subscriptions.
Hope that helps
···
On Thursday, April 6, 2017 at 10:58:18 AM UTC-4, Kyle Anderson wrote:
On Thu, Apr 6, 2017 at 7:56 AM, Kyle Anderson ky...@xkyle.com wrote:
The state of the art hasn’t changed since that other mailing list post.
subscribers is designed to trigger checks on a set of hosts. It isn’t designed to be used as a kind of service discovery for checks on a different host.
Here are a couple options that I’ve done in the past to simulate this:
Make a “reflector” and still schedule checks on the hosts using the normal sensu subscriber mechanism. (I did this with ipmi interfaces, where I wanted to make sure the ipmi interface could be reached, but the host itself couldn’t reach it) I used xinetd and a bash script. I don’t have an open source example of this.
Use “normal” discovery and make a script that iterates through your servers and emits events on localhost:3030 in their name. (using the “source” attribute). At yelp we call this the “fleet check” https://github.com/Yelp/puppet-monitoring_check/blob/master/files/fleet_check.rb where the code runs centrally, yet checks something about a whole class of servers, and emits something using their hostname. (An example of this is checking for aws maintenance events)
Use configuration management to deploy a check on the single source for each server. This is “easy” if you have a good configuration management setup, and assumes that “the list” of servers you want to check is obtainable programmatically. In puppet at Yelp we like to use the new puppet 4 for-loops. I know you said you “don’t” want to change the monitoring configuration, but I think using configuration management is a solid way to take the pain out of updating a monitoring configuration tool.
I’m sure there are even more creative ways, but those are 3 ways I’ve done this before.
I have ~300 fronend servers and it make sense to check HTTP port reachability from remote location , as I have also firewalls and network hardware in front of those frontends and I want to be sure that Users can get to my frontend.
I know that I can define for each server separate check with command definition like “check_http serverX” , but one of the reason that I chose sesu is possibility to automate installation and configuration , so that I could just spun up new servers without considering to edit/adjust my monitoring configuration.
Is it possible to define check which will run from single source , but for all servers included in specific subscription list. Something like :