Hello again Sensu users,
I have another check design question for you. We have checks for certain nodes which need to access the node’s public IP. For example, I have a check whose command is: /usr/bin/sudo /etc/sensu/plugins/check-memcached.pl -H 10.255.255.10 -p 11211
This check runs on multiple nodes.
I see two methods for this:
- Set up the check on every client node, using that node’s public IP address to run the check (we can’t use localhost, because the daemon doesn’t run on localhost). The monitoring server will get a check definition with a placeholder IP address (this doesn’t seem to be very clean).
- Set up all the checks on the monitoring server as standalone checks. These will hit the public IP of each memcached server. A few issues:
We’ll have to define a dozen or so checks, depending on how many memcached servers are up at any one time
Old checks won’t be cleaned up when a node is deleted (the json file will still exist and Chef will never clean it up - just create new ones as new nodes come up)
When we decommission a server, we’ll have to manually remove the check from the sensu monitoring server.
It’s unclean from an organizational and ease-of-use standpoint (most of a node’s checks are on the node, but some are on the monitoring server??)
Of these options, it seems that it 1) is the preferred way of doing this, but is there a reason that check definitions are the same on the server and the client? Does the server even have to know what a check’s command is? All it really is responsible for is ensuring checks are run on clients at regular intervals. I suppose it’s possible that clients could get out of synch and both be running different versions of the same check and the “correct” one would be the one defined on the server, but I think this may be unnecessary.
Any advice is appreciated.