We have a bunch of monitoring checks that check things external to the machine: query this metrics database to see if this value is under a threshold, curl this server to see if it’s accessible, etc. Currently, we’re handling these as round-robin checks on a set of machines dedicated to running all of these.
One of the problems we have is that the checks often create dependency hell between them. One check will require a certain version of a gem, another will require a different version, and we have to spend a lot of time trying to upgrade things so that they’re all happy. These checks don’t actually need to run at the same time, or communicate with each other, though - they’re independent scripts that just happen to run on the same machine. This got me thinking that there might be a way to isolate them from each other.
Docker is the obvious solution these days, but a) I don’t think there’s a way for Sensu to orchestrate booting up containers for checks and b) that seems very heavy-weight for checks that run in under a second (I imagine most of the time would be spent provisioning the containers). I suppose that I could try using rvm/bundler/rbenv/etc., but it seems like that might be a lot of work to get going with Sensu.
Does anyone have any experience with this sort of thing, or have alternate approaches to solving this problem?