I’m trying to create a monitoring/management solution for a project and have been toying with integrating Sensu.
Here’s the gist: it’s a road exhibit getting internet access via a Cradlepoint modem with a 4g network card. This means that we don’t have a public IP address we can forward traffic from and Internet connectivity in general is incredibly spotty. We want to setup a monitoring dashboard so we can see when the machines are connected to the internet, get metrics from the machines, and generally do a few checks to make sure the software is running correctly. So far, Sensu seems like a great tool for the job and I’d love to integrate it.
However, we also want to do stuff like trigger machine restarts remotely. Now if it was just a matter of restarting all of the machines at once it would be simple enough just to subscribe all the machines to the same check, set it as “publish”: false and trigger it with the “/request” endpoint, but the idea is that we should be able to trigger restarts on each particular machine, which leads me to my question: Is there a way for Sensu to manually trigger a check on a particular client? I could theoretically subscribe each machine to it’s own channel and create a check for each one but we’d like the solution we come up with to be reusable with other projects and I can’t imagine such a system would scale to something with 30+ machines (especially whenever we want to add a new command to be run)
I should note that I realize this is not really the kind of thing that Sensu is designed for. We’ve done stuff like this in the past using things like Ansible and raw ssh commands and I do have each of the machines setup with a reverse ssh tunnel so we can connect to the machines to debug, but it’s flaky at best what with the unstable internet. I guess I just felt that since we’d already have Sensu running, sending keepalive heartbeats, and maintaining a transport layer to send commands to every machine already it would be great if we could leverage it for our purposes. Any advice?