Hey guys,
I’m trying to create a monitoring/management solution for a project and have been toying with integrating Sensu.
Here’s the gist: it’s a road exhibit getting internet access via a Cradlepoint modem with a 4g network card. This means that we don’t have a public IP address we can forward traffic from and Internet connectivity in general is incredibly spotty. We want to setup a monitoring dashboard so we can see when the machines are connected to the internet, get metrics from the machines, and generally do a few checks to make sure the software is running correctly. So far, Sensu seems like a great tool for the job and I’d love to integrate it.
However, we also want to do stuff like trigger machine restarts remotely. Now if it was just a matter of restarting all of the machines at once it would be simple enough just to subscribe all the machines to the same check, set it as “publish”: false and trigger it with the “/request” endpoint, but the idea is that we should be able to trigger restarts on each particular machine, which leads me to my question: Is there a way for Sensu to manually trigger a check on a particular client? I could theoretically subscribe each machine to it’s own channel and create a check for each one but we’d like the solution we come up with to be reusable with other projects and I can’t imagine such a system would scale to something with 30+ machines (especially whenever we want to add a new command to be run)
I should note that I realize this is not really the kind of thing that Sensu is designed for. We’ve done stuff like this in the past using things like Ansible and raw ssh commands and I do have each of the machines setup with a reverse ssh tunnel so we can connect to the machines to debug, but it’s flaky at best what with the unstable internet. I guess I just felt that since we’d already have Sensu running, sending keepalive heartbeats, and maintaining a transport layer to send commands to every machine already it would be great if we could leverage it for our purposes. Any advice?
You are right that creating a subscription (channel) for each host is one way to do it. I’ve experimented with that approach. I don’t see any particular reason it wouldn’t scale.
Keep in mind with just a subscription of the fqdn and a check name called “remote_reboot” or something, sounds pretty doable. You dont need a unique check per host:
https://sensuapp.org/docs/0.25/api/checks-api.html#request-post-specification
This is kinda similar to how the sensu remediation plugin works:
https://github.com/sensu-plugins/sensu-plugins-sensu/blob/master/bin/handler-sensu.rb
···
On Wed, Jun 22, 2016 at 8:08 AM, charlesdimaggio@localprojects.com wrote:
Hey guys,
I’m trying to create a monitoring/management solution for a project and have been toying with integrating Sensu.
Here’s the gist: it’s a road exhibit getting internet access via a Cradlepoint modem with a 4g network card. This means that we don’t have a public IP address we can forward traffic from and Internet connectivity in general is incredibly spotty. We want to setup a monitoring dashboard so we can see when the machines are connected to the internet, get metrics from the machines, and generally do a few checks to make sure the software is running correctly. So far, Sensu seems like a great tool for the job and I’d love to integrate it.
However, we also want to do stuff like trigger machine restarts remotely. Now if it was just a matter of restarting all of the machines at once it would be simple enough just to subscribe all the machines to the same check, set it as “publish”: false and trigger it with the “/request” endpoint, but the idea is that we should be able to trigger restarts on each particular machine, which leads me to my question: Is there a way for Sensu to manually trigger a check on a particular client? I could theoretically subscribe each machine to it’s own channel and create a check for each one but we’d like the solution we come up with to be reusable with other projects and I can’t imagine such a system would scale to something with 30+ machines (especially whenever we want to add a new command to be run)
I should note that I realize this is not really the kind of thing that Sensu is designed for. We’ve done stuff like this in the past using things like Ansible and raw ssh commands and I do have each of the machines setup with a reverse ssh tunnel so we can connect to the machines to debug, but it’s flaky at best what with the unstable internet. I guess I just felt that since we’d already have Sensu running, sending keepalive heartbeats, and maintaining a transport layer to send commands to every machine already it would be great if we could leverage it for our purposes. Any advice?
Oh that’s great, I didn’t realize you could actually override the “subscribers” field through the request. It looks like according to https://github.com/sensu-plugins/sensu-plugins-sensu/blob/master/bin/handler-sensu.rb (lines 83/92) it’s actually setting the “subscribers” field to the client’s name by default but testing it out I wasn’t able make it work so I might look into that. But in any case it’s trivial just to give each client a unique subscription, thank you so much!
···
On Wed, Jun 22, 2016 at 11:16 AM, Kyle Anderson kyle@xkyle.com wrote:
You are right that creating a subscription (channel) for each host is one way to do it. I’ve experimented with that approach. I don’t see any particular reason it wouldn’t scale.
Keep in mind with just a subscription of the fqdn and a check name called “remote_reboot” or something, sounds pretty doable. You dont need a unique check per host:
https://sensuapp.org/docs/0.25/api/checks-api.html#request-post-specification
This is kinda similar to how the sensu remediation plugin works:
https://github.com/sensu-plugins/sensu-plugins-sensu/blob/master/bin/handler-sensu.rb
On Wed, Jun 22, 2016 at 8:08 AM, charlesdimaggio@localprojects.com wrote:
Hey guys,
I’m trying to create a monitoring/management solution for a project and have been toying with integrating Sensu.
Here’s the gist: it’s a road exhibit getting internet access via a Cradlepoint modem with a 4g network card. This means that we don’t have a public IP address we can forward traffic from and Internet connectivity in general is incredibly spotty. We want to setup a monitoring dashboard so we can see when the machines are connected to the internet, get metrics from the machines, and generally do a few checks to make sure the software is running correctly. So far, Sensu seems like a great tool for the job and I’d love to integrate it.
However, we also want to do stuff like trigger machine restarts remotely. Now if it was just a matter of restarting all of the machines at once it would be simple enough just to subscribe all the machines to the same check, set it as “publish”: false and trigger it with the “/request” endpoint, but the idea is that we should be able to trigger restarts on each particular machine, which leads me to my question: Is there a way for Sensu to manually trigger a check on a particular client? I could theoretically subscribe each machine to it’s own channel and create a check for each one but we’d like the solution we come up with to be reusable with other projects and I can’t imagine such a system would scale to something with 30+ machines (especially whenever we want to add a new command to be run)
I should note that I realize this is not really the kind of thing that Sensu is designed for. We’ve done stuff like this in the past using things like Ansible and raw ssh commands and I do have each of the machines setup with a reverse ssh tunnel so we can connect to the machines to debug, but it’s flaky at best what with the unstable internet. I guess I just felt that since we’d already have Sensu running, sending keepalive heartbeats, and maintaining a transport layer to send commands to every machine already it would be great if we could leverage it for our purposes. Any advice?