Jef Practice: Monitoring Legacy Unix systems with Sensu Go

Having Sensu Go written in Golang comes with some great forward looking benefits, especially for containerized workloads. But… Golang doesn’t compile natively on legacy Unix environments so we can’t easily offer a native agent to run on critical systems that are still in operation. If they weren’t critical, we’d all have found a way to move them to linux most likely. Even though there’s no native agent for these legacy systems Sensu Go can still support some monitoring workloads by making use of agent running on another system acting as a proxy entity in a few different ways.

Proxy Check
You can make use of Sensu Go’s proxy check concepts to use an Sensu Go agent to run a command on behalf of the legacy system and get full access to the Sensu pub/sub model to centrally define new checks. There are a couple of very powerful patterns here you could consider using:

  • Service Check
    If your agentless system is exporting a service that provides health information, you can implement a proxy check to interact with that service endpoint and flow the data back into the Sensu Go event pipeline as if the check came for the target system.We’ve got a guide to how to do exactly this:
    https://docs.sensu.io/sensu-go/5.0/guides/monitor-external-resources/

  • Remote command
    We can go further here and use an ssh (or for very old systems maybe even rsh) to collect the output of a remotely executed command. You won’t be able to take advantage of assets with this pattern, and you will have to pre-provision the check commands you want to remotely run on the target syste. It will also require you take some care to setup a user on the target system and securely provision the access keys to the sensu user running the agent.

Sensu Events API
It’s possible to generate Sensu check events directly on the legacy system and upload that event into the Sensu event pipeline. You can export simple JSON strings from your custom monitoring checks directly into the Sensu backend via the event API, or to events api exposed by the agent. The nice thing about using an agent events api for this is you can leverage its existing authorization to the Sensu backend instead of having to deal with the API auth in your custom monitoring.
Using this pattern, you can’t rely on Sensu Go’s pub/sub model to schedule checks, you’ll be doing that scheduling on the target system with cron. But you still have access to the full Sensu event pipeline, so you can request specific defined handlers to run, like it was an event generated from a Sensu check running on an agent.
Check out an example in the Agent reference documentation:
https://docs.sensu.io/sensu-go/latest/reference/agent/#creating-monitoring-events-using-the-agent-api

Sensu Agent Statd Collector
The simplest way to upload monitoring events from an agentless system, is to have it emit statsd metrics to the agent’s statsd collector socket. The agent will convert that statsd metric into a specially constructed Sensu event and send it on the Sensu backend for processing. The statsd events are the least flexible option as all statsd events created by the agent the use the same statsd handler. So when working with multiple legacy systems, the choice between generating a statsd metric or a Sensu event ends up being weighing the trade off monitoring script complexity versus Sensu backend handler complexity.

2 Likes

When you say “agent socket” here, does that mean the agent socket on the sensu-backend, sensu agent?

let me clean it up… i meant to say the agent events api. I’ll try to put together an example as well.

Howdy @Kristian, @jspaleta there is an example in the docs of sending events to the agent API: https://docs.sensu.io/sensu-go/latest/reference/agent/#creating-monitoring-events-using-the-agent-api that might help in this case.

1 Like