Using Sensu Go with the TICK stack

Kapacitor has a pre-existing integration for Sensu Classic, making use of the client socket which is now deprecated in the Sensu Go agent, so I’d advise to avoid using that built in integration and instead use Kapacitor’s httppost event handler instead and post JSON to the Sensu Go agent events API.

The benefit to using the agent’s events api is that you can make full use of Sensu Go’s check resources attributes such as labels and annotations.

Here’s a really quick example I spun up,

In my kapacitor.conf I have created an httppost section to produce a sensu critical event using the httppost’s alert-template. The alert-template fills in the POST data with a JSON representation of a Sensu event. The url is just the standard Sensu agent events api, for an agent running on the same host as Kapacitor.

   [[httppost]]
     endpoint = "sensu-critical-alert"
     url = "http://127.0.0.1:3031/events"
     headers = { "Content-Type" = "application/json" }
     alert-template = "{\"check\": {\"metadata\": {\"name\": \"Kapacitor_Alert\", \"labels\": { \"alert_id\":\"{{.ID}}\",\"alert_level\": \"{{.Level}}\" } },\"status\": 2,\"output\": \"{{.Message}}\" } }"

I have the Kapacitor alert id and alert level encoded as Sensu check labels. And the Kapacitor alert message encoded as the Sensu Check output.

You can define multiple httppost sections, each with a unique endpoint in your kapacitor config
With this defined I can now write tickscripts that make use of the sensu-critical-alert endpoint.
Here’s an example:

dbrp "telegraf"."autogen"

stream
  |from()
    .measurement('cpu')
  |alert()
    .crit(lambda: "usage_idle" < 80)
    .message('CPU in heavy use')
    .post()
      .endpoint('sensu-critical-alert')
      .captureResponse()

Next up, I’m trying to figure out how to extend that alert-template so I map Kapacitor alert level into numeric Sensu status values. The templating language does allow for some conditional statements, but its a little complicated. To keep it easier to read I was just going to have separate endpoints for each Sensu numeric status that I want to use and have the tickscript post to different endpoints conditionally. Anyone have thoughts on the best way to do that?

1 Like