Using Sensu Go with the TICK stack

Sensu Go
1

Kapacitor has a pre-existing integration for Sensu Classic, making use of the client socket which is now deprecated in the Sensu Go agent, so I’d advise to avoid using that built in integration and instead use Kapacitor’s httppost event handler instead and post JSON to the Sensu Go agent events API.

The benefit to using the agent’s events api is that you can make full use of Sensu Go’s check resources attributes such as labels and annotations.

Here’s a really quick example I spun up,

In my kapacitor.conf I have created an httppost section to produce a sensu critical event using the httppost’s alert-template. The alert-template fills in the POST data with a JSON representation of a Sensu event. The url is just the standard Sensu agent events api, for an agent running on the same host as Kapacitor.

   [[httppost]]
     endpoint = "sensu-critical-alert"
     url = "http://127.0.0.1:3031/events"
     headers = { "Content-Type" = "application/json" }
     alert-template = "{\"check\": {\"metadata\": {\"name\": \"Kapacitor_Alert\", \"labels\": { \"alert_id\":\"{{.ID}}\",\"alert_level\": \"{{.Level}}\" } },\"status\": 2,\"output\": \"{{.Message}}\" } }"

I have the Kapacitor alert id and alert level encoded as Sensu check labels. And the Kapacitor alert message encoded as the Sensu Check output.

You can define multiple httppost sections, each with a unique endpoint in your kapacitor config
With this defined I can now write tickscripts that make use of the sensu-critical-alert endpoint.
Here’s an example:

dbrp "telegraf"."autogen"

stream
  |from()
    .measurement('cpu')
  |alert()
    .crit(lambda: "usage_idle" < 80)
    .message('CPU in heavy use')
    .post()
      .endpoint('sensu-critical-alert')
      .captureResponse()

Next up, I’m trying to figure out how to extend that alert-template so I map Kapacitor alert level into numeric Sensu status values. The templating language does allow for some conditional statements, but its a little complicated. To keep it easier to read I was just going to have separate endpoints for each Sensu numeric status that I want to use and have the tickscript post to different endpoints conditionally. Anyone have thoughts on the best way to do that?

2 Likes
2

Did you get any further with this? I was also looking at doing something similar.

I’m slightly put off by the need to add a httppost endpoint for every agent I want to send data too. I’ve been trying to get around this but it seems you can only specify an alert-template to reformat the body if you define it in the config file and not if you use the inline .post(URL) syntack in the tick script.

3

Hey,
I think perhaps you misunderstand what’s going on in the example above.
I’ll try to clear up any confusion. You don’t have to have to implement an httppost for each agent. You are just using a single sensu agent’s event api as a way to have Kapacitor send events into Sensu. Its not a matter of sending data to every Sensu agent, its just a matter of selecting a single sensu agent to send data to.

You just need to have a single Sensu agent that Kapacitor can communicate with. In my example I had the agent running on the same host as the Kapacitor service, which works with the default bindings for the Sensu agent’s event api, which by default only binds to localhost instead of the public network (but you can override that as part of the agent config).

Does that clear it up for you?

4

In tick you can use .Details to compose the POST and conditionals within it. Then you can use it handler. Something like:

tick:
|alert()
.crit(lambda: “usage_idle” < 80)
.message(‘CPU in heavy use’)
.details(’’’{“check”: {“metadata”: {“name”: “whatever_name” },“handlers”: [“slack”, “email”],“status”: {{ if eq .Level “CRITICAL” }}2{{ else if eq .Level “WARNING” }}1{{ else }}0{{ end }},“output”: “{{.Message}}” } }"’’’)
.post()
.endpoint(‘sensu-critical-alert’)
.captureResponse()

handler:
alert-template="{{.Details}}"

You may want to edit message as it may sound strange for different alert levels…

5

Hey!

Thanks for the input!
FYI, this httppost workaround I originally post will be obsolete real soon.

There is a new Sensu Go integration merged into telegraph and should be available in telegraph next release:
https://github.com/influxdata/telegraf/tree/master/plugins/outputs/sensu

1 Like