Pushing check results to influxdb?

Hello,

I can’t seem to figure out how to push sensu go check results into influxdb. I’m not referring to metrics collection (which works fine for me), but a basic service check that returns OK, WARNING, CRITICAL results. I want to keep long term data for certain checks in influxdb and all the examples and documentation I find seems to refer only to metrics collection. I’ve tried mixing some of the example metrics collection config parameters into regular checks without joy. I’m sure it’s easy and I am doing something silly, but can anyone provide a basic service check example that push result to influxdb?

Thanks!

2 Likes

Hey!

I think the best way forward for you is to add a mutator into your event pipeline and mutator the event definition and convert the check status into a metric of your choice. You should be able to do this with jq in a bash script for now, I’m going to work on a more general golang mutator asset that should work in the more general case and get back to you.

Thanks Jef, that would be super helpful if you could provide a generic example!

@paulp actually the latest version of the Influxdb handler has a command line option --check-status-metric that should do that for you.

However, along the lines of what Jef had mentioned, I’ve created a mutator that does similar and should work with any TSDB. It’s on Bonsai here.

1 Like

Thanks so much @todd! I will look into these solutions…

@todd @jspaleta thank you both for your help. I have confirmed the latest version of InfluxDB handler 3.2.0 (which for some reason hasn’t been pushed to Bonsai) does indeed help me push check status into influxdb by utilizing the newly added parameter “–check-status-metric”.

Thanks for letting us know it wasn’t updated on Bonsai. That has been fixed now as well.

Awesome I just enabled --check-status-metric and added the handler parameters to the related checks. Works like a charm :+1:

2 Likes

For the next update of the influxdb handler I would like to request the following feature:

Send the check result message next to the check result status. This would make the reason of a failed check more transparent.

The next release will include support for annotation events, a 1.x feature parity. Is this similar to what you are requesting? If not, I would encourage you to submit this is an issue.

By annotation you dont mean the check response message or?

AllNodesAreReady OK: All nodes are reporting as ready

It’s a Grafana thing. It allows you to align events to your graph’s timeline.

Example of annotation with this Grafana Annotation definition: select description from sensu_event where $timeFilter ORDER BY time ASC

1 Like

This looks definitely promising.
Is there already an estimated release date available?

Just awaiting PR approval/merge. We will cut a release after that. Hopefully done by the end of this week.

1 Like

hi @todd I have seen that the new version has been released.
Any hint about what I have to do to get the annotation within grafana now?

Thanks

It’s a per-dashboard configuration, so go to the dashboard you wish to add annotations to and click on the gear icon. In the left navigation column you should see Annotations. It’s directly under General in my instance. And then click on New. Here’s what the definition for mine looks like.

sorry for asking again - but I am still struggling with that.
Where is the sensu_event table coming from to which you are referring in your query?

It’s a table that gets populated by the handler whenever an event it receives includes a check that is failing and when it resolves.

1 Like

ahhh I see - works like a charm
many thanks :slight_smile:

3 Likes