Using Prometheus as a TSDB for Sensu Go

[dravenstone]

Sensu Go already has the ability to scrape metrics from Prometheus. However, we have a specific environment where the pull method that Prometheus utilizes is not an option. We are also in the process of deprecating our Graphite cluster and replace it with Prometheus. What we would like to do is deploy a sensu go agent + statsd to the clients in this restricted environment and have the metrics stored in Prometheus. We would like to avoid maintaining 2 separate TSDB’s. As I am writing this, I realize this is more of a feature request over an issue, however, I am curious if anyone else has this scenario and what have they tried to make it work.

I was thinking of taking the handler-opentsdb.rb (https://github.com/sensu-plugins/sensu-plugins-opentsdb) and modifying to send metrics to prometheus.

[cwjohnston]

Hi Dan, I’m not aware that anyone has implemented a handler for sending metrics to Prometheus TSDB but I can see how that would be useful!

The OpenTSDB handler you linked is written for the event structure used by Sensu 1.x. Since Sensu Go uses a different event structure, you might want to take a look at one of the following handler projects:

• https://github.com/sensu/sensu-influxdb-handler
• https://github.com/nixwiz/sensu-go-graphite-handler
• https://github.com/jkerry/sensu-opentsdb-handler
• https://github.com/jkerry/sensu-go-elasticsearch

Each of these projects use the new event format, and so benefit from sensu-agent’s capability to automatically transform various metric formats into Sensu’s intermediate metric point format.

Hope this helps!

[calebhailey]

Hi there, Dan? I hope all is well. This is on our radar as a new supported integration, as quite a few folks have inquired about similar workflows, but we haven’t schedule work on it yet.

We have two thoughts on the matter: use the Pushgateway¹, or leverage Sensu Go’s impending support for gRPC extensions (long running handler processes), and develop our own Prometheus integration which spools metrics for Prom to scrape.

prometheus/pushgateway

If you were interested in rolling your own I’d start with the Pushgateway.

[dravenstone]

Hey @calebhailey and @cwjohnston, thank you for the replies and feedback.
I think we are going to have to go with our own handler based on the suggestions you provided. Once we have a working prototype, I will let you know.

[Tarun_Prakash]

Any ETA on this? when are you guys planning to provide this. I will be great to as an addition to already amazing system.