Cassandra monitoring with Sensu currently, need to send to Prometheus now

Currently using Sensu to monitor Cassandra Servers, and recently another division started to use Prometheus. That group would like us to modify Sensu to send to Prometheus and display on Grafana.
Sensu is run as a service on a Red Hat Linux VM and in /etc/sensu/conf.d there are 2 files client.json, and rabbitmq.json the rabbitmq.json has the IP to send to. I can log in to the Red Hat VM where the IP is directed to and see the rabbitMQ server running. I also see Ruby Gems is involved. Is there an easy connection I can make or do I have to remove Sensu for the Prometheus exporter?

Hey!

So here’s a pattern you might try right now.

  1. Setup a promethus push gateway:
    https://github.com/prometheus/pushgateway

  2. Use the sensu-prom-mutator to convert Sensu events with metrics into Prometheus format:
    https://bonsai.sensu.io/assets/portertech/sensu-prom-mutator

  3. define a tcp metrics handler that sends the mutated event into the pushgateway

I’ve used a similar pattern to send promethus formatted metrics into sumologic collector, which provides a Prometheus pushgateway interface but sends to sumologic service instead of to a Prometheus server. But the pattern should be the same, you’ll just have to setup the pushgateway yourself.

Prometheus is explicitly a pull only approach and the pushgateway is there to provide the shim so you can make Prometheus work with push workflows.

Maybe this will be helpful, here’s the tcp handler I use to interact with the sumologic collector’s prometheus gateway
running on the same host as the sensu backend.

type: Handler
api_version: core/v2
metadata:
  name: sumologic_collector_prom_gateway
spec:
  env_vars: null
  filters: null
  handlers: null
  mutator: sensu-prometheus-mutator
  runtime_assets: null
  socket:
    host: localhost
    port: 9003
  timeout: 0
  type: tcp