Forward sensu keepalive metrics in graphite

Sensu Classic (EOL)
1

Hello all,

right now im looking for a solution for an availability reporting based in sensu agent keepalive.

unfortunately by default this feature doesn’t exist!

So right know I try to configure the keepalive handler to forward the keepalive result in graphite.
for this I create a handler named keepalive

---
type: Handler
api_version: core/v2
metadata:
  name: keepalive
  namespace: default
spec:
  handlers:
    - file
    # - graphite
  type: set

and he call another handler who should store in a file the check result (for now then in graphite). But I cant interpolate the sense value

---
api_version: core/v2
type: Handler
metadata:
  name: file
spec:
  type: pipe
  command: 'echo {{ .Check.Status }} >> /tmp/test'
  timeout: 10

maybe there is a better solution than this!
Thanks for you help !

2

Hey!
I think you might want to try using the built in only_check_output mutator a a tcp handler configured to communicate with graphite’s tcp port.

https://docs.sensu.io/sensu-go/latest/observability-pipeline/observe-transform/mutators/#built-in-mutator-only_check_output

if this isn’t the right solution, then I’m pretty sure some sort of custom mutator probably is what you want.

A couple of things to note in your attempt so far:
First, handlers in a handler set/array are not run in sequential order, they are run in parallel, so you can’t be sure that a handler named file will run before a handler named graphite. This is why mutators are there in the design. Mutators let you arbitrarily convert the json representation of a Sensu event into any text format before passing it on to the handler the mutator is attached to.

Second, in your command you are mixing posix shell executable echo with what looks like golang string template syntax, which isn’t going to work. You just have to be careful with regard to what syntax is appropriate for a specific context.

Hope this helps

3

Thanks for the answer, i’ll take a look on this, thanks !

4

this mutator is almost what we want (check.status would be better), but I have some trouble, seems the mutator is configured correctly because I can see the handler is executed successfully in the sensu backend and I see a connection on graphite but I can find anything in grafana!
sensu backend log

sensu/sensu-backend-2[sensu-backend]: {"check_name":"keepalive","check_namespace":"LAB","component":"pipelined","entity_name":"fedora-dpg","entity_namespace":"LAB","event_id":"0bfd0688-1b8e-4c0f-83ad-12a60014d8ba","handler":"graphite-keepalive","level":"debug","msg":"allowing event","time":"2021-07-01T12:32:28+02:00"}
sensu/sensu-backend-2[sensu-backend]: {"check_name":"keepalive","check_namespace":"LAB","component":"pipelined","entity_name":"fedora-dpg","entity_namespace":"LAB","event_id":"0bfd0688-1b8e-4c0f-83ad-12a60014d8ba","handler":"graphite-keepalive","level":"info","msg":"sending event to handler","time":"2021-07-01T12:32:28+02:00"}
sensu/sensu-backend-2[sensu-backend]: {"check_name":"keepalive","check_namespace":"LAB","component":"pipelined","entity_name":"fedora-dpg","entity_namespace":"LAB","event_id":"0bfd0688-1b8e-4c0f-83ad-12a60014d8ba","handler_name":"graphite-keepalive","handler_namespace":"LAB","handler_protocol":"tcp","level":"debug","msg":"sending event to socket handler","time":"2021-07-01T12:32:28+02:00"}
sensu/sensu-backend-2[sensu-backend]: {"bytes":69,"check_name":"keepalive","check_namespace":"LAB","component":"pipelined","entity_name":"fedora-dpg","entity_namespace":"LAB","event_id":"0bfd0688-1b8e-4c0f-83ad-12a60014d8ba","handler_name":"graphite-keepalive","handler_namespace":"LAB","handler_protocol":"tcp","level":"info","msg":"event socket handler executed","time":"2021-07-01T12:32:28+02:00"}

graphite log:

graphite/graphite-0[graphite]: 01/07/2021 10:32:28 :: [listener] MetricLineReceiver connection with x.x.x.x:42424 established
graphite/graphite-0[graphite]: 01/07/2021 10:32:28 :: [listener] MetricLineReceiver connection with x.x.x.x:42424 closed cleanly

this is my handler definition:

---
type: Handler
api_version: core/v2
metadata:
  name: keepalive
  namespace: LAB
spec:
  handlers:
  - graphite-keepalive
  type: set
---
type: Handler
api_version: core/v2
metadata:
  name: graphite-keepalive
  namespace: LAB
spec:
  mutator: only_check_output
  socket:
    host: x.x.x.x
    port: 2003
  type: tcp
5

the following Handler is what i want, but it return an invalid line erro on graphite.
The outut of the following command :

echo ".keepalivestatus $(jq .check.status) `date +%s`"

and it return .keepalivestatus 0 1625145848

my handler definition:

---
type: Handler
api_version: core/v2
metadata:
  name: keepalive
  namespace: LAB
spec:
  handlers:
  - graphite-keepalive
  type: set
---
api_version: core/v2
type: Handler
metadata:
  name: graphite-keepalive
spec:
  type: pipe
  command: echo ".keepalivestatus $(jq .check.status) `date +%s`" | sensu-go-graphite-handler --host x.x.x.x --port 2003
  timeout: 10
  runtime_assets:
  - sensu-go-graphite-handler

graphite log:

graphite/graphite-0[graphite]: 01/07/2021 13:21:48 :: [listener] invalid line received from client 11.32.242.2:52004, ignoring [0]
6

I don’t think graphite allows metrics_path that starts with a period. You may need to do something like
something.keepalivestatus

7

I forgot to reply yesterday but you’re right

the following command works:

echo "sensu.my_host.keepalive-metrics_.keepalive-metrics.status $(jq -r .check.status) `date +%s`" | nc x.x.x.x 2003

Thanks!