How to debug sensugo influxdb handler to troublshoot not writing metrics in influxdb?

I have sensugo server cluster of 3 nodes having one sensuctl configured and one influxdb server.
Sensugo server can telnet influxdb server on 8086 port so no networking issue.
I can also see event of sensu-metric-cpu of one agent by using following command. but these metric are not being written in influxdb. How can we debug the issue?

sudo sensuctl event info --format json feed-01 sensu-metrics-cpu

Can you post your sensu-metrics-cpu check definition?

My guess is you’re not using the right output_metric_format definition. It needs to be what the metric check is generating, not what you are sending it to. And if you are using the check I think you are, the output_metric_format should be “graphite_plaintext”.

I also tried graphite_plaintext. Interesting thing is i just copied it from sensu-metrics-memory and changes command param. sensu-metrics-memory was working fine.


type: CheckConfig
api_version: core/v2
metadata:
name: sensu-metrics-cpu
namespace: Pakcloud-Prod
spec:
check_hooks: null
command: “/opt/sensu-plugins-ruby/embedded/bin/metrics-cpu.rb”
env_vars: null
handlers:
high_flap_threshold: 0
interval: 30
low_flap_threshold: 0
output_metric_format: influxdb_line
output_metric_handlers:

  • influx-dbnew
    proxy_entity_name: ‘’
    publish: true
    round_robin: false
    runtime_assets:
    stdin: false
    subdue: null
    subscriptions:
  • metrics
    timeout: 300
    ttl: 0

A larg output can be the reason? Memory metric is working fine but cou and disk are facing issue. Both cpu and disk carry a larg output.

No, large output should not be an issue.

First let’s make sure that the metrics are coming through as expected. If you could create the following handler:

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "debug",
    "namespace": "Pakcloud-Prod"
  },
  "spec": {
    "command": "jq . >> /tmp/event_metrics.json",
    "env_vars": null,
    "filters": null,
    "handlers": null,
    "runtime_assets": null,
    "timeout": 0,
    "type": "pipe"
  }
}

And then attach that handler to your ‘sensu-metrics-cpu’ check as an additional item in ‘output_metric_handlers’ for a few minutes to allow the check to run a few times, then remove it from the check ‘output_metric_handlers’ so as not to fill up the filesystem.

On one of your backend nodes, you should find that /tmp/event_metrics.json file that contains the events generated.

Assuming you have the ‘jq’ utility installed, running the following:

$ jq '.metrics.points | length' /tmp/event_metrics.json

You should see a count of the metrics points collected for each event. If this is zero (0) then Sensu is not picking up your metrics. My running of metrics-cpu.rb produces 27 metrics points. And the ‘output_metric_format’ is definitely ‘graphite_plaintext’.

It is returning stuff like this. Where issue can bee then?
57
97
57
97
97
57
97
57
97
57
57

Metrics of some machine were saving in influxdb but some of not. I just removed “has_metrics” from influxdb handler and all data start comming in influxdb. Graphs are of this format as attached. I think these are not metrics.

@todd Please we were at the edge of fixing it.
I am starting from scratch again.
I am using metrics-cpu.rb, metrics-memory.rb and metrics-disk.rb for getting metrics. Please let me know what I am missing.?

Or let me know what is best monitoring stack, like sensugo+influxdb+grafana or something else.