How to debug sensugo influxdb handler to troublshoot not writing metrics in influxdb?

[masifpak]

I have sensugo server cluster of 3 nodes having one sensuctl configured and one influxdb server.
Sensugo server can telnet influxdb server on 8086 port so no networking issue.
I can also see event of sensu-metric-cpu of one agent by using following command. but these metric are not being written in influxdb. How can we debug the issue?

sudo sensuctl event info --format json feed-01 sensu-metrics-cpu

[todd]

Can you post your sensu-metrics-cpu check definition?

My guess is you’re not using the right output_metric_format definition. It needs to be what the metric check is generating, not what you are sending it to. And if you are using the check I think you are, the output_metric_format should be “graphite_plaintext”.

[masifpak]

I also tried graphite_plaintext. Interesting thing is i just copied it from sensu-metrics-memory and changes command param. sensu-metrics-memory was working fine.

---
type: CheckConfig
api_version: core/v2
metadata:
  name: sensu-metrics-cpu
  namespace: Pakcloud-Prod
spec:
  check_hooks: null
  command: "/opt/sensu-plugins-ruby/embedded/bin/metrics-cpu.rb"
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 30
  low_flap_threshold: 0
  output_metric_format: influxdb_line
  output_metric_handlers:
  - influx-dbnew
  proxy_entity_name: ''
  publish: true
  round_robin: false
  runtime_assets: []
  stdin: false
  subdue: null
  subscriptions:
  - metrics
  timeout: 300
  ttl: 0

[masifpak]

A larg output can be the reason? Memory metric is working fine but cou and disk are facing issue. Both cpu and disk carry a larg output.

[todd]

No, large output should not be an issue.

First let’s make sure that the metrics are coming through as expected. If you could create the following handler:

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "debug",
    "namespace": "Pakcloud-Prod"
  },
  "spec": {
    "command": "jq . >> /tmp/event_metrics.json",
    "env_vars": null,
    "filters": null,
    "handlers": null,
    "runtime_assets": null,
    "timeout": 0,
    "type": "pipe"
  }
}

And then attach that handler to your ‘sensu-metrics-cpu’ check as an additional item in ‘output_metric_handlers’ for a few minutes to allow the check to run a few times, then remove it from the check ‘output_metric_handlers’ so as not to fill up the filesystem.

On one of your backend nodes, you should find that /tmp/event_metrics.json file that contains the events generated.

Assuming you have the ‘jq’ utility installed, running the following:

$ jq '.metrics.points | length' /tmp/event_metrics.json

You should see a count of the metrics points collected for each event. If this is zero (0) then Sensu is not picking up your metrics. My running of metrics-cpu.rb produces 27 metrics points. And the ‘output_metric_format’ is definitely ‘graphite_plaintext’.

[masifpak]

It is returning stuff like this. Where issue can bee then?
57
97
57
97
97
57
97
57
97
57
57

[masifpak]

Metrics of some machine were saving in influxdb but some of not. I just removed “has_metrics” from influxdb handler and all data start comming in influxdb. Graphs are of this format as attached. I think these are not metrics.

[masifpak]

@todd Please we were at the edge of fixing it.
I am starting from scratch again.
I am using metrics-cpu.rb, metrics-memory.rb and metrics-disk.rb for getting metrics. Please let me know what I am missing.?

Or let me know what is best monitoring stack, like sensugo+influxdb+grafana or something else.

[masifpak]

[Sensu and InfluxDB] Storing Data from Metrics Collection Checks