Adding tags to metrics from system-check

Sensu Go
1

I collect metrics with system-check and I inject them into InfluxDB via the handler. All works well.

Agents are labeled like this, with different values for different clusters:

labels:
  - cluster: "sensu"

Looking at the columns in the Influx measurements, I only see these columns:

> select * from system_cpu_idle limit 10
name: system_cpu_idle
time                cpu       host     prom_type value
----                ---       ----     --------- -----
1639082899000000000 cpu-total sensu-01 gauge     87.39352640560011
1639082899000000000 cpu-total sensu-02 gauge     88.34459459463778
1639082899000000000 cpu-total sensu-03 gauge     88.79456706229149

Looking at the metrics['points'] section in the event JSON, I see this, which matches the two extra columns in measurements:

      {
        "name": "system_cpu_idle",
        "value": 99.91670137628196,
        "timestamp": 1639082959,
        "tags": [
          {
            "name": "prom_type",
            "value": "gauge"
          },
          {
            "name": "cpu",
            "value": "cpu-total"
          }
        ]
      },

What is the canonical way to take an agent label (such as cluster=sensu) and add it as a tag to the metrics points? I believe that would add the tag as a column in Influx (correct me if I’m wrong). This would simplify filtering metrics in Grafana.

2

Perhaps related - I have declared the label in agent.yml, but I do not see it when I describe the agent entity with sensuctl. I’ve tried to delete the agent entity, restart the agent, etc. Nothing works.

Here’s agent.yml:

---
namespace: "default"
subscriptions:
  - http-checks-remote
  - system-check
labels:
  - cluster: "sensu"
annotations:
  - cluster: "sensu"
agent-managed-entity: true
backend-url:
  - "ws://127.0.0.1:8081"
deregister: false
deregistration-handler: "slack"
detect-cloud-provider: true
keepalive-critical-timeout: 60
keepalive-handlers:
  - slack
keepalive-interval: 10
keepalive-warning-timeout: 30
log-level: "info" # available log levels: panic, fatal, error, warn, info, debug
user: "XXXX"
password: "XXXX"

This is the agent entity info:

type: Entity
api_version: core/v2
metadata:
  created_by: spree3d
  labels:
    sensu.io/managed_by: sensu-agent
  name: sensu-01
  namespace: default
spec:
  deregister: false
  deregistration:
    handler: slack
  entity_class: agent
  keepalive_handlers:
  - slack
  last_seen: 1639098298
  redact:
  - password
  - passwd
  - pass
  - api_key
  - api_token
  - access_key
  - secret_key
  - private_key
  - secret
  sensu_agent_version: 6.6.2
  subscriptions:
  - http-checks-remote
  - system-check
  - entity:sensu-01
  system:
    arch: amd64
    cloud_provider: EC2
    hostname: sensu-01
    libc_type: glibc
    network:
      interfaces:
      - addresses:
        - 127.0.0.1/8
        name: lo
      - addresses:
        - 10.2.82.167/21
        - fe80::b6:4fff:fe55:f7f9/64
        mac: 02:b6:4f:55:f7:f9
        name: eth0
    os: linux
    platform: ubuntu
    platform_family: debian
    platform_version: "20.04"
    processes: null
    vm_role: guest
    vm_system: xen
  user: XXXX
3

Apparently labels can only be added via environment variables. I have this in /etc/default/sensu-agent and now I see labels and annotations in the agent entity info:

SENSU_LABELS='{"cluster": "sensu"}'
SENSU_ANNOTATIONS='{"cluster": "sensu"}'

However, the initial problem remains - how to inject a cluster (instance type) differentiator into the metrics sent to InfluxDB. The metrics are initially collected with system-check:

---
type: CheckConfig
api_version: core/v2
metadata:
  name: collect-system-metrics
spec:
  check_hooks: null
  command: system-check
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 10
  low_flap_threshold: 0
  output_metric_format: prometheus_text
  output_metric_handlers:
  - pipe_handler_minimum
{% for host in groups['sensu_server'] %}
  - influxdb-handler-{{ hostvars[host]['hostname'] }}
{% endfor %}
  pipelines: []
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - system-check
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - system-check
  timeout: 0
  ttl: 0
4

Your agent.yml example showed labels and annotations set as YAML arrays (of dictionaries), but they should be YAML dictionaries.

Array (wrong):

labels:
- foo: bar 
- bar: baz

Dictionary (correct):

labels:
  foo: bar 
  bar: baz
annotations:
  foo: bar 
  bar: baz

The answer to your original question is output_metric_tags, which also works with Sensu Tokens!

Example:

output_metric_tags:
  - name: entity
    value: "{{ .name }}"
  - name: namespace
    value: "{{ .namespace }}"
  - name: os
    value: "{{ .system.os }}"

This PR has a complete example:

I hope this helps!

5

Caleb - great answer! That solved both issues. Thank you!