Aye! You’ve hit on something there for sure.
This is why my brain probably jumped to running an agent at the host level, so you can run existing checks that provide an instant warn/critical without going through a tsdb first and having to do a query.
The “everything is a metrics” way of thinking puts the alert conditional logic after the tsdb ingestion, which works great for complicated questions that require looking at trending behavior over time.
But if you want to be alerted just based on latest thresholds, it can feel a bit cumbersome to do it that way.
So off the top of my head, you have a couple of options (there are probably more options as well).
- you can write a thin wrapper script that runs as the check command that parses the metric you want looking for threshold values from the node exporter metrics, before passing them on as output for sensu-agent to repackage in Sensu’s internal metrics format.
Because the Sensu check definition allows you specify different handlers for status handling and for metric handling… its possible to write your node exporter wrapper script such that it reports non-zero status (triggering an actionable alert via a Sensu event) and also exports the metrics to your tsdb for trending queries to be done later. As long as you don’t disrupt the output by printing information to stdout or stderr and just change the return status.
2) you can write a check that queries the tsdb and reports a warning/critical status based on that query.