Is there a way to generate an alert when a metric exceeds a threshold?
Let’s say I’m using cpu-pcnt-usage-metrics.rb to push metrics into Graphite. But I also want to generate a warning when CPU idle is below 50%, and a critical when below 25% (fictional values).
One way to do it, I could also enable check-cpu.rb as a regular check and set thresholds there. But this seems wasteful - I am collecting CPU data already with the other plugin, there’s no need to increase the load everywhere to collect the same thing twice.
Another way, I could use check-data.rb, go into Graphite, retrieve the data I’ve collected with the metric plugin, and set thresholds there. This seems unnecessarily complicated, in a way. Also, now I depend on Graphite to generate my alerts, so if something happens to Graphite, I will not receive much-needed alerts. If past experience is any indication, things tend to break exactly when you need them most.
Is there a third way to do this, that uses Sensu exclusively, without redundant data collection? check-aggregate.rb seems a bit close to what I need, but I don’t actually need aggregates, only individual checks, and also I do need to dump that data into Graphite eventually.
If I could only insert a check in the “pipeline” while the data is being routed from sensu-client to Graphite - that would be perfect.