Is there a way to generate an alert when a metric exceeds a threshold?
Let’s say I’m using cpu-pcnt-usage-metrics.rb to push metrics into Graphite. But I also want to generate a warning when CPU idle is below 50%, and a critical when below 25% (fictional values).
One way to do it, I could also enable check-cpu.rb as a regular check and set thresholds there. But this seems wasteful - I am collecting CPU data already with the other plugin, there’s no need to increase the load everywhere to collect the same thing twice.
Another way, I could use check-data.rb, go into Graphite, retrieve the data I’ve collected with the metric plugin, and set thresholds there. This seems unnecessarily complicated, in a way. Also, now I depend on Graphite to generate my alerts, so if something happens to Graphite, I will not receive much-needed alerts. If past experience is any indication, things tend to break exactly when you need them most. 
Is there a third way to do this, that uses Sensu exclusively, without redundant data collection? check-aggregate.rb seems a bit close to what I need, but I don’t actually need aggregates, only individual checks, and also I do need to dump that data into Graphite eventually.
If I could only insert a check in the “pipeline” while the data is being routed from sensu-client to Graphite - that would be perfect.
You can have your check send its data to multiple handlers though they need to receive the same ‘type’ (Pipe, metric, etc) . One handler push the data to Graphite, a second process your CPU thresholds.
“handlers”: [“graphite_tcp”,“CPU_check”]
-Fred
···
On Tuesday, September 23, 2014 6:55:05 PM UTC-7, Florin Andrei wrote:
Is there a way to generate an alert when a metric exceeds a threshold?
Let’s say I’m using cpu-pcnt-usage-metrics.rb to push metrics into Graphite. But I also want to generate a warning when CPU idle is below 50%, and a critical when below 25% (fictional values).
One way to do it, I could also enable check-cpu.rb as a regular check and set thresholds there. But this seems wasteful - I am collecting CPU data already with the other plugin, there’s no need to increase the load everywhere to collect the same thing twice.
Another way, I could use check-data.rb, go into Graphite, retrieve the data I’ve collected with the metric plugin, and set thresholds there. This seems unnecessarily complicated, in a way. Also, now I depend on Graphite to generate my alerts, so if something happens to Graphite, I will not receive much-needed alerts. If past experience is any indication, things tend to break exactly when you need them most. 
Is there a third way to do this, that uses Sensu exclusively, without redundant data collection? check-aggregate.rb seems a bit close to what I need, but I don’t actually need aggregates, only individual checks, and also I do need to dump that data into Graphite eventually.
If I could only insert a check in the “pipeline” while the data is being routed from sensu-client to Graphite - that would be perfect.
Okay, I’m new to Sensu, please bear with me if I’m talking nonsense.
My understanding is that a handler will fire every time it’s called. If it’s the basic email handler, it will email whatever I give it. In order to get the behavior I’m seeking, I’ll have to write some kind of custom handler, that takes the data, processes it, looks at the thresholds I defined in the handler config, and only reacts when thresholds are being crossed. Basically, I would have to simulate a check with a custom-written handler, essentially replicating the check logic. It’s probably doable, but it would take some effort. I was hoping to re-use existing components.
Any chance I could accomplish what I want via remediation? Although, again, that seems geared towards checks, not metrics - it’s triggered by checks severity.
http://dev.nuclearrooster.com/2013/07/27/remediation-with-sensu/
···
On Wednesday, September 24, 2014 9:06:47 PM UTC-7, Fred Licht wrote:
You can have your check send its data to multiple handlers though they need to receive the same ‘type’ (Pipe, metric, etc) . One handler push the data to Graphite, a second process your CPU thresholds.
“handlers”: [“graphite_tcp”,“CPU_check”]
-Fred
On Tuesday, September 23, 2014 6:55:05 PM UTC-7, Florin Andrei wrote:
Is there a way to generate an alert when a metric exceeds a threshold?
Let’s say I’m using cpu-pcnt-usage-metrics.rb to push metrics into Graphite. But I also want to generate a warning when CPU idle is below 50%, and a critical when below 25% (fictional values).
One way to do it, I could also enable check-cpu.rb as a regular check and set thresholds there. But this seems wasteful - I am collecting CPU data already with the other plugin, there’s no need to increase the load everywhere to collect the same thing twice.
Another way, I could use check-data.rb, go into Graphite, retrieve the data I’ve collected with the metric plugin, and set thresholds there. This seems unnecessarily complicated, in a way. Also, now I depend on Graphite to generate my alerts, so if something happens to Graphite, I will not receive much-needed alerts. If past experience is any indication, things tend to break exactly when you need them most.
Is there a third way to do this, that uses Sensu exclusively, without redundant data collection? check-aggregate.rb seems a bit close to what I need, but I don’t actually need aggregates, only individual checks, and also I do need to dump that data into Graphite eventually.
If I could only insert a check in the “pipeline” while the data is being routed from sensu-client to Graphite - that would be perfect.
Hi All,
I also have similar issue.Could you please share steps you had done to fix.
···
On Thursday, September 25, 2014 at 11:19:22 PM UTC+5:30, Florin Andrei wrote:
Okay, I’m new to Sensu, please bear with me if I’m talking nonsense.
My understanding is that a handler will fire every time it’s called. If it’s the basic email handler, it will email whatever I give it. In order to get the behavior I’m seeking, I’ll have to write some kind of custom handler, that takes the data, processes it, looks at the thresholds I defined in the handler config, and only reacts when thresholds are being crossed. Basically, I would have to simulate a check with a custom-written handler, essentially replicating the check logic. It’s probably doable, but it would take some effort. I was hoping to re-use existing components.
Any chance I could accomplish what I want via remediation? Although, again, that seems geared towards checks, not metrics - it’s triggered by checks severity.
http://dev.nuclearrooster.com/2013/07/27/remediation-with-sensu/
On Wednesday, September 24, 2014 9:06:47 PM UTC-7, Fred Licht wrote:
You can have your check send its data to multiple handlers though they need to receive the same ‘type’ (Pipe, metric, etc) . One handler push the data to Graphite, a second process your CPU thresholds.
“handlers”: [“graphite_tcp”,“CPU_check”]
-Fred
On Tuesday, September 23, 2014 6:55:05 PM UTC-7, Florin Andrei wrote:
Is there a way to generate an alert when a metric exceeds a threshold?
Let’s say I’m using cpu-pcnt-usage-metrics.rb to push metrics into Graphite. But I also want to generate a warning when CPU idle is below 50%, and a critical when below 25% (fictional values).
One way to do it, I could also enable check-cpu.rb as a regular check and set thresholds there. But this seems wasteful - I am collecting CPU data already with the other plugin, there’s no need to increase the load everywhere to collect the same thing twice.
Another way, I could use check-data.rb, go into Graphite, retrieve the data I’ve collected with the metric plugin, and set thresholds there. This seems unnecessarily complicated, in a way. Also, now I depend on Graphite to generate my alerts, so if something happens to Graphite, I will not receive much-needed alerts. If past experience is any indication, things tend to break exactly when you need them most.
Is there a third way to do this, that uses Sensu exclusively, without redundant data collection? check-aggregate.rb seems a bit close to what I need, but I don’t actually need aggregates, only individual checks, and also I do need to dump that data into Graphite eventually.
If I could only insert a check in the “pipeline” while the data is being routed from sensu-client to Graphite - that would be perfect.