# check on metric threshold, generate alert

**URL:** <https://discourse.sensu.io/t/check-on-metric-threshold-generate-alert/262>\
**Category:** Sensu Classic (EOL)\
**Created:** [September 24, 2014, 1:55am UTC](https://discourse.sensu.io/t/check-on-metric-threshold-generate-alert/262 "2014-09-24T01:55:04Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Florin\_Andrei](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/florin_andrei/32/674_2.png) [@Florin\_Andrei](https://discourse.sensu.io/u/Florin_Andrei)\
**Post date:** [September 24, 2014, 1:55am UTC](https://discourse.sensu.io/t/check-on-metric-threshold-generate-alert/262/1 "2014-09-24T01:55:04Z")

</div>

Is there a way to generate an alert when a metric exceeds a threshold?

Let’s say I’m using cpu-pcnt-usage-metrics.rb to push metrics into Graphite. But I also want to generate a warning when CPU idle is below 50%, and a critical when below 25% (fictional values).

One way to do it, I could also enable check-cpu.rb as a regular check and set thresholds there. But this seems wasteful - I am collecting CPU data already with the other plugin, there’s no need to increase the load everywhere to collect the same thing twice.

Another way, I could use check-data.rb, go into Graphite, retrieve the data I’ve collected with the metric plugin, and set thresholds there. This seems unnecessarily complicated, in a way. Also, now I depend on Graphite to generate my alerts, so if something happens to Graphite, I will not receive much-needed alerts. If past experience is any indication, things tend to break exactly when you need them most. 🙂

Is there a third way to do this, that uses Sensu exclusively, without redundant data collection? check-aggregate.rb seems a bit close to what I need, but I don’t actually need aggregates, only individual checks, and also I do need to dump that data into Graphite eventually.

If I could only insert a check in the “pipeline” while the data is being routed from sensu-client to Graphite - that would be perfect.

---

<div class="post-metadata">

**Author:** ![Fred\_Licht](https://avatars.discourse-cdn.com/v4/letter/f/85e7bf/32.png) [@Fred\_Licht](https://discourse.sensu.io/u/Fred_Licht)\
**Post date:** [September 25, 2014, 4:06am UTC](https://discourse.sensu.io/t/check-on-metric-threshold-generate-alert/262/2 "2014-09-25T04:06:46Z")

</div>

You can have your check send its data to multiple handlers though they need to receive the same ‘type’ (Pipe, metric, etc) . One handler push the data to Graphite, a second process your CPU thresholds.

“handlers”: [“graphite\_tcp”,“CPU\_check”]

-Fred

> **···**
>
> On Tuesday, September 23, 2014 6:55:05 PM UTC-7, Florin Andrei wrote:
> 
> > Is there a way to generate an alert when a metric exceeds a threshold?
> 
> > Let’s say I’m using cpu-pcnt-usage-metrics.rb to push metrics into Graphite. But I also want to generate a warning when CPU idle is below 50%, and a critical when below 25% (fictional values).
> 
> > 
> 
> > One way to do it, I could also enable check-cpu.rb as a regular check and set thresholds there. But this seems wasteful - I am collecting CPU data already with the other plugin, there’s no need to increase the load everywhere to collect the same thing twice.
> 
> > 
> 
> > Another way, I could use check-data.rb, go into Graphite, retrieve the data I’ve collected with the metric plugin, and set thresholds there. This seems unnecessarily complicated, in a way. Also, now I depend on Graphite to generate my alerts, so if something happens to Graphite, I will not receive much-needed alerts. If past experience is any indication, things tend to break exactly when you need them most. 🙂
> 
> > 
> 
> > Is there a third way to do this, that uses Sensu exclusively, without redundant data collection? check-aggregate.rb seems a bit close to what I need, but I don’t actually need aggregates, only individual checks, and also I do need to dump that data into Graphite eventually.
> 
> > 
> 
> > If I could only insert a check in the “pipeline” while the data is being routed from sensu-client to Graphite - that would be perfect.

---

<div class="post-metadata">

**Author:** ![Florin\_Andrei](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/florin_andrei/32/674_2.png) [@Florin\_Andrei](https://discourse.sensu.io/u/Florin_Andrei)\
**Post date:** [September 25, 2014, 5:49pm UTC](https://discourse.sensu.io/t/check-on-metric-threshold-generate-alert/262/3 "2014-09-25T17:49:22Z")

</div>

Okay, I’m new to Sensu, please bear with me if I’m talking nonsense.

My understanding is that a handler will fire every time it’s called. If it’s the basic email handler, it will email whatever I give it. In order to get the behavior I’m seeking, I’ll have to write some kind of custom handler, that takes the data, processes it, looks at the thresholds I defined in the handler config, and only reacts when thresholds are being crossed. Basically, I would have to simulate a check with a custom-written handler, essentially replicating the check logic. It’s probably doable, but it would take some effort. I was hoping to re-use existing components.

Any chance I could accomplish what I want via remediation? Although, again, that seems geared towards checks, not metrics - it’s triggered by checks severity.

[http://dev.nuclearrooster.com/2013/07/27/remediation-with-sensu/](http://dev.nuclearrooster.com/2013/07/27/remediation-with-sensu/)

> **···**
>
> On Wednesday, September 24, 2014 9:06:47 PM UTC-7, Fred Licht wrote:
> 
> > You can have your check send its data to multiple handlers though they need to receive the same ‘type’ (Pipe, metric, etc) . One handler push the data to Graphite, a second process your CPU thresholds.
> 
> > 
> 
> > “handlers”: [“graphite\_tcp”,“CPU\_check”]
> 
> > 
> 
> > -Fred
> 
> > 
> 
> > On Tuesday, September 23, 2014 6:55:05 PM UTC-7, Florin Andrei wrote:
> > 
> > > Is there a way to generate an alert when a metric exceeds a threshold?
> 
> > > Let’s say I’m using cpu-pcnt-usage-metrics.rb to push metrics into Graphite. But I also want to generate a warning when CPU idle is below 50%, and a critical when below 25% (fictional values).
> 
> > >
> 
> > > One way to do it, I could also enable check-cpu.rb as a regular check and set thresholds there. But this seems wasteful - I am collecting CPU data already with the other plugin, there’s no need to increase the load everywhere to collect the same thing twice.
> 
> > >
> 
> > > Another way, I could use check-data.rb, go into Graphite, retrieve the data I’ve collected with the metric plugin, and set thresholds there. This seems unnecessarily complicated, in a way. Also, now I depend on Graphite to generate my alerts, so if something happens to Graphite, I will not receive much-needed alerts. If past experience is any indication, things tend to break exactly when you need them most. 🙂
> 
> > >
> 
> > > Is there a third way to do this, that uses Sensu exclusively, without redundant data collection? check-aggregate.rb seems a bit close to what I need, but I don’t actually need aggregates, only individual checks, and also I do need to dump that data into Graphite eventually.
> 
> > >
> 
> > > If I could only insert a check in the “pipeline” while the data is being routed from sensu-client to Graphite - that would be perfect.

---

<div class="post-metadata">

**Author:** ![Hari\_krishna\_kumar](https://avatars.discourse-cdn.com/v4/letter/h/a9a28c/32.png) [@Hari\_krishna\_kumar](https://discourse.sensu.io/u/Hari_krishna_kumar)\
**Post date:** [February 12, 2016, 7:35am UTC](https://discourse.sensu.io/t/check-on-metric-threshold-generate-alert/262/4 "2016-02-12T07:35:39Z")

</div>

Hi All,

I also have similar issue.Could you please share steps you had done to fix.

> **···**
>
> On Thursday, September 25, 2014 at 11:19:22 PM UTC+5:30, Florin Andrei wrote:
> 
> > Okay, I’m new to Sensu, please bear with me if I’m talking nonsense.
> 
> > 
> 
> > My understanding is that a handler will fire every time it’s called. If it’s the basic email handler, it will email whatever I give it. In order to get the behavior I’m seeking, I’ll have to write some kind of custom handler, that takes the data, processes it, looks at the thresholds I defined in the handler config, and only reacts when thresholds are being crossed. Basically, I would have to simulate a check with a custom-written handler, essentially replicating the check logic. It’s probably doable, but it would take some effort. I was hoping to re-use existing components.
> 
> > 
> 
> > Any chance I could accomplish what I want via remediation? Although, again, that seems geared towards checks, not metrics - it’s triggered by checks severity.
> 
> > 
> 
> > [http://dev.nuclearrooster.com/2013/07/27/remediation-with-sensu/](http://dev.nuclearrooster.com/2013/07/27/remediation-with-sensu/)
> 
> > 
> 
> > On Wednesday, September 24, 2014 9:06:47 PM UTC-7, Fred Licht wrote:
> > 
> > > You can have your check send its data to multiple handlers though they need to receive the same ‘type’ (Pipe, metric, etc) . One handler push the data to Graphite, a second process your CPU thresholds.
> 
> > >
> 
> > > “handlers”: [“graphite\_tcp”,“CPU\_check”]
> 
> > >
> 
> > > -Fred
> 
> > >
> 
> > > On Tuesday, September 23, 2014 6:55:05 PM UTC-7, Florin Andrei wrote:
> > > 
> > > > Is there a way to generate an alert when a metric exceeds a threshold?
> 
> > > > Let’s say I’m using cpu-pcnt-usage-metrics.rb to push metrics into Graphite. But I also want to generate a warning when CPU idle is below 50%, and a critical when below 25% (fictional values).
> 
> > > >
> 
> > > > One way to do it, I could also enable check-cpu.rb as a regular check and set thresholds there. But this seems wasteful - I am collecting CPU data already with the other plugin, there’s no need to increase the load everywhere to collect the same thing twice.
> 
> > > >
> 
> > > > Another way, I could use check-data.rb, go into Graphite, retrieve the data I’ve collected with the metric plugin, and set thresholds there. This seems unnecessarily complicated, in a way. Also, now I depend on Graphite to generate my alerts, so if something happens to Graphite, I will not receive much-needed alerts. If past experience is any indication, things tend to break exactly when you need them most. 🙂
> 
> > > >
> 
> > > > Is there a third way to do this, that uses Sensu exclusively, without redundant data collection? check-aggregate.rb seems a bit close to what I need, but I don’t actually need aggregates, only individual checks, and also I do need to dump that data into Graphite eventually.
> 
> > > >
> 
> > > > If I could only insert a check in the “pipeline” while the data is being routed from sensu-client to Graphite - that would be perfect.