Sensu OKs are filtered out

Sensu Classic (EOL)
1

Hi All,

We use filters by occurrences in order to filter unnecessary alerts. But the problem is that I don’t receive OK messages in this case. My assumption is that ok messages are filteres as well as other by occurrence.
Here is the example of resolve message from logs:

{“timestamp”:“2016-09-30T11:12:06.381458+0300”,“level”:“info”,“message”:“event was filtered”,“handler”:{“type”:“pipe”,“command”:“handler-slack.rb -j slack_jenkins”,“channel”:“alerts-sensu”,“severites”:[“unknown”,“warning”,“critical”],“filters”:[“recurrence_5_1440”],“name”:“slack_jenkins_handler”},“event”:{“client”:{“name”:“node009-jenkins-prod”,“address”:“192.168.81.2”,“subscriptions”:[“common”,“infra”,“infra_prod”],“keepalive”:{“handlers”:[“slack”],“interval”:60,“refresh”:60},“version”:“0.25.2”,“timestamp”:1475223115},“check”:{“command”:"/etc/sensu/plugins/check-jenkins-job-status.rb -u http://node009-jenkins-prod:8080 -j ansible-tests -l jenkins_api -p /opt/sensu/.sensu_jenkins_api",“interval”:60,“handlers”:[“slack_jenkins_handler”],“source”:“node009-jenkins-prod”,“subscribers”:[“infra_prod_sensu_masters”],“refresh”:60,“name”:“jenkins_job_ansible_tests”,“issued”:1475223125,“executed”:1475223125,“duration”:0.865,“output”:“JenkinsJobChecker OK: All queried jobs reports success\n”,“status”:0,“type”:“standard”,“origin”:“sensu-server001-prod”,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“0”],“total_state_change”:6},“occurrences”:45,“action”:“resolve”,“timestamp”:1475223126,“id”:“3f2d92b3-c4d7-4459-81da-7c5e4c69d6c0”,“last_state_change”:1475223126,“last_ok”:1475223126}}

“recurrence_5_1440”: {
“attributes”: {
“occurrences”: “eval: value == 5 || value % 1440 == 0”
}

Is there any workaround to filter unnecessary alerts but still receive OK messages?

Thanks,
Alexey

2

See https://sensuapp.org/docs/0.26/reference/checks.html#check-attributes section “type”.

···

On Friday, September 30, 2016 at 3:52:03 PM UTC+2, Алексей Максимов wrote:

Hi All,

We use filters by occurrences in order to filter unnecessary alerts. But the problem is that I don’t receive OK messages in this case. My assumption is that ok messages are filteres as well as other by occurrence.
Here is the example of resolve message from logs:

{“timestamp”:“2016-09-30T11:12:06.381458+0300”,“level”:“info”,“message”:“event was filtered”,“handler”:{“type”:“pipe”,“command”:“handler-slack.rb -j slack_jenkins”,“channel”:“alerts-sensu”,“severites”:[“unknown”,“warning”,“critical”],“filters”:[“recurrence_5_1440”],“name”:“slack_jenkins_handler”},“event”:{“client”:{“name”:“node009-jenkins-prod”,“address”:“192.168.81.2”,“subscriptions”:[“common”,“infra”,“infra_prod”],“keepalive”:{“handlers”:[“slack”],“interval”:60,“refresh”:60},“version”:“0.25.2”,“timestamp”:1475223115},“check”:{“command”:“/etc/sensu/plugins/check-jenkins-job-status.rb -u http://node009-jenkins-prod:8080 -j ansible-tests -l jenkins_api -p /opt/sensu/.sensu_jenkins_api”,“interval”:60,“handlers”:[“slack_jenkins_handler”],“source”:“node009-jenkins-prod”,“subscribers”:[“infra_prod_sensu_masters”],“refresh”:60,“name”:“jenkins_job_ansible_tests”,“issued”:1475223125,“executed”:1475223125,“duration”:0.865,“output”:“JenkinsJobChecker OK: All queried jobs reports success\n”,“status”:0,“type”:“standard”,“origin”:“sensu-server001-prod”,“history”:[“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“0”],“total_state_change”:6},“occurrences”:45,“action”:“resolve”,“timestamp”:1475223126,“id”:“3f2d92b3-c4d7-4459-81da-7c5e4c69d6c0”,“last_state_change”:1475223126,“last_ok”:1475223126}}

“recurrence_5_1440”: {
“attributes”: {
“occurrences”: “eval: value == 5 || value % 1440 == 0”
}

Is there any workaround to filter unnecessary alerts but still receive OK messages?

Thanks,
Alexey

3

Thanks for reply,

I don’t think that this can be solution here. I need to get OK only once when issue is resolved.
If I set metric type I believe I will get every OK message in slack

···

воскресенье, 2 октября 2016 г., 20:00:20 UTC+3 пользователь Alexander Skiba написал:

See https://sensuapp.org/docs/0.26/reference/checks.html#check-attributes section “type”.

4

on resolve event I try to check the occurrences value is equal or bigger than the predefined value (in your example occurrences = 45).

“recurrence_5_1440”: {

“attributes”: {

"occurrences": "eval: value == 5 || value % 1440 == 0  **|| ':::action:::' == 'resolve' && value >= 5**"

},

}

But this doesn’t seem to work for me, see “Resolve events are being filtered for an unknown reason” thread.

···

On Thursday, October 13, 2016 at 11:41:38 AM UTC+3, Алексей Максимов wrote:

Thanks for reply,

I don’t think that this can be solution here. I need to get OK only once when issue is resolved.
If I set metric type I believe I will get every OK message in slack

воскресенье, 2 октября 2016 г., 20:00:20 UTC+3 пользователь Alexander Skiba написал:

See https://sensuapp.org/docs/0.26/reference/checks.html#check-attributes section “type”.

5

Hi! Thanks, I have checked this thread and tried this solution.
Looks like the value cannot be used in this case because this is not number of “resolve” messages. This is the number of last critical or warning messages even if event already resolved.

For example
history":[“0”,“0”,“0”,“0”,“0”,“1”,“1”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“2”,“0”],“total_state_change”:15},“occurrences”:13,“action”:“resolve”

So I am still unable to find appropriate solution.

···

четверг, 13 октября 2016 г., 14:35:46 UTC+3 пользователь Oded Ben Ozer написал:

on resolve event I try to check the occurrences value is equal or bigger than the predefined value (in your example occurrences = 45).

“recurrence_5_1440”: {

“attributes”: {

"occurrences": "eval: value == 5 || value % 1440 == 0  **|| ':::action:::' == 'resolve' && value >= 5**"

},

}

But this doesn’t seem to work for me, see “Resolve events are being filtered for an unknown reason” thread.

On Thursday, October 13, 2016 at 11:41:38 AM UTC+3, Алексей Максимов wrote:

Thanks for reply,

I don’t think that this can be solution here. I need to get OK only once when issue is resolved.
If I set metric type I believe I will get every OK message in slack

воскресенье, 2 октября 2016 г., 20:00:20 UTC+3 пользователь Alexander Skiba написал:

See https://sensuapp.org/docs/0.26/reference/checks.html#check-attributes section “type”.

6

try using occurrences_watermark (requires newer sensu)

Note, solution to same problem in the occurrences filter extension: Updating logic to consider occurrence_watermark when handling resolve events by cwjohnston · Pull Request #2 · sensu/sensu-extensions-occurrences · GitHub

or depending on what you’re doing, just try using the above extension.

···

On Oct 14, 2016, at 5:34 AM, Алексей Максимов <neizmirasego@gmail.com> wrote:

Hi! Thanks, I have checked this thread and tried this solution.
Looks like the value cannot be used in this case because this is not number of "resolve" messages. This is the number of last critical or warning messages even if event already resolved.

For example
history":["0","0","0","0","0","1","1","2","2","2","2","2","2","2","2","2","2","2","2","2","0"],"total_state_change":15},"occurrences":13,"action":"resolve"

So I am still unable to find appropriate solution.

четверг, 13 октября 2016 г., 14:35:46 UTC+3 пользователь Oded Ben Ozer написал:
on resolve event I try to check the occurrences value is equal or bigger than the predefined value (in your example occurrences = 45).

"recurrence_5_1440": {
  "attributes": {
    "occurrences": "eval: value == 5 || value % 1440 == 0 || ':::action:::' == 'resolve' && value >= 5"
},
}

But this doesn't seem to work for me, see "Resolve events are being filtered for an unknown reason" thread.

On Thursday, October 13, 2016 at 11:41:38 AM UTC+3, Алексей Максимов wrote:
Thanks for reply,

I don't think that this can be solution here. I need to get OK only once when issue is resolved.
If I set metric type I believe I will get every OK message in slack

воскресенье, 2 октября 2016 г., 20:00:20 UTC+3 пользователь Alexander Skiba написал:
See https://sensuapp.org/docs/0.26/reference/checks.html#check-attributes section "type".