Sensu 0.29 occurences/refresh logic

Hi everyone!

I see that after upgrading sensu to sensu-0.29.0-11 the logic of occurences/refresh filtering is not working as expected before. Here is example of the check defenition:

“checks”: {

“some_port_check”: {

“command”: “check-ports.rb -H somehost -p 443 -P tcp”,

“interval”: 60,

“refresh”: 1800,

“occurrences”: 5,

“subscribers”: [ “infra_prod” ],

“source”: “source_name”,

“handlers”: [ “slack_handler_alerts_sensu”, “opsgenie” ],

},

From logs I see that alert is handled on the first occurence but I expect that only after 5 minutes:

{“timestamp”:“2017-08-18T01:14:30.713862+0300”,“level”:“info”,“message”:“processing event”,“event”:{“client”:{“name”:"",“address”:“unknown”,“subscriptions”:,“keepalives”:false,“version”:“0.25.5”},“check”:{“command”:“check-ports.rb -H somehost -p 443 -P tcp”,“interval”:60,“refresh”:1800,“occurrences”:5,“subscribers”:[“infra_prod_sensu_masters”],“source”:“source_name”,“handlers”:[“slack_handler_alerts_sensu”,“opsgenie”],“name”:“some_port_check”,“issued”:1503008040,“executed”:1503008040,“duration”:30.066,“output”:“CheckPort CRITICAL: Connection or read timed out (somehost:443)\n”,“status”:2,“type”:“standard”,“origin”:“sensu-server”,“history”:[“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“2”],“total_state_change”:6},“occurrences”:1,“occurrences_watermark”:1,“action”:“create”,“timestamp”:1503008070,“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”,“last_ok”:1503008070,“last_state_change”:1503008070,“silenced”:false,“silenced_by”:}}

{“timestamp”:“2017-08-18T01:14:31.254429+0300”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/handler-opsgenie.rb",“severities”:[“critical”,“ok”],“name”:“opsgenie”},“event”:{“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”},“output”:[“opsgenie – Created incident – somehost\n”]}

Any help will be appreciated because alerts noise got higher since we upgraded sensu.

Thanks,

Alexey

0.29 removed the deprecated filtering API:
https://sensuapp.org/docs/0.29/overview/changelog.html#core-v0-29-0-changes

You can get the old functionality by enabling this filter (referenced in the changelog):

https://github.com/sensu-extensions/sensu-extensions-occurrences

···

On Fri, Aug 18, 2017 at 1:29 AM, Алексей Максимов neizmirasego@gmail.com wrote:

Hi everyone!

I see that after upgrading sensu to sensu-0.29.0-11 the logic of occurences/refresh filtering is not working as expected before. Here is example of the check defenition:

“checks”: {

“some_port_check”: {

“command”: “check-ports.rb -H somehost -p 443 -P tcp”,

“interval”: 60,

“refresh”: 1800,

“occurrences”: 5,

“subscribers”: [ “infra_prod” ],

“source”: “source_name”,

“handlers”: [ “slack_handler_alerts_sensu”, “opsgenie” ],

},

From logs I see that alert is handled on the first occurence but I expect that only after 5 minutes:

{“timestamp”:“2017-08-18T01:14:30.713862+0300”,“level”:“info”,“message”:“processing event”,“event”:{“client”:{“name”:"",“address”:“unknown”,“subscriptions”:,“keepalives”:false,“version”:“0.25.5”},“check”:{“command”:“check-ports.rb -H somehost -p 443 -P tcp”,“interval”:60,“refresh”:1800,“occurrences”:5,“subscribers”:[“infra_prod_sensu_masters”],“source”:“source_name”,“handlers”:[“slack_handler_alerts_sensu”,“opsgenie”],“name”:“some_port_check”,“issued”:1503008040,“executed”:1503008040,“duration”:30.066,“output”:“CheckPort CRITICAL: Connection or read timed out (somehost:443)\n”,“status”:2,“type”:“standard”,“origin”:“sensu-server”,“history”:[“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“2”],“total_state_change”:6},“occurrences”:1,“occurrences_watermark”:1,“action”:“create”,“timestamp”:1503008070,“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”,“last_ok”:1503008070,“last_state_change”:1503008070,“silenced”:false,“silenced_by”:}}

{“timestamp”:“2017-08-18T01:14:31.254429+0300”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/handler-opsgenie.rb",“severities”:[“critical”,“ok”],“name”:“opsgenie”},“event”:{“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”},“output”:[“opsgenie – Created incident – somehost\n”]}

Any help will be appreciated because alerts noise got higher since we upgraded sensu.

Thanks,

Alexey

Kyle, thanks for the answer and links. As far as I understand the only thing I should do is just add

"filters": ["occurrences"]
property to handler defenitions.
I will defenitely try it on Monday and write the results here =)

Thanks,
Alexey

···

пятница, 18 августа 2017 г., 17:49:18 UTC+3 пользователь Kyle Anderson написал:

0.29 removed the deprecated filtering API:
https://sensuapp.org/docs/0.29/overview/changelog.html#core-v0-29-0-changes

You can get the old functionality by enabling this filter (referenced in the changelog):

https://github.com/sensu-extensions/sensu-extensions-occurrences

On Fri, Aug 18, 2017 at 1:29 AM, Алексей Максимов neizmi...@gmail.com wrote:

Hi everyone!

I see that after upgrading sensu to sensu-0.29.0-11 the logic of occurences/refresh filtering is not working as expected before. Here is example of the check defenition:

“checks”: {

“some_port_check”: {

“command”: “check-ports.rb -H somehost -p 443 -P tcp”,

“interval”: 60,

“refresh”: 1800,

“occurrences”: 5,

“subscribers”: [ “infra_prod” ],

“source”: “source_name”,

“handlers”: [ “slack_handler_alerts_sensu”, “opsgenie” ],

},

From logs I see that alert is handled on the first occurence but I expect that only after 5 minutes:

{“timestamp”:“2017-08-18T01:14:30.713862+0300”,“level”:“info”,“message”:“processing event”,“event”:{“client”:{“name”:"",“address”:“unknown”,“subscriptions”:,“keepalives”:false,“version”:“0.25.5”},“check”:{“command”:“check-ports.rb -H somehost -p 443 -P tcp”,“interval”:60,“refresh”:1800,“occurrences”:5,“subscribers”:[“infra_prod_sensu_masters”],“source”:“source_name”,“handlers”:[“slack_handler_alerts_sensu”,“opsgenie”],“name”:“some_port_check”,“issued”:1503008040,“executed”:1503008040,“duration”:30.066,“output”:“CheckPort CRITICAL: Connection or read timed out (somehost:443)\n”,“status”:2,“type”:“standard”,“origin”:“sensu-server”,“history”:[“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“2”],“total_state_change”:6},“occurrences”:1,“occurrences_watermark”:1,“action”:“create”,“timestamp”:1503008070,“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”,“last_ok”:1503008070,“last_state_change”:1503008070,“silenced”:false,“silenced_by”:}}

{“timestamp”:“2017-08-18T01:14:31.254429+0300”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/handler-opsgenie.rb",“severities”:[“critical”,“ok”],“name”:“opsgenie”},“event”:{“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”},“output”:[“opsgenie – Created incident – somehost\n”]}

Any help will be appreciated because alerts noise got higher since we upgraded sensu.

Thanks,

Alexey

After adding filter field to the handler defenition everything works fine!

Thanks for help again,
Alexey

···

пятница, 18 августа 2017 г., 11:29:48 UTC+3 пользователь Алексей Максимов написал:

Hi everyone!

I see that after upgrading sensu to sensu-0.29.0-11 the logic of occurences/refresh filtering is not working as expected before. Here is example of the check defenition:

“checks”: {

“some_port_check”: {

“command”: “check-ports.rb -H somehost -p 443 -P tcp”,

“interval”: 60,

“refresh”: 1800,

“occurrences”: 5,

“subscribers”: [ “infra_prod” ],

“source”: “source_name”,

“handlers”: [ “slack_handler_alerts_sensu”, “opsgenie” ],

},

From logs I see that alert is handled on the first occurence but I expect that only after 5 minutes:

{“timestamp”:“2017-08-18T01:14:30.713862+0300”,“level”:“info”,“message”:“processing event”,“event”:{“client”:{“name”:"",“address”:“unknown”,“subscriptions”:,“keepalives”:false,“version”:“0.25.5”},“check”:{“command”:“check-ports.rb -H somehost -p 443 -P tcp”,“interval”:60,“refresh”:1800,“occurrences”:5,“subscribers”:[“infra_prod_sensu_masters”],“source”:“source_name”,“handlers”:[“slack_handler_alerts_sensu”,“opsgenie”],“name”:“some_port_check”,“issued”:1503008040,“executed”:1503008040,“duration”:30.066,“output”:“CheckPort CRITICAL: Connection or read timed out (somehost:443)\n”,“status”:2,“type”:“standard”,“origin”:“sensu-server”,“history”:[“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“2”],“total_state_change”:6},“occurrences”:1,“occurrences_watermark”:1,“action”:“create”,“timestamp”:1503008070,“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”,“last_ok”:1503008070,“last_state_change”:1503008070,“silenced”:false,“silenced_by”:}}

{“timestamp”:“2017-08-18T01:14:31.254429+0300”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/handler-opsgenie.rb",“severities”:[“critical”,“ok”],“name”:“opsgenie”},“event”:{“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”},“output”:[“opsgenie – Created incident – somehost\n”]}

Any help will be appreciated because alerts noise got higher since we upgraded sensu.

Thanks,

Alexey