Sensu 0.29 occurences/refresh logic

Sensu Classic (EOL)
1

Hi everyone!

I see that after upgrading sensu to sensu-0.29.0-11 the logic of occurences/refresh filtering is not working as expected before. Here is example of the check defenition:

“checks”: {

“some_port_check”: {

“command”: “check-ports.rb -H somehost -p 443 -P tcp”,

“interval”: 60,

“refresh”: 1800,

“occurrences”: 5,

“subscribers”: [ “infra_prod” ],

“source”: “source_name”,

“handlers”: [ “slack_handler_alerts_sensu”, “opsgenie” ],

},

From logs I see that alert is handled on the first occurence but I expect that only after 5 minutes:

{“timestamp”:“2017-08-18T01:14:30.713862+0300”,“level”:“info”,“message”:“processing event”,“event”:{“client”:{“name”:"",“address”:“unknown”,“subscriptions”:,“keepalives”:false,“version”:“0.25.5”},“check”:{“command”:“check-ports.rb -H somehost -p 443 -P tcp”,“interval”:60,“refresh”:1800,“occurrences”:5,“subscribers”:[“infra_prod_sensu_masters”],“source”:“source_name”,“handlers”:[“slack_handler_alerts_sensu”,“opsgenie”],“name”:“some_port_check”,“issued”:1503008040,“executed”:1503008040,“duration”:30.066,“output”:“CheckPort CRITICAL: Connection or read timed out (somehost:443)\n”,“status”:2,“type”:“standard”,“origin”:“sensu-server”,“history”:[“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“2”],“total_state_change”:6},“occurrences”:1,“occurrences_watermark”:1,“action”:“create”,“timestamp”:1503008070,“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”,“last_ok”:1503008070,“last_state_change”:1503008070,“silenced”:false,“silenced_by”:}}

{“timestamp”:“2017-08-18T01:14:31.254429+0300”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:"/etc/sensu/handlers/handler-opsgenie.rb",“severities”:[“critical”,“ok”],“name”:“opsgenie”},“event”:{“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”},“output”:[“opsgenie – Created incident – somehost\n”]}

Any help will be appreciated because alerts noise got higher since we upgraded sensu.

Thanks,

Alexey

2

0.29 removed the deprecated filtering API:
https://sensuapp.org/docs/0.29/overview/changelog.html#core-v0-29-0-changes

You can get the old functionality by enabling this filter (referenced in the changelog):

https://github.com/sensu-extensions/sensu-extensions-occurrences

···

On Fri, Aug 18, 2017 at 1:29 AM, Алексей Максимов neizmirasego@gmail.com wrote:

Hi everyone!

I see that after upgrading sensu to sensu-0.29.0-11 the logic of occurences/refresh filtering is not working as expected before. Here is example of the check defenition:

“checks”: {

“some_port_check”: {

“command”: “check-ports.rb -H somehost -p 443 -P tcp”,

“interval”: 60,

“refresh”: 1800,

“occurrences”: 5,

“subscribers”: [ “infra_prod” ],

“source”: “source_name”,

“handlers”: [ “slack_handler_alerts_sensu”, “opsgenie” ],

},

From logs I see that alert is handled on the first occurence but I expect that only after 5 minutes:

{“timestamp”:“2017-08-18T01:14:30.713862+0300”,“level”:“info”,“message”:“processing event”,“event”:{“client”:{“name”:“”,“address”:“unknown”,“subscriptions”:,“keepalives”:false,“version”:“0.25.5”},“check”:{“command”:“check-ports.rb -H somehost -p 443 -P tcp”,“interval”:60,“refresh”:1800,“occurrences”:5,“subscribers”:[“infra_prod_sensu_masters”],“source”:“source_name”,“handlers”:[“slack_handler_alerts_sensu”,“opsgenie”],“name”:“some_port_check”,“issued”:1503008040,“executed”:1503008040,“duration”:30.066,“output”:“CheckPort CRITICAL: Connection or read timed out (somehost:443)\n”,“status”:2,“type”:“standard”,“origin”:“sensu-server”,“history”:[“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“2”],“total_state_change”:6},“occurrences”:1,“occurrences_watermark”:1,“action”:“create”,“timestamp”:1503008070,“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”,“last_ok”:1503008070,“last_state_change”:1503008070,“silenced”:false,“silenced_by”:}}

{“timestamp”:“2017-08-18T01:14:31.254429+0300”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“/etc/sensu/handlers/handler-opsgenie.rb”,“severities”:[“critical”,“ok”],“name”:“opsgenie”},“event”:{“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”},“output”:[“opsgenie – Created incident – somehost\n”]}

Any help will be appreciated because alerts noise got higher since we upgraded sensu.

Thanks,

Alexey

3

Kyle, thanks for the answer and links. As far as I understand the only thing I should do is just add

"filters": ["occurrences"]
property to handler defenitions.
I will defenitely try it on Monday and write the results here =)

Thanks,
Alexey
···

пятница, 18 августа 2017 г., 17:49:18 UTC+3 пользователь Kyle Anderson написал:

0.29 removed the deprecated filtering API:
https://sensuapp.org/docs/0.29/overview/changelog.html#core-v0-29-0-changes

You can get the old functionality by enabling this filter (referenced in the changelog):

https://github.com/sensu-extensions/sensu-extensions-occurrences

On Fri, Aug 18, 2017 at 1:29 AM, Алексей Максимов neizmi...@gmail.com wrote:

Hi everyone!

I see that after upgrading sensu to sensu-0.29.0-11 the logic of occurences/refresh filtering is not working as expected before. Here is example of the check defenition:

“checks”: {

“some_port_check”: {

“command”: “check-ports.rb -H somehost -p 443 -P tcp”,

“interval”: 60,

“refresh”: 1800,

“occurrences”: 5,

“subscribers”: [ “infra_prod” ],

“source”: “source_name”,

“handlers”: [ “slack_handler_alerts_sensu”, “opsgenie” ],

},

From logs I see that alert is handled on the first occurence but I expect that only after 5 minutes:

{“timestamp”:“2017-08-18T01:14:30.713862+0300”,“level”:“info”,“message”:“processing event”,“event”:{“client”:{“name”:“”,“address”:“unknown”,“subscriptions”:,“keepalives”:false,“version”:“0.25.5”},“check”:{“command”:“check-ports.rb -H somehost -p 443 -P tcp”,“interval”:60,“refresh”:1800,“occurrences”:5,“subscribers”:[“infra_prod_sensu_masters”],“source”:“source_name”,“handlers”:[“slack_handler_alerts_sensu”,“opsgenie”],“name”:“some_port_check”,“issued”:1503008040,“executed”:1503008040,“duration”:30.066,“output”:“CheckPort CRITICAL: Connection or read timed out (somehost:443)\n”,“status”:2,“type”:“standard”,“origin”:“sensu-server”,“history”:[“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“2”],“total_state_change”:6},“occurrences”:1,“occurrences_watermark”:1,“action”:“create”,“timestamp”:1503008070,“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”,“last_ok”:1503008070,“last_state_change”:1503008070,“silenced”:false,“silenced_by”:}}

{“timestamp”:“2017-08-18T01:14:31.254429+0300”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“/etc/sensu/handlers/handler-opsgenie.rb”,“severities”:[“critical”,“ok”],“name”:“opsgenie”},“event”:{“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”},“output”:[“opsgenie – Created incident – somehost\n”]}

Any help will be appreciated because alerts noise got higher since we upgraded sensu.

Thanks,

Alexey

4

After adding filter field to the handler defenition everything works fine!

Thanks for help again,
Alexey

···

пятница, 18 августа 2017 г., 11:29:48 UTC+3 пользователь Алексей Максимов написал:

Hi everyone!

I see that after upgrading sensu to sensu-0.29.0-11 the logic of occurences/refresh filtering is not working as expected before. Here is example of the check defenition:

“checks”: {

“some_port_check”: {

“command”: “check-ports.rb -H somehost -p 443 -P tcp”,

“interval”: 60,

“refresh”: 1800,

“occurrences”: 5,

“subscribers”: [ “infra_prod” ],

“source”: “source_name”,

“handlers”: [ “slack_handler_alerts_sensu”, “opsgenie” ],

},

From logs I see that alert is handled on the first occurence but I expect that only after 5 minutes:

{“timestamp”:“2017-08-18T01:14:30.713862+0300”,“level”:“info”,“message”:“processing event”,“event”:{“client”:{“name”:“”,“address”:“unknown”,“subscriptions”:,“keepalives”:false,“version”:“0.25.5”},“check”:{“command”:“check-ports.rb -H somehost -p 443 -P tcp”,“interval”:60,“refresh”:1800,“occurrences”:5,“subscribers”:[“infra_prod_sensu_masters”],“source”:“source_name”,“handlers”:[“slack_handler_alerts_sensu”,“opsgenie”],“name”:“some_port_check”,“issued”:1503008040,“executed”:1503008040,“duration”:30.066,“output”:“CheckPort CRITICAL: Connection or read timed out (somehost:443)\n”,“status”:2,“type”:“standard”,“origin”:“sensu-server”,“history”:[“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“0”,“2”],“total_state_change”:6},“occurrences”:1,“occurrences_watermark”:1,“action”:“create”,“timestamp”:1503008070,“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”,“last_ok”:1503008070,“last_state_change”:1503008070,“silenced”:false,“silenced_by”:}}

{“timestamp”:“2017-08-18T01:14:31.254429+0300”,“level”:“info”,“message”:“handler output”,“handler”:{“type”:“pipe”,“command”:“/etc/sensu/handlers/handler-opsgenie.rb”,“severities”:[“critical”,“ok”],“name”:“opsgenie”},“event”:{“id”:“9032e06d-2967-41ab-8934-7d9a5adf0ab8”},“output”:[“opsgenie – Created incident – somehost\n”]}

Any help will be appreciated because alerts noise got higher since we upgraded sensu.

Thanks,

Alexey