I have been trying to read through the documentation and posts and am not finding what I am looking for. I was hoping someone might be able to help figure out if what I am trying to do is even possible with Sensu at the moment.
I created a custom script that checks the RabbitMQ api for our consistent group of consumers, and reports back which server is missing.
Status: 0
No missing consumers found.
``
Status: 2
myserver1.fqdn is a missing consumer
``
Status: 2
myserver1.fqdn is a missing consumer
myserver4.fqdn is a missing consumer
myserver6.fqdn is a missing consumer
``
^ This checks the RabbitMQ api every 6 seconds. Above is sample output and the status code of the script.
I would like to make it so Sensu alerts (Slack handler) when it first recognizes a Consumer missing, then do not alert (Slack handler) again unless another Consumer becomes missing -OR- ‘X’ minutes have passed -OR- the check resolves.
On an earlier version of Sensu (<0.26), just setting “refresh”: ‘x’ would have the Slack handler alert every ‘x’ seconds. With a newer version of Sensu (1.4.3), the Slack handler alerts every 60 seconds while in a critical state.
I would like to make it so Sensu alerts (Slack handler) when it first recognizes a Consumer missing, then do not alert (Slack handler) again unless another Consumer becomes missing
It seems to me that this case could be solved for more directly by storing state local to the check, e.g. writing a temp file with a list of known-missing consumers. The [check-log.rb1 ](https://github.com/sensu-plugins/sensu-plugins-logs/blob/1.3.2/bin/check-log.rb)plugin as packaged by sensu-plugins-logs provides an example of this pattern – using a state file to track the byte position of the last read log line in a file.
On an earlier version of Sensu (<0.26), just setting “refresh”: ‘x’ would have the Slack handler alert every ‘x’ seconds. With a newer version of Sensu (1.4.3), the Slack handler alerts every 60 seconds while in a critical state.
Sounds like your Slack handler plugin is no longer applying occurrence filtering. Per the changelog for the sensu-plugins-slack handlers, version 3.0.0 should be used with the built-in occurrences filter in order to retain the old behavior with respect to “refresh” and “occurrences” attributes.
Hope these notes are helpful to you.
Best,
Cameron
···
On Monday, September 24, 2018 at 8:24:54 PM UTC-6, mil1i wrote:
I have been trying to read through the documentation and posts and am not finding what I am looking for. I was hoping someone might be able to help figure out if what I am trying to do is even possible with Sensu at the moment.
I created a custom script that checks the RabbitMQ api for our consistent group of consumers, and reports back which server is missing.
Status: 0
No missing consumers found.
``
Status: 2
myserver1.fqdn is a missing consumer
``
Status: 2
myserver1.fqdn is a missing consumer
myserver4.fqdn is a missing consumer
myserver6.fqdn is a missing consumer
``
^ This checks the RabbitMQ api every 6 seconds. Above is sample output and the status code of the script.
I would like to make it so Sensu alerts (Slack handler) when it first recognizes a Consumer missing, then do not alert (Slack handler) again unless another Consumer becomes missing -OR- ‘X’ minutes have passed -OR- the check resolves.
On an earlier version of Sensu (<0.26), just setting “refresh”: ‘x’ would have the Slack handler alert every ‘x’ seconds. With a newer version of Sensu (1.4.3), the Slack handler alerts every 60 seconds while in a critical state.
On an earlier version of Sensu (<0.26), just setting “refresh”: ‘x’ would have the Slack handler alert every ‘x’ seconds. With a newer version of Sensu (1.4.3), the Slack handler alerts every 60 seconds while in a critical state.
Sounds like your Slack handler plugin is no longer applying occurrence filtering. Per the changelog for the sensu-plugins-slack handlers , version 3.0.0 should be used with the built-in occurrences filter in order to retain the old behavior with respect to “refresh” and “occurrences” attributes.
I remembered reading this in the documentation after I posted it, it had just been a while since I setup this check and forgot about it! I did make that change and added occurrences into the slack handler filters. That did take care of the refresh.
I would like to make it so Sensu alerts (Slack handler) when it first recognizes a Consumer missing, then do not alert (Slack handler) again unless another Consumer becomes missing
It seems to me that this case could be solved for more directly by storing state local to the check, e.g. writing a temp file with a list of known-missing consumers. The [check-log.rb1 ](https://github.com/sensu-plugins/sensu-plugins-logs/blob/1.3.2/bin/check-log.rb) plugin as packaged by sensu-plugins-logs provides an example of this pattern – using a state file to track the byte position of the last read log line in a file.
I will definitely look through this check to gather an understanding on how it is storing state locally and how I might implement with my check. Thank you for the suggestion!
···
On Wednesday, September 26, 2018 at 12:03:17 PM UTC-6, Cameron Johnston wrote:
Hello!
Two thoughts on your query:
I would like to make it so Sensu alerts (Slack handler) when it first recognizes a Consumer missing, then do not alert (Slack handler) again unless another Consumer becomes missing
It seems to me that this case could be solved for more directly by storing state local to the check, e.g. writing a temp file with a list of known-missing consumers. The [check-log.rb1 ](https://github.com/sensu-plugins/sensu-plugins-logs/blob/1.3.2/bin/check-log.rb)plugin as packaged by sensu-plugins-logs provides an example of this pattern – using a state file to track the byte position of the last read log line in a file.
On an earlier version of Sensu (<0.26), just setting “refresh”: ‘x’ would have the Slack handler alert every ‘x’ seconds. With a newer version of Sensu (1.4.3), the Slack handler alerts every 60 seconds while in a critical state.
Sounds like your Slack handler plugin is no longer applying occurrence filtering. Per the changelog for the sensu-plugins-slack handlers, version 3.0.0 should be used with the built-in occurrences filter in order to retain the old behavior with respect to “refresh” and “occurrences” attributes.
Hope these notes are helpful to you.
Best,
Cameron
On Monday, September 24, 2018 at 8:24:54 PM UTC-6, mil1i wrote:
I have been trying to read through the documentation and posts and am not finding what I am looking for. I was hoping someone might be able to help figure out if what I am trying to do is even possible with Sensu at the moment.
I created a custom script that checks the RabbitMQ api for our consistent group of consumers, and reports back which server is missing.
Status: 0
No missing consumers found.
``
Status: 2
myserver1.fqdn is a missing consumer
``
Status: 2
myserver1.fqdn is a missing consumer
myserver4.fqdn is a missing consumer
myserver6.fqdn is a missing consumer
``
^ This checks the RabbitMQ api every 6 seconds. Above is sample output and the status code of the script.
I would like to make it so Sensu alerts (Slack handler) when it first recognizes a Consumer missing, then do not alert (Slack handler) again unless another Consumer becomes missing -OR- ‘X’ minutes have passed -OR- the check resolves.
On an earlier version of Sensu (<0.26), just setting “refresh”: ‘x’ would have the Slack handler alert every ‘x’ seconds. With a newer version of Sensu (1.4.3), the Slack handler alerts every 60 seconds while in a critical state.
Another option to storing local state is to leverage a sensu stash but that does require api access, which hopefully is protected with authentication and encrypted communication. As cameron said we changed the filtering to be pushed into a native sensu process to avoid forking to realize that we really did not even want to execute in the first place. Just adding "filters": ["occurrences"] should be enough to have your handler start honoring the way that it used to work but now with less load on your sensu server.