How can we get check output and decision on the bases of a time period (like 10 minutes)?

Sensu Go
1

I want to monitor number of jobs in gearman queue. My use case is that if no. of jobs in a queue remain 100 or more than 100 for 10 minutes then its response should be WARNING and if it remain 200 or more than 200 for 10 minutes then its response should be CRITICAL.

2

Hi @masifpak,

You could modify your script to make it stateful so it keep track of the last 10 minutes results and only alert based on your business logic.

Alternatively, you could also create and assign a filter to your handler(s) that inspect the number of occurrences of an event and only alert if the number of occurrences exceeded a certain trigger (e.g. if your check’s interval is 60 seconds, you would discard any occurrences below 10).

3

Thanks for your reply. Actually It is possible in handler (It supports occurrences) like mailing or sending to any alert management system. But I also don’t want status WARNING, CRITICAL in any turn.
I mean if my check run in first place, jobs are more than 100, it should wait for 2 minutes, then run again, wait for 2 minutes then run, so on. If in 5 turns even once it found jobs less than 100, it should return OK otherwise WARNING.

4

@aaronsachs Please some words on this issue. Can we use occurrences in checks. I mean mean Check geerate CRITICAL, WARNING on the bases of occurrences

5

@masifpak you might consider looking at https://docs.sensu.io/sensu-go/latest/guides/reduce-alert-fatigue/, which discusses how to do occurrence filtering and covers # of occurrences + the timeframe that you want to realert on. Let us know if that doesn’t get at what you’re trying to accomplish.