For Discussion: checks with time of day alert thresholds

There was a very interesting question this week in the Sensu Community slack that I think is worth re-posting here for other people to find:

Hi,how can I implement the different checks(with different threshold parameter passing to the check command) in different time range(business_hour & non_business_hour)? Shall I use the event filter for assign different event.check.command for these two time ranges? Or is there a better way to implement it?

Here’s my suggestion,

Use Check Resource Cron Attribute
if you want to have a check command run with different options at different times, use the cron-like scheduling option and create different checks that run on different schedules.

For example, lets say i want to use different warning thresholds for cpu usage during your peak usage time, with a handler that auto-remediates and lets you scale up your computer capacity by adding and additional compute vm into your environment to take on more of hte load. But off peak you use less aggressive threshold.

What I would do in this situation is construct 2 checks: peak-cpu and offpeak-cpu.
In the peak-cpu check, instead of setting the interval attribute i would set the cron attribute to look something like this:
cron: "* 0-8 * * *"
which requests the check to be scheduled once every minute from 00:00 to 08:59 inclusive, in the locally configured timezone the sensu agent is running in.

I would then schedule the offpeak-check with the complementary cron schedule:
cron: "* 9-23 * * *"
which requests the check to be scheduled once every minute from 09:00 to 23:59 inclusive, in the locally configured timezone the sensu agent is running in.

A Comment About Event Filters
Event filters are best used to control how handlers are triggered, irrespective of what a check is actually doing. I like to think about it this way, time of day event filters let you control how alerts flow to your humans and your remediation automation. Maybe during US business hours you want a different group of people being alerted during China business hours. Or you want the auto-remediation scripts to fire more quickly. Or you want slack alerts during one part of the day and email alerts during other parts. Event filters are great for that sort of logic.

If you know you want to use different alert thresholds based on time of day, encode that operational knowledge in your check resources, by using the cron-like scheduling. Build your event filtering as if every alert is real instead of trying to figure out if the alert is a false positive because the operational settings for the check are wrong.