Did you know Sensu Go supports check scheduling using cron format?
If you are looking to run checks periodically, but on a more complicated schedule than a simple heartbeat interval you’ll want to take a look at using the cron format.
One complication though, if you have Sensu Go backends and agents operating in different timezones, you may have the check scheduled at surprising times.
For example, let’s assume you want to run a check once per day right after midnight.
This check would be scheduled on a subscribed agent using the timezone information from the Sensu backend an agent has an active websocket connection with. Different agents connected to different Sensu backends in different timezones would get a check scheduled at different times.
Sensu Go checks do not yet support the CRON_TZ prefix, but in the meantime we can work around that by adjusting the Sensu Go backend service environment to explicitly set a specific timezone. I like using UTC, but if you want to pin your Sensu Go backend to another location specific timezone that’s fine to. The key is to make sure the Sensu Go backends are using a consistent timezone and you should get consistent cron scheduling.
When using the official Sensu Go backend packages for linux, the systemd services are configured to load envvars from a /etc/sysconfig/sensu-backend We can add ad a TZ definition there:
cat /etc/sysconfig/sensu-backend
# Set Sensu Backend Service timezone explicitly to UTC
TZ=UTC
Once you restart the backend service, all cron scheduled checks will default to assuming the timezone you’ve set via the TZ envvar for the backend service.
I’ll update this post again once we have CRON_TZ prefix support for check scheduling and we can use that to modify the TZ information on a check by check basis.
Even with CRON_TZ support, in general I think it’s a good practice to make sure all the Sensu Go backends in your cluster are set to use an explicit TZ envvar to avoid head scratching surprises, especially as federation across regional clusters becomes available soon - hint hint. In a multi-cluster federation, with a different sensu cluster in each region, you may want to set a different default TZ for each cluster, to ensure you can write reusable check definitions across regions for those once-a-day off hours checks you want to perform.