I see existing plugins that will substitute nicely for my existing Monit checks (CPU, disk space, temps etc) but I’d like to leverage the platform further. However I’m struggling to figure out the standard approach to be taken with custom checks.
If we take systemd timers as a case study, I would ultimately like to alert if a timer hasn’t triggered in a timely manner, and perhaps if a timer fails too.
should the sensu agent poll and parse systemctl list-timers? Should it always generate an (informational I guess) event for each entry when it does so, or only when there’s an issue (late timer or failure).
Or should the timers themselves trigger an event when they are fired?
Notifications can happen on an event, but how do you notify on more abstract things like the lack of an expected event?