I’m looking to use Docker to make scaling Sensu Clients easier when it comes to running workloads that involve a lot of HTTP checks/metrics against external URLs. What I did was spin up X docker instances, named for simplicity docker-sensu-1, docker-sensu-2, etc. and have them subscribe to roundrobin:www-checks.
This works great! The problem is the Sensu alerting. Let’s say HTTP check “is_google_com_up” roundrobins to docker-sensu-1, and fails, triggering an alert with the handler. The handler sees failed check “is_google_com_up” executed by client docker-sensu-1 and alerts me. Cool! However when the same check roundrobins to a different Sensu client, say docker-sensu-3, it’ll fail again (as expected) but generate yet another alert. This pattern repeats until I’ve been alerted by every client executing the roundrobin checks in which the “refresh” value kicks in.
I’m assuming I’m missing an obvious component here but I can’t for the life of me figure out what. My initial thought was that it was something to modify in the handlers themselves, but from what I understand by the time the sensu server triggers a handler it’s already made the decision to alert, so it needs to happen before the result makes it to the handler.
Any points would be greatly appreciated.