Hi all,
I’m looking to use Docker to make scaling Sensu Clients easier when it comes to running workloads that involve a lot of HTTP checks/metrics against external URLs. What I did was spin up X docker instances, named for simplicity docker-sensu-1, docker-sensu-2, etc. and have them subscribe to roundrobin:www-checks.
This works great! The problem is the Sensu alerting. Let’s say HTTP check “is_google_com_up” roundrobins to docker-sensu-1, and fails, triggering an alert with the handler. The handler sees failed check “is_google_com_up” executed by client docker-sensu-1 and alerts me. Cool! However when the same check roundrobins to a different Sensu client, say docker-sensu-3, it’ll fail again (as expected) but generate yet another alert. This pattern repeats until I’ve been alerted by every client executing the roundrobin checks in which the “refresh” value kicks in.
I’m assuming I’m missing an obvious component here but I can’t for the life of me figure out what. My initial thought was that it was something to modify in the handlers themselves, but from what I understand by the time the sensu server triggers a handler it’s already made the decision to alert, so it needs to happen before the result makes it to the handler.
Any points would be greatly appreciated.
Yes, the missing component here is to specify the “source” setting on the check:
https://sensuapp.org/docs/0.25/reference/checks.html#check-attributes
In this case the source could be “google.com” and the check could be called “check_is_up” if you want. Then the check will show up as google.com and be “deduped” in a sense and not show up as “docker-sensu-3”, etc.
Setting the source will initialize a “proxy-client” automatically:
https://sensuapp.org/docs/latest/reference/clients.html#proxy-clients
···
On Tue, Aug 30, 2016 at 3:14 PM, uberamd uberamd@gmail.com wrote:
Hi all,
I’m looking to use Docker to make scaling Sensu Clients easier when it comes to running workloads that involve a lot of HTTP checks/metrics against external URLs. What I did was spin up X docker instances, named for simplicity docker-sensu-1, docker-sensu-2, etc. and have them subscribe to roundrobin:www-checks.
This works great! The problem is the Sensu alerting. Let’s say HTTP check “is_google_com_up” roundrobins to docker-sensu-1, and fails, triggering an alert with the handler. The handler sees failed check “is_google_com_up” executed by client docker-sensu-1 and alerts me. Cool! However when the same check roundrobins to a different Sensu client, say docker-sensu-3, it’ll fail again (as expected) but generate yet another alert. This pattern repeats until I’ve been alerted by every client executing the roundrobin checks in which the “refresh” value kicks in.
I’m assuming I’m missing an obvious component here but I can’t for the life of me figure out what. My initial thought was that it was something to modify in the handlers themselves, but from what I understand by the time the sensu server triggers a handler it’s already made the decision to alert, so it needs to happen before the result makes it to the handler.
Any points would be greatly appreciated.
Perfect, thank you! I wasn’t sure if the handler would see the check as being from the proxy client or from the check executor itself but it sounds like the handler will see it from the proxy, which is great.
I appreciate the assistance.
···
On Tuesday, August 30, 2016 at 9:39:39 PM UTC-5, Kyle Anderson wrote:
Yes, the missing component here is to specify the “source” setting on the check:
https://sensuapp.org/docs/0.25/reference/checks.html#check-attributes
In this case the source could be “google.com” and the check could be called “check_is_up” if you want. Then the check will show up as google.com and be “deduped” in a sense and not show up as “docker-sensu-3”, etc.
Setting the source will initialize a “proxy-client” automatically:
https://sensuapp.org/docs/latest/reference/clients.html#proxy-clients
On Tue, Aug 30, 2016 at 3:14 PM, uberamd ube...@gmail.com wrote:
Hi all,
I’m looking to use Docker to make scaling Sensu Clients easier when it comes to running workloads that involve a lot of HTTP checks/metrics against external URLs. What I did was spin up X docker instances, named for simplicity docker-sensu-1, docker-sensu-2, etc. and have them subscribe to roundrobin:www-checks.
This works great! The problem is the Sensu alerting. Let’s say HTTP check “is_google_com_up” roundrobins to docker-sensu-1, and fails, triggering an alert with the handler. The handler sees failed check “is_google_com_up” executed by client docker-sensu-1 and alerts me. Cool! However when the same check roundrobins to a different Sensu client, say docker-sensu-3, it’ll fail again (as expected) but generate yet another alert. This pattern repeats until I’ve been alerted by every client executing the roundrobin checks in which the “refresh” value kicks in.
I’m assuming I’m missing an obvious component here but I can’t for the life of me figure out what. My initial thought was that it was something to modify in the handlers themselves, but from what I understand by the time the sensu server triggers a handler it’s already made the decision to alert, so it needs to happen before the result makes it to the handler.
Any points would be greatly appreciated.