Sensu Go doesn’t yet support being able to specify check dependencies (i.e the ability to filter events so they are not handled based on the result of a different check).
For me, this is essential to reduce alert floods. If you are monitoring a number of services on a host, a number of virtual machines on a host, a number of hosts on the same network, a number of hosts behind the same firewall/gateway, a number of hosts from a remote location, etc - then a single failure, such as a vm host dying or a link failing can mean lots of checks fail, resulting in a flood of alerts.
This is both annoying and can make it difficult to diagnose because it can be hard to see what is symptom and what is cause in the flood.
There is an issue open to add this feature, but it’s quite old and has apparently had no action (https://github.com/sensu/sensu-go/issues/3122) - which leads me to feel like i’m missing something.
Is this not an issue others have?
Or is there a workaround that i’m not thinking of to be able to achieve this filtering with the current functionality?
I was trying to think of a workaround and it’s a bit cumbersome, but I think a similar result could be achieved with check hooks - you could have check hooks so it ping’d the dependancies on a service failure and then used a filter to filter the hook results, but it would only work for the incident. You’d then get a flood of resolution messages because by the time the resolution event comes through, the hooks would return true and the events would be handled, causing a flood of OK alerts.
Any thoughts/experiences?
Thanks,
Ian
