Reducing event floods and check dependencies

Sensu Go doesn’t yet support being able to specify check dependencies (i.e the ability to filter events so they are not handled based on the result of a different check).

For me, this is essential to reduce alert floods. If you are monitoring a number of services on a host, a number of virtual machines on a host, a number of hosts on the same network, a number of hosts behind the same firewall/gateway, a number of hosts from a remote location, etc - then a single failure, such as a vm host dying or a link failing can mean lots of checks fail, resulting in a flood of alerts.

This is both annoying and can make it difficult to diagnose because it can be hard to see what is symptom and what is cause in the flood.

There is an issue open to add this feature, but it’s quite old and has apparently had no action (https://github.com/sensu/sensu-go/issues/3122) - which leads me to feel like i’m missing something.

Is this not an issue others have?

Or is there a workaround that i’m not thinking of to be able to achieve this filtering with the current functionality?

I was trying to think of a workaround and it’s a bit cumbersome, but I think a similar result could be achieved with check hooks - you could have check hooks so it ping’d the dependancies on a service failure and then used a filter to filter the hook results, but it would only work for the incident. You’d then get a flood of resolution messages because by the time the resolution event comes through, the hooks would return true and the events would be handled, causing a flood of OK alerts.

Any thoughts/experiences?

Thanks,

Ian

@ichilton ya it’s rather unfortunate but sensu-go even over a year after GA is still missing (in my opinion) essential feature parity with the ruby based product. My org has only started our work to migration and really our hand was forced by the lack of security patches to the old product. As someone who has relied on this feature across multiple orgs setups I feel you. Last I spoke to Sensu Inc folks about this they were gonna expose some apis that would be prerequisites to getting started on this. I can try to follow some time next week.

1 Like

Thanks - glad it’s not just me! :slight_smile:

We are all in this together, that is why community is so important.

As promised I chatted with Caleb, we should have good news to share on this front and I will make sure to update the upstream issue if no one beats me to it.

2 Likes