avoiding a race condition in a subscriber group check


#1

Hey Guys,

Question for you all. So we have a certain check which is highly susceptible to race conditions but it needs to be run on several nodes in a role, lets call them gateways. So the gateways are in the same subscriber group ( gateways ), and as such the check is run on all of the gateways simultaneously. This of course presents a problem for this check. So my questions is if Sensu provides any locking to avoid this sort of thing ? I figured I would ask here just incase it did, this way I don’t inadvertently reinvent the wheel :). Thanks !

-John Dyer


#2

Hmm, is this check something externally checkable? (like a ping or
http) or internal? (like disk, proc). Doesn't matter I suppose in
reality.

If you must have the checks well ordered, seems like letting the
individual sensu clients schedule their own checks is a no-go.
Coordinated locking sounds hard and overkill.

What if you did something like this?

And instead of ping it was your check, and instead or parallel
(https://gist.github.com/joemiller/5806570#file-pantheon-check-ping-endpionts-rb-L181)
you did them in serial?

That way you can add what ever special waiting or lock logic you need there?

Seems like this is a special thing, I'm not currently aware of an
existing extension or plugin to do coordinated checks like this.
I think your final solution will look special, as most checks don't
have this requirement?

···

On Thu, Jan 16, 2014 at 8:15 AM, John Dyer <johntdyer@gmail.com> wrote:

Hey Guys,

Question for you all. So we have a certain check which is highly
susceptible to race conditions but it needs to be run on several nodes in a
role, lets call them gateways. So the gateways are in the same subscriber
group ( gateways ), and as such the check is run on all of the gateways
simultaneously. This of course presents a problem for this check. So my
questions is if Sensu provides any locking to avoid this sort of thing ?
I figured I would ask here just incase it did, this way I don't
inadvertently reinvent the wheel :). Thanks !

-John Dyer


#3