I’m not sure why or how the init could wait for the “pod” become ready on upgrade in a statefulset.
It does make sense that it would wait for etcd cluster to be ready for scale up/scale down operations and on initial creation of the cluster.
Hmm does k8s differentiate between updating and creating a statefulset? As in can you configure k8s in such a way that it allows for only readiness check when updating an existing stateful replica, versus creating it for the first time?
Because when creating a 3 member cluster for the first time, two of the members must be up and running before either of those will bind the sensu api at 8080 and let sensu-backend init run.
Is there a way to instruct k8s to do rolling updates if at least 2 members are already healthy?
Here’s the details of what the sensu docker image is doing. I believe the command script in the official docker image does have some logic to wait for the backend service to be ready. Pulling apart the image locally the default command has a wait loop for 127.0.0.1 8080 to call backend init
And if you are using statefulsets, the only time backend init should be doing anything at all is if it needs to seed the etcd database store. If the database is already seeded (ie using a stateful volume mount for the data store directory) backend init should basically be a noop.