Error starting etcd: multiple discovery or boot strap flags set

Hi, I cloned GitHub - betorvs/sensu-go-statefulset: Helm chart to deploy sensu-go Backend using statefulset in kubernetes (as-is) and successfully deployed Sensu Go v5.21.3 on a Ubuntu 20.04.01 system under MicroK8s.

When I modify the values.yaml file to use Sensu Go v6.0.0 or v6.2.0 I receive the following error:

== waiting for sensu-backend-sensu-go-statefulset-0:2379 to become available before running backend-init...
{"component":"sensu-enterprise","error":"error starting etcd: multiple discovery or bootstrap flags are set. Choose one of \"initial-cluster\", \"discovery\" or \"discovery-srv\"","level":"fatal","msg":"error executing sensu-backend","time":"2021-01-04T22:56:34Z"}

The problem appears to be raised with the following command:

sensu-backend start --etcd-name ${HOST_NAME}.sensu.sensu.svc.cluster.local \
--etcd-discovery-srv sensu.sensu.svc.cluster.local \
--etcd-initial-advertise-peer-urls http://${HOST_NAME}.sensu.sensu.svc.cluster.local:2380 \
--etcd-initial-cluster-token sensu --etcd-initial-cluster-state new \
--etcd-advertise-client-urls http://${HOST_NAME}.sensu.sensu.svc.cluster.local:2379 \
--etcd-listen-client-urls http://0.0.0.0:2379 --etcd-listen-peer-urls http://0.0.0.0:2380 \
--state-dir /var/lib/sensu/sensu-backend/${HOST_NAME} --log-level debug --debug \
--api-url http://sensu-api.example.local:8080

Reading through migration notes from 5.21.3 to 6.0.0 I did find anything that would change the above command.

I did find a similar problem for etcd: Multiple discovery flags error with valid config file · Issue #7516 · etcd-io/etcd · GitHub

Per the “hack fix” I tried inserting a blank --etcd-initial-cluster, also tried setting using the environment variable, but no luck.

Is there something I am missing in the above command for the migration?

I have performed addition testing

Using a fresh install of Ubuntu 20.4.1 LTS and Docker 20.10.2 I ran the following commands (based on Install Sensu - Sensu Docs )

$ docker run -v /var/lib/sensu:/var/lib/sensu \
-d --name sensu-backend \
-p 3000:3000 -p 8080:8080 -p 8081:8081 \
-e SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=admin \
-e SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=admin \
sensu/sensu:6.2.0 \
sensu-backend start --state-dir /var/lib/sensu/sensu-backend --log-level debug \
--etcd-discovery-srv test

Tailing the container log I find the following error:

$ docker container logs --tail 100 sensu-backend
== waiting for a2992910a825:2379 to become available before running backend-init...
{"component":"sensu-enterprise","error":"error starting etcd: multiple discovery or bootstrap flags are set. Choose one of \"initial-cluster\", \"discovery\" or \"discovery-srv\"","level":"fatal","msg":"error executing sensu-backend","time":"2021-01-13T21:49:18Z"}

I should not see this error as I am only specifying the single flag etcd-discovery-srv (i.e. neither etcd-initial-cluster or etcd-discovery are specified).

Next, I tried different container versions in the run command above and found:

  • sensu/sensu:5.21.3 does not get the error
  • sensu/sensu:6.0.0 does get the error

Last, I tried install v6.2.0 directly onto Ubuntu 20.4.1 LTS as follows:

$ curl -s https://packagecloud.io/install/repositories/sensu/stable/script.deb.sh | sudo bash
$ sudo apt-get install sensu-go-backend

I then ran the following tests:

$ sensu-backend start --etcd-initial-cluster default=http://127.0.0.1:2380

No errors from this command as I am only specifying the single bootstrap flag (etcd-initial-cluster)

$ sensu-backend start --etcd-discovery-srv test.org

No errors from this command as I am only specifying the single discovery flag (etcd-discovery-srv)

$ sensu-backend start --etcd-initial-cluster default=http://127.0.0.1:2380 --etcd-discovery-srv test
{"component":"sensu-enterprise","error":"error starting etcd: multiple discovery or bootstrap flags are set. Choose one of \"initial-cluster\", \"discovery\" or \"discovery-srv\"","level":"fatal","msg":"error executing sensu-backend","time":"2021-01-14T13:44:16-07:00"}

The above error is expected as I am specifying multiple conflicting flags (etcd-initial-cluster and etcd-discovery-srv)

Given the above testing I believe I have shown there was a bug introduced into the Sensu Go v6+ etcd cluster member discovery logic.