Couldn't start sensu-backend using external etcd - SOLVED

Sensu Go
1

error message

[eshimizu@mysqlpf-sensu-besand01 ~]$ sudo sensu-backend start --debug -c /etc/sensu/backend.yml
{"component":"backend","level":"debug","msg":"Initializing store...","time":"2019-06-12T15:21:50+09:00"}
{"component":"backend","level":"debug","msg":"Done initializing store","time":"2019-06-12T15:21:50+09:00"}
{"component":"backend","level":"debug","msg":"Registering backend...","time":"2019-06-12T15:21:50+09:00"}
{"component":"backend","level":"debug","msg":"Done registering backend.","time":"2019-06-12T15:21:51+09:00"}
{"component":"backend","entity":{"entity_class":"backend","system":{"hostname":"<HOSTNAME>","os":"linux","platform":"centos","platform_family":"rhel","platform_version":"7.3.1611","network":{"interfaces":[{"name":"lo","addresses":["127.0.0.1/8","<IP>/32"]},{"name":"eth0","mac":"<MAC>","addresses":["<NW>/23"]}]},"arch":"amd64"},"subscriptions":null,"last_seen":0,"deregister":false,"deregistration":{},"metadata":{"name":"<HOSTNAME>"}},"level":"info","msg":"backend entity information","time":"2019-06-12T15:21:51+09:00"}
{"cache":"/var/cache/sensu/sensu-backend","component":"asset-manager","level":"debug","msg":"initializing cache directory","time":"2019-06-12T15:21:51+09:00"}
{"cache":"/var/cache/sensu/sensu-backend","component":"asset-manager","level":"debug","msg":"done initializing cache directory","time":"2019-06-12T15:21:51+09:00"}
{"component":"licensing","level":"info","msg":"starting the license enforcer","time":"2019-06-12T15:21:51+09:00"}
{"component":"licensing","level":"info","msg":"no enterprise license found","time":"2019-06-12T15:21:51+09:00"}
{"component":"store","level":"debug","msg":"starting a watcher for key /sensu.io/api/enterprise/licensing/v2/license","time":"2019-06-12T15:21:51+09:00"}
{"component":"sensu-enterprise","level":"fatal","msg":"error starting schedulerd: invalid character 'k' looking for beginning of value","time":"2019-06-12T15:21:51+09:00"}
  1. What have you already tried? Please include links to gists and/or code blocks (if relatively small)
  • Recreate etcd cluster
  • Reinstall sensu-go-backend
  • Remove no-embed-etcd: true from backend.yml. (Success for start sensu-backend)
  1. Tell us about your setup, this should include OS, version of Sensu, version of Sensu components (redis, rabbitmq), plugin versions (if applicable), anything special about your setup such as an airgapped network or strict ACLs
$ uname -a
Linux <HOSTNAME> 3.10.0-957.12.1.el7.x86_64 #1 SMP Mon Apr 29 14:59:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

state-dir: "/var/lib/sensu"

etcd-listen-client-urls: "http://<etcd node1 IP>:2379,http://<etcd node2 IP>:2379,http://<etcd node3 IP>:2379"
etcd-advertise-client-urls:
  - "http://<etcd node1 IP>:2379"
  - "http://<etcd node2 IP>:2379"
  - "http://<etcd node3 IP>:2379"
no-embed-etcd: true

log-level: "debug"

  1. Is there a Github issue related to your issue?
    Couldn’t find.

  2. Is there anything else that can help us effectively help you?

2

Hi @eshimizu

Could you let us know which version of Sensu Go you are running?

Also, is this a brand new etcd cluster or is it used by other tools?

Thanks

3

@palourde
Thank you for your reply.

I use version 5.2.1(sensu-go-backend/agent/cli).
Etcd version is 3.3.12.
Etcd cluster is built for sensu.

// cluster is healthy
$ sudo etcdctl cluster-health
[sudo] password for eshimizu:
member <ID1> is healthy: got healthy result from http://<IP1>:2379
member <ID2> is healthy: got healthy result from http://<IP2>:2379
member <ID3> is healthy: got healthy result from http://<IP3>:2379
cluster is healthy

// but it is empty
$ sudo etcdctl ls | wc -l
0

If you have any other necessary information, I will report.

4

Sorry, I didn’t know 5.2.1 is not recommended version.
https://docs.sensu.io/sensu-go/5.9/installation/versions/

I will check release note for check difference and attempt upgrade to latest supported version(5.9.0).

1 Like
5

I updraded sensu-go packages.
Our sensu-backends have run correctly.
(But I couldn’t understand what is bad for sensu-backends.)

Thank you for your mention of sensu version and I’m sorry I wasted your time.