Sensu agent misconfigured: keepalived error etcdserver: server stopped: error on switch

Sensu Go
1

I’m using Sensu 5.17.1 agent and server.

The agent is running and sending data.

{“component”:“agent”,“level”:“info”,“msg”:“compacting api queue”,“time”:“2020-02-18T12:45:38-05:00”}
{“component”:“agent”,“level”:“info”,“msg”:“finished api queue compaction”,“time”:“2020-02-18T12:45:38-05:00”}
{“component”:“agent”,“level”:“info”,“msg”:“starting UDP listener on address: 127.0.0.1:3030”,“time”:“2020-02-18T12:45:38-05:00”}
{“component”:“agent”,“level”:“info”,“msg”:“starting TCP listener on address: 127.0.0.1:3030”,“time”:“2020-02-18T12:45:38-05:00”}
{“component”:“agent”,“level”:“info”,“msg”:“starting api on address: 127.0.0.1:3031”,“time”:“2020-02-18T12:45:38-05:00”}
{“component”:“agent”,“level”:“info”,“msg”:“using password auth”,“time”:“2020-02-18T12:45:38-05:00”}
{“cache”:"/var/cache/sensu/sensu-agent",“component”:“asset-manager”,“level”:“debug”,“msg”:“initializing cache directory”,“time”:“2020-02-18T12:45:38-05:00”}
{“cache”:"/var/cache/sensu/sensu-agent",“component”:“asset-manager”,“level”:“debug”,“msg”:“done initializing cache directory”,“time”:“2020-02-18T12:45:38-05:00”}
{“component”:“agent”,“level”:“info”,“msg”:“starting statsd server on address: 127.0.0.1:8125”,“time”:“2020-02-18T12:45:38-05:00”}
{“component”:“agent”,“level”:“info”,“msg”:“connecting to backend URL “wss://sensu-agent.insidelabsk8stest.mathworks.com:443"”,“time”:“2020-02-18T12:45:38-05:00”}
{“component”:“agent”,“header”:“Accept: application/octet-stream”,“level”:“debug”,“msg”:“setting header”,“time”:“2020-02-18T12:45:38-05:00”}
{“component”:“agent”,“event”:{“Title”:“Gostatsd started”,“Text”:“Gostatsd started”,“DateHappened”:1582047938,“Hostname”:“kchenowe-deb9-64”,“AggregationKey”:”",“SourceTypeName”:"",“Tags”:null,“SourceIP”:"",“Priority”:1,“AlertType”:0},“level”:“info”,“msg”:“statsd received an event”,“time”:“2020-02-18T12:45:38-05:00”}
{“component”:“agent”,“level”:“info”,“msg”:“successfully connected”,“time”:“2020-02-18T12:45:39-05:00”}
{“component”:“agent”,“header”:“Accept: [application/octet-stream application/json]”,“level”:“debug”,“msg”:“received header”,“time”:“2020-02-18T12:45:39-05:00”}
{“component”:“agent”,“format”:“protobuf”,“level”:“debug”,“msg”:“setting serialization/deserialization”,“time”:“2020-02-18T12:45:39-05:00”}
{“component”:“agent”,“header”:“Content-Type: application/octet-stream”,“level”:“debug”,“msg”:“setting header”,“time”:“2020-02-18T12:45:39-05:00”}
{“component”:“agent”,“level”:“info”,“msg”:“sending keepalive”,“time”:“2020-02-18T12:45:39-05:00”}
{“component”:“agent”,“level”:“info”,“msg”:“sending keepalive”,“time”:“2020-02-18T12:45:59-05:00”}

The server sees the incoming request and is producing an error.

{“addr”:“10.90.16.92:34096”,“agent”:“kchenowe-deb9-64”,“component”:“agentd”,“error”:“Connection closed: websocket: close 1001 (going away): bye”,“level”:“warning”,“msg”:“stopping session”,“time”:“2020-02-18T17:45:37Z”}
{“component”:“keepalived”,“error”:“etcdserver: server stopped”,“level”:“error”,“msg”:"error on switch “splunk/kchenowe-deb9-64"”,“time”:“2020-02-18T17:45:39Z”}
{“component”:“keepalived”,“error”:“etcdserver: server stopped”,“level”:“error”,“msg”:"error on switch “splunk/kchenowe-deb9-64"”,“time”:“2020-02-18T17:45:59Z”}
{“component”:“keepalived”,“error”:“etcdserver: server stopped”,“level”:“error”,“msg”:"error on switch “splunk/kchenowe-deb9-64"”,“time”:“2020-02-18T17:46:19Z”}

What does this mean?

2

I found the problem. One of my Sensu server instances is having an etcd cluster problem. So now, my new question is how do I fix that?

sensu-backend-0 1/2 CrashLoopBackOff 6 7m3s
sensu-backend-1 2/2 Running 0 6m59s
sensu-backend-2 2/2 Running 0 6m55s

For sensu-backend-0
{“component”:“etcd”,“level”:“fatal”,“msg”:“recovering backend from snapshot error: database snapshot file path error: snap: snapshot file doesn’t exist”,“pkg”:“etcdserver”,“time”:“2020-02-18T18:07:46Z”}

For what it’s worth, I’m using persistent storage in AKS.

Thanks!