Sensu is not stable

Sensu Go
1

My sensu is not stable. You may see from the following, I run “sensuctl cluster health” several times and every time I got different results. Can anyone please help how to fix this? Thanks in advance.

~$ sensuctl cluster health
Error: GET “/health”: failed to request new refresh token; client returned ‘The server returned the error: 500 Internal Server Error’

~$ sensuctl cluster health
=== Etcd Cluster ID: 6757f62c6737d318
ID Name Error Healthy
────────────────── ───────────── ─────── ─────────
dbda02213d86af4f azv-sensu00 true
75377d4d0c37f7ac azv-sensu01 true
935b2beb5939c35 azv-sensu02 true

~$ sensuctl cluster health
Error: GET “/health”: Get http://10.206.159.150:8080/health: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

~$ sensuctl cluster health
=== Etcd Cluster ID: 6757f62c6737d318
ID Name Error Healthy
────────────────── ───────────── ─────── ─────────
dbda02213d86af4f azv-sensu00 true
75377d4d0c37f7ac azv-sensu01 true
935b2beb5939c35 azv-sensu02 true

2

Hope the following error is helpful

Error: GraphQL error: etcdserver: request timed out
at new ApolloError (http:///static/js/vendor_89b2.js:90:188247)
at http:///static/js/vendor_89b2.js:90:210401
at http:///static/js/vendor_89b2.js:90:216828
at Set.forEach ()
at http:///static/js/vendor_89b2.js:90:216805
at Map.forEach ()
at QueryManager.broadcastQueries (http:///static/js/vendor_89b2.js:90:216756)
at http:///static/js/vendor_89b2.js:90:218737
at Object.next (http:///static/js/vendor_89b2.js:105:28920)
at notifySubscription (http:///static/js/vendor_89b2.js:105:26227)
at onNotify (http:///static/js/vendor_89b2.js:105:26843)
at SubscriptionObserver.next (http:///static/js/vendor_89b2.js:105:27657)
at http:///static/js/vendor_89b2.js:90:217848
at tryCatcher (http:///static/js/vendor_89b2.js:90:141139)
at Promise._settlePromiseFromHandler (http:///static/js/vendor_89b2.js:90:128795)
at Promise._settlePromise (http:///static/js/vendor_89b2.js:90:129633)

3

Hello,
One possible cause is etcd cluster is under resource pressure ( diskio usually, but could be cpu or memory pressure) etcd will fail to respond to requests causing timeouts like the error you see here.

Ref:
https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/troubleshoot/#symptoms-of-poor-performance

4

Thanks for your help