Sensu-backend service is throwing an erorr as below

When I deployed a check with sensugo the backend service is throwing an error as below. Has anyone else also seen this kind of issue?

Jun 16 10:10:55 sensu1 sensu-backend[10962]: {"check_name":"registration","check_namespace":"default","component":"pipelined","entity_name":"sensu1","entity_namespace":"default","event_id":"00000000-0000-0000-0000-000000000000","level":"warning","msg":"pipeline has no workflows, skipping execution of pipeline","pipeline_adapter":"AdapterV1","pipeline_reference":"core/v2.LegacyPipeline(Name=legacy-pipeline)","time":"2022-06-16T10:10:55Z"}
Jun 16 10:28:43 sensu1 sensu-backend[10962]: {"addr":"54.161.205.132:42958","agent":"sensu1","component":"agentd","error":"Connection closed: websocket: close 1001 (going away): bye","level":"warning","msg":"stopping session","time":"2022-06-16T10:28:43Z"}
Jun 16 10:28:45 sensu1 sensu-backend[10962]: {"component":"etcd","level":"warning","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"10.954µs","request":"header:<ID:15246797364230709779 username:\"example.com\" auth_revision:1 > put:<key:\"/sensu.io/switchsets/keepalived/default/sensu1\" value_size:3 prev_kv:true ignore_lease:true >","response":"","error":"etcdserver: key not found","time":"2022-06-16T10:28:45Z"}
Jun 16 11:43:45 sensu1 sensu-backend[10962]: {"check_name":"registration","check_namespace":"default","component":"pipelined","entity_name":"sensu1","entity_namespace":"default","event_id":"00000000-0000-0000-0000-000000000000","level":"warning","msg":"pipeline has no workflows, skipping execution of pipeline","pipeline_adapter":"AdapterV1","pipeline_reference":"core/v2.LegacyPipeline(Name=legacy-pipeline)","time":"2022-06-16T11:43:45Z"}
Jun 16 11:43:45 sensu1 sensu-backend[10962]: {"component":"etcd","level":"warning","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"12.092µs","request":"header:<ID:15246797364230714407 username:\"example.com\" auth_revision:1 > put:<key:\"/sensu.io/switchsets/keepalived/default/sensu1\" value_size:3 prev_kv:true ignore_lease:true >","response":"","error":"etcdserver: key not found","time":"2022-06-16T11:43:45Z"}
/usr/local/bin/sensu-backend version
sensu-backend version 6.6.6+ce, community edition, build 3a31c256f42c47adcd918079c7980ff201d6487e, built 2022-02-16, built with go1.18

Hi there,

The log snippet you provided doesn’t give us enough information. Can you post a longer, redacted version of your logs? This seems like etcd may be having issues, but again, it’s not possible to tell from the snippet.

OK Let me explain what is the issue.
It’s the fresh sensugo setup 6.6.6 version with 3nodes cluster I’m trying to start.
When I start the cluster for a few mins the cluster looks good. After some time the sensu-backend service throws an error as below…

Jun 17 05:50:14 sensu1 sensu-backend[23444]: {"component":"apid","level":"warning","msg":"starting apid on address: [::]:8080","time":"2022-06-17T05:50:14Z"}
Jun 17 05:50:14 sensu1 sensu-backend[23444]: {"component":"agentd","level":"warning","msg":"starting agentd on address: [::]:8081","time":"2022-06-17T05:50:14Z"}
Jun 17 05:50:14 sensu1 sensu-backend[23444]: {"component":"backend","level":"warning","msg":"backend is running and ready to accept events","time":"2022-06-17T05:50:14Z"}
Jun 17 05:54:37 sensu1 sensu-backend[23444]: {"component":"etcd","level":"warning","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"24.47µs","request":"header:<ID:15246797382432991284 username:\"example.com\" auth_revision:1 > put:<key:\"/sensu.io/switchsets/keepalived/default/sensu1\" value_size:3 prev_kv:true ignore_lease:true >","response":"","error":"etcdserver: key not found","time":"2022-06-17T05:54:37Z"}
Jun 17 05:54:37 sensu1 sensu-backend[23444]: {"check_name":"registration","check_namespace":"default","component":"pipelined","entity_name":"sensu1","entity_namespace":"default","event_id":"00000000-0000-0000-0000-000000000000","level":"warning","msg":"pipeline has no workflows, skipping execution of pipeline","pipeline_adapter":"AdapterV1","pipeline_reference":"core/v2.LegacyPipeline(Name=legacy-pipeline)","time":"2022-06-17T05:54:37Z"}
Jun 17 05:57:58 sensu1 sensu-backend[23444]: {"addr":"192.168.9.109:35056","agent":"sensu1","component":"agentd","error":"Connection closed: websocket: close 1001 (going away): bye","level":"warning","msg":"stopping session","time":"2022-06-17T05:57:58Z"}
Jun 17 05:58:01 sensu1 sensu-backend[23444]: {"component":"etcd","level":"warning","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"10.612µs","request":"header:<ID:15246797382432991435 username:\"example.com\" auth_revision:1 > put:<key:\"/sensu.io/switchsets/keepalived/default/sensu1\" value_size:3 prev_kv:true ignore_lease:true >","response":"","error":"etcdserver: key not found","time":"2022-06-17T05:58:01Z"}
Jun 17 06:18:24 sensu1 sensu-backend[23444]: {"addr":"192.168.9.109:35382","agent":"sensu1","component":"agentd","error":"Connection closed: websocket: close 1001 (going away): bye","level":"warning","msg":"stopping session","time":"2022-06-17T06:18:24Z"}
Jun 17 06:21:44 sensu1 sensu-backend[23444]: {"component":"etcd","level":"warning","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"16.423µs","request":"header:<ID:13612553667369269524 username:\"example.com\" auth_revision:1 > put:<key:\"/sensu.io/switchsets/eventd/default/test-dummy-service/sensu2\" value_size:2 prev_kv:true ignore_lease:true >","response":"","error":"etcdserver: key not found","time":"2022-06-17T06:21:44Z"}
Jun 17 06:21:44 sensu1 sensu-backend[23444]: {"component":"etcd","level":"warning","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"12.039µs","request":"header:<ID:7584767041095749136 username:\"example.com\" auth_revision:1 > put:<key:\"/sensu.io/switchsets/eventd/default/test-dummy-service/sensu3\" value_size:2 prev_kv:true ignore_lease:true >","response":"","error":"etcdserver: key not found","time":"2022-06-17T06:21:44Z"}

and cluster datails

{
  "etcd": {
    "etcdserver": "3.5.2",
    "etcdcluster": "3.5.0"
  },
  "sensu_backend": "6.6.6",
  "api_groups": {
    "core/v2": "v2.14.0",
    "core/v3": "v3.6.1"
  }
}

the health of cluster is good

sensuctl cluster health
=== Etcd Cluster ID: 51eb7547875474ce
         ID                              Name                        Error   Healthy
─────────────────── ─────────────────────────────────────────────── ─────── ──────────
  39255cb0f04d397    sensu1.example.com           true
  93681b5a46a96942   sensu2.example.com           true
  b1dbf87322e1bce9   sensu3.example.com           true

Not sure what went wrong with my setup…
im not using external etcd cluster here… its internal to sensu

Let me explain what i have done.
I installed the fresh setup of sensugo.

{
  "etcd": {
    "etcdserver": "3.5.2",
    "etcdcluster": "3.0.0"
  },
  "sensu_backend": "6.6.6",
  "api_groups": {
    "core/v2": "v2.14.0",
    "core/v3": "v3.6.1"
  }
}

When i check for cluster health

sensuctl cluster health
=== Etcd Cluster ID: 51eb7547875474ce
         ID                              Name                        Error   Healthy
----
  39255cb0f04d397     sensu1.example.com           true
  93681b5a46a96942    sensu2.example.com           true
  b1dbf87322e1bce9    sensu3.example.com           true

the backend service is failing with below error

Jun 20 05:00:38 sensu1 sensu-backend[4564]: {"component":"etcd","level":"warning","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"12.853µs","request":"header:<ID:13612553733156438326 username:\"example.com\" auth_revision:1 > put:<key:\"/sensu.io/switchsets/keepalived/default/sensu3\" value_size:3 prev_kv:true ignore_lease:true >","response":"","error":"etcdserver: key not found","time":"2022-06-20T05:00:38Z"}
Jun 20 05:00:41 sensu1 sensu-backend[4564]: {"component":"etcd","level":"warning","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"14.545µs","request":"header:<ID:13612553733156438368 username:\"example.com\" auth_revision:1 > put:<key:\"/sensu.io/switchsets/keepalived/default/sensu2\" value_size:3 prev_kv:true ignore_lease:true >","response":"","error":"etcdserver: key not found","time":"2022-06-20T05:00:41Z"}