# SensuGo - check not exist but the event still occurring

**URL:** <https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487>\
**Category:** Sensu Go\
**Created:** [December 9, 2019, 3:20pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487 "2019-12-09T15:20:53Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![dorroddorrod](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@dorroddorrod](https://discourse.sensu.io/u/dorroddorrod)\
**Post date:** [December 9, 2019, 3:20pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/1 "2019-12-09T15:20:53Z")

</div>

Hi all,

I’m running a 5 nodes sensugo cluster version sensuctl version 5.14.0#071880b, build 071880b248c40729fb4b7b848df2964612c29707, built 2019-10-08T16:52:58Z  
Running on Ubuntu 14.04

I had a check that executes the following command on my agents every 60’s : `env`  
I have deleted the check but i can see that the events still occurring every 60’s.

I have already tried to re-create the check with the same name and delete it again, delete all the events, delete the entities, stop and start the agent - with no luck

Couldn’t find any related docs about this issue  
Please advice

 ![Screen Shot 2019-12-09 at 17.05.08](https://us1.discourse-cdn.com/flex016/uploads/sensu/original/1X/9e0e128a644584601e21d81d4afb1aa924cc74b4.png)

---

<div class="post-metadata">

**Author:** ![dorroddorrod](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@dorroddorrod](https://discourse.sensu.io/u/dorroddorrod)\
**Post date:** [December 9, 2019, 3:21pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/2 "2019-12-09T15:21:34Z")

</div>

![Screen Shot 2019-12-09 at 17.08.13](https://us1.discourse-cdn.com/flex016/uploads/sensu/original/1X/79e8be7c871a9525719935aaabf3fa85663027e2.png)

---

<div class="post-metadata">

**Author:** ![dorroddorrod](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@dorroddorrod](https://discourse.sensu.io/u/dorroddorrod)\
**Post date:** [December 9, 2019, 3:22pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/3 "2019-12-09T15:22:07Z")

</div>

![Screen Shot 2019-12-09 at 17.13.33](https://us1.discourse-cdn.com/flex016/uploads/sensu/original/1X/49e32a97d7d57d749a18262cf5c48c7dce7df582.png)

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [December 9, 2019, 11:44pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/4 "2019-12-09T23:44:34Z")

</div>

Hey,  
Is there anyway you could share a redacted version of the event? It might provide some clues. You aren’t making use of the agent’s event api via an external process are you? Looking at the detailed event json would provide some clues there.

Easiest way to export an event for review is with sensuctl:

`sensuctl event info --format json <entity> <check>`

Elsewise, there may be an internal problem with the sensu-backend. Have you tried restarting the sensu-backends the agent is communicating with? You’ll probably be able to restart them one at a time.

The backend scheduler maybe using a memory cached representation of the check, and the logic meant to watch for check changes and refresh that cache could be having problems.

-jef

---

<div class="post-metadata">

**Author:** ![dorroddorrod](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@dorroddorrod](https://discourse.sensu.io/u/dorroddorrod)\
**Post date:** [December 10, 2019, 7:55am UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/5 "2019-12-10T07:55:34Z")

</div>

sensuctl event info --format json \*\*\*\*\*\*\*\*\*\*\*\*\*\* test :

```auto
	"timestamp": 1575964143,
	"entity": {
		"entity_class": "agent",
		"system": {
			"hostname": "sensugo",
			"os": "linux",
			"platform": "ubuntu",
			"platform_family": "debian",
			"platform_version": "14.04",
			"network": {
				"interfaces": [{
						"name": "lo",
						"addresses": [
							"127.0.0.1/8",
							"::1/128"
						]
					},
					{
						"name": "eth0",
						"mac": "DELETED",
						"addresses": [
							"DELETED",
							"DELETED"
						]
					},
					{
						"name": "docker0",
						"mac": "DELETED",
						"addresses": [
							"172.17.0.1/16"
						]
					}
				]
			},
			"arch": "amd64"
		},
		"subscriptions": [
			"DELETED",
			"DELETED",
			"DELETED",
			"DELETED",
			"DELETED",
			"DELETED",
			"entity:DELETED"
		],
		"last_seen": 1575963171,
		"deregister": false,
		"deregistration": {},
		"user": "agent",
		"redact": [
			"password",
			"passwd",
			"pass",
			"api_key",
			"api_token",
			"access_key",
			"secret_key",
			"private_key",
			"secret"
		],
		"metadata": {
			"name": "DELETED",
			"namespace": "default",
			"labels": {
				"region": "DELETED"
			},
			"annotations": {

			}
		},
		"sensu_agent_version": "5.14.0#071880b"
	},
	"check": {
		"command": "env",
		"handlers": [
			"slack-handler"
		],
		"high_flap_threshold": 0,
		"interval": 60,
		"low_flap_threshold": 0,
		"publish": true,
		"runtime_assets": null,
		"subscriptions": [
			"system"
		],
		"proxy_entity_name": "",
		"check_hooks": null,
		"stdin": false,
		"subdue": null,
		"ttl": 0,
		"timeout": 180,
		"round_robin": false,
		"duration": 0.001506105,
		"executed": 1575964143,
		"history": [{
				"status": 1,
				"executed": 1575962943
			},
			{
				"status": 1,
				"executed": 1575963003
			},
			{
				"status": 1,
				"executed": 1575963063
			},
			{
				"status": 1,
				"executed": 1575963123
			},
			{
				"status": 1,
				"executed": 1575963183
			},
			{
				"status": 1,
				"executed": 1575963243
			},
			{
				"status": 1,
				"executed": 1575963303
			},
			{
				"status": 1,
				"executed": 1575963363
			},
			{
				"status": 1,
				"executed": 1575963423
			},
			{
				"status": 1,
				"executed": 1575963483
			},
			{
				"status": 1,
				"executed": 1575963543
			},
			{
				"status": 1,
				"executed": 1575963603
			},
			{
				"status": 1,
				"executed": 1575963663
			},
			{
				"status": 1,
				"executed": 1575963723
			},
			{
				"status": 1,
				"executed": 1575963783
			},
			{
				"status": 1,
				"executed": 1575963843
			},
			{
				"status": 1,
				"executed": 1575963903
			},
			{
				"status": 1,
				"executed": 1575963963
			},
			{
				"status": 1,
				"executed": 1575964023
			},
			{
				"status": 1,
				"executed": 1575964083
			},
			{
				"status": 1,
				"executed": 1575964143
			}
		],
		"issued": 1575964143,
		"output": "",
		"state": "failing",
		"status": 1,
		"total_state_change": 0,
		"last_ok": 1575905583,
		"occurrences": 976,
		"occurrences_watermark": 976,
		"silenced": [
			"entity:DELETED"
		],
		"output_metric_format": "",
		"output_metric_handlers": [
			"DELETED"
		],
		"env_vars": null,
		"metadata": {
			"name": "test",
			"namespace": "default",
			"labels": {
				"region": "DELETED"
			},
			"annotations": {}
		}
	},
	"metadata": {
		"namespace": "default"
	}
}

```

---

<div class="post-metadata">

**Author:** ![dorroddorrod](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@dorroddorrod](https://discourse.sensu.io/u/dorroddorrod)\
**Post date:** [December 10, 2019, 8:06am UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/6 "2019-12-10T08:06:39Z")

</div>

Restart to all sensu-backend process helped

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [December 10, 2019, 7:42pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/7 "2019-12-10T19:42:45Z")

</div>

Hey!  
Glad the remediation helped.

Follow up questions,  
Are you using embedded etcd or external etcd for your backend cluster?

how much load is the backend cluster under?  
How many Sensu events per minute are being processed?  
How many Sensu checks per minute are being processed?  
Are the backend system experience memory pressure and swapping?  
Are the backend systems under high cpu load?

Trying to get a sense of what condition is stressing the backend so this can be reproduced.

There maybe useful errors in the backend logs indicated an error associated with etcd operation that can give us pointers.

If you are willing to share the raw sensu-backend in logs with me, that might be helpful as well. I know you are sensitive to information leaking, if you prefer you can send the logs to me via email ([jef@sensu.io](mailto:jef@sensu.io)) instead of here so I can share them internally with the engineering team. Just a matter of what you are comfortable with.

---

<div class="post-metadata">

**Author:** ![dorroddorrod](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@dorroddorrod](https://discourse.sensu.io/u/dorroddorrod)\
**Post date:** [December 10, 2019, 8:02pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/8 "2019-12-10T20:02:41Z")

</div>

Hi @jspaleta,  
I really appreciate your help.

I will try to provide as much context as i can, about your questions :

running on embedded etcd cluster  
how much load is the backend cluster under? - None, i’ve got only 20 checks.  
How many Sensu events per minute are being processed? - 30 ±  
How many Sensu checks per minute are being processed? - 8  
Are the backend system experience memory pressure and swapping? - Nope  
Are the backend systems under high cpu load? - Nope

I’m not sure if i can share the raw logs, i will check it tomorrow morning.

 ![Screen Shot 2019-12-10 at 21.59.59](https://us1.discourse-cdn.com/flex016/uploads/sensu/original/1X/748d1c3d3b0116ce24f4fa1d6563d221665df462.png)

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [December 10, 2019, 8:03pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/9 "2019-12-10T20:03:56Z")

</div>

if you can’t share the raw logs, you should be able to grep for “error” messages and potentially share redacted versions of those messages.

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [December 10, 2019, 10:05pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/10 "2019-12-10T22:05:36Z")

</div>

hmm generally when there is a failure of this mode there is something stressing the etcd throughput and causing etcd to misbehave.

Other potential problems include  
slow disk i/o: are you using a spinning disk instead of an SSD?  
or  
high network latency between the etcd peers: what high is is a little fuzzy

There’s definitely some potential here for someone to build a useful grafana dashboard for users to use to turn their operational install to discover these sorts of bottle necks.  
There is a dashboard as part of the sensu-perf repository that the engineering team uses when performance tuning under load. I wonder if that dashboard could be adapted easily for general use, to help users like you discover potential resource bottlenecks…like the disk or network io issues.

Here’s the link to the performance tuning repo:

> **[sensu/sensu-perf](https://github.com/sensu/sensu-perf)**
>
> The Sensu performance testing project. Contribute to sensu/sensu-perf development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![dorroddorrod](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@dorroddorrod](https://discourse.sensu.io/u/dorroddorrod)\
**Post date:** [December 14, 2019, 2:09pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/11 "2019-12-14T14:09:52Z")

</div>

I’m running on SSD disks (AWS ebs) and use up to 25GB NIC  
I would like to run the perf-test on my cluster, where can i find the `/root/go/bin/loadit` binary file ?

---

<div class="post-metadata">

**Author:** ![jspaleta](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.sensu.io/jspaleta/32/141_2.png) [@jspaleta](https://discourse.sensu.io/u/jspaleta)\
**Post date:** [December 14, 2019, 8:07pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/12 "2019-12-14T20:07:20Z")

</div>

Hey  
loadit is part of the sensu-go codebase. Right now its meant as a development tool…so its not provided as part of the end-user packaging. It puts your system under vey aggressive load. You can pull the source and build it from the sensu-go repo…but doing it in production is going to be…inadvisable.

I think more interesting for you is to use the grafana dashboard and install a prometheus exporter on that system, so you can collect metrics from sensu-backend and the system’s prom exporter and try to catch something.

-jef

---

<div class="post-metadata">

**Author:** ![dorroddorrod](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@dorroddorrod](https://discourse.sensu.io/u/dorroddorrod)\
**Post date:** [December 14, 2019, 9:00pm UTC](https://discourse.sensu.io/t/sensugo-check-not-exist-but-the-event-still-occurring/1487/13 "2019-12-14T21:00:13Z")

</div>

I already implemented the grafana dashboard but with graphite metrics (Scraping the prometheus endpoint with telegraf ) needed to change the metric path for all graphs from prometheus to graphite but it looks OK.

I changed a bit my sensugo setup, created 5 nodes of sensu-backend (c4.xlarge) + 5 nodes etcd cluster (c5n.xlarge).  
BTW - I watched Sean Porter’s video and he that etcd should run on 3 nodes cluster but most of the designs that uses etcd recommend 5 nodes setup. what is the reason to use maximum of 3 nodes ? ([https://www.youtube.com/watch?v=EhmzmEJ4EmI](https://www.youtube.com/watch?v=EhmzmEJ4EmI))

About the perf-tests - i have few clusters that i’ve setup for test purposes .