# cannot get rid of keepalive's

**URL:** <https://discourse.sensu.io/t/cannot-get-rid-of-keepalives/601>\
**Category:** Sensu Classic (EOL)\
**Created:** [May 17, 2016, 5:13pm UTC](https://discourse.sensu.io/t/cannot-get-rid-of-keepalives/601 "2016-05-17T17:13:54Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Stuart\_Cracraft](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Stuart\_Cracraft](https://discourse.sensu.io/u/Stuart_Cracraft)\
**Post date:** [May 17, 2016, 5:13pm UTC](https://discourse.sensu.io/t/cannot-get-rid-of-keepalives/601/1 "2016-05-17T17:13:54Z")

</div>

Hi, I have keepalive’s configured on all my nodes on one Sensu server.

They used to be pointing at the other Sensu server.

My problem is that even though I’ve API deleted the nodes from the old Sensu server, they keep getting rediscovered and reported as keepalive failures.

Has anyone seen this?

---

<div class="post-metadata">

**Author:** ![Kyle\_Anderson](https://avatars.discourse-cdn.com/v4/letter/k/e5b9ba/32.png) [@Kyle\_Anderson](https://discourse.sensu.io/u/Kyle_Anderson)\
**Post date:** [May 18, 2016, 2:14am UTC](https://discourse.sensu.io/t/cannot-get-rid-of-keepalives/601/2 "2016-05-18T02:14:11Z")

</div>

Can you confirm there is no shared rabbitmq or redis between the new and old?  
Also when you did the move, is it possible there is some stale dns?  
(most processes, sensu included, don't re-resolve and re-connect, you  
have to restart sensu clients to get them to point to a new rabbitmq,  
etc.)

> **···**
>
> On Tue, May 17, 2016 at 10:13 AM, Stuart Cracraft \<smcracraft@gmail.com\> wrote:
> 
> > Hi, I have keepalive's configured on all my nodes on one Sensu server.
> > 
> > They used to be pointing at the other Sensu server.
> > 
> > My problem is that even though I've API deleted the nodes from the old Sensu  
> > server, they keep getting rediscovered and reported as keepalive failures.
> > 
> > Has anyone seen this?

---

<div class="post-metadata">

**Author:** ![Stuart\_Cracraft1](https://avatars.discourse-cdn.com/v4/letter/s/2bfe46/32.png) [@Stuart\_Cracraft1](https://discourse.sensu.io/u/Stuart_Cracraft1)\
**Post date:** [May 18, 2016, 2:19am UTC](https://discourse.sensu.io/t/cannot-get-rid-of-keepalives/601/3 "2016-05-18T02:19:45Z")

</div>

No shared RabbitMQ. One in each server.

No shared Redis either. We use RabbitMQ.

Also restarted Sensu client after API node delete.

Bizarre.

> **···**
>
> > On May 17, 2016, at 7:14 PM, Kyle Anderson \<kyle@xkyle.com\> wrote:
> > 
> > Can you confirm there is no shared rabbitmq or redis between the new and old?  
> > Also when you did the move, is it possible there is some stale dns?  
> > (most processes, sensu included, don't re-resolve and re-connect, you  
> > have to restart sensu clients to get them to point to a new rabbitmq,  
> > etc.)
> > 
> > > On Tue, May 17, 2016 at 10:13 AM, Stuart Cracraft \<smcracraft@gmail.com\> wrote:  
> > > Hi, I have keepalive's configured on all my nodes on one Sensu server.
> > > 
> > > They used to be pointing at the other Sensu server.
> > > 
> > > My problem is that even though I've API deleted the nodes from the old Sensu  
> > > server, they keep getting rediscovered and reported as keepalive failures.
> > > 
> > > Has anyone seen this?

---

<div class="post-metadata">

**Author:** ![Kyle\_Anderson](https://avatars.discourse-cdn.com/v4/letter/k/e5b9ba/32.png) [@Kyle\_Anderson](https://discourse.sensu.io/u/Kyle_Anderson)\
**Post date:** [May 18, 2016, 2:24am UTC](https://discourse.sensu.io/t/cannot-get-rid-of-keepalives/601/4 "2016-05-18T02:24:56Z")

</div>

Are you really sure you don't use redis? It is the only supported  
datastore of Sensu.  
[https://sensuapp.org/docs/latest/data-store](https://sensuapp.org/docs/latest/data-store)

I think it is very likely these two servers are sharing a redis.

Well keep in mind that the sensu client is pretty much always  
re-registering itself, so if you restart the client \*after\* you delete  
it from the api, it may re-register during that race.  
You should probably restart the client first, allow it to connect to  
the new rabbitmq, and \*then\* delete the client from the old server.

> **···**
>
> On Tue, May 17, 2016 at 7:19 PM, Stuart Cracraft \<smcracraft@me.com\> wrote:
> 
> > No shared RabbitMQ. One in each server.
> > 
> > No shared Redis either. We use RabbitMQ.
> > 
> > Also restarted Sensu client after API node delete.
> > 
> > Bizarre.
> > 
> > > On May 17, 2016, at 7:14 PM, Kyle Anderson \<kyle@xkyle.com\> wrote:
> > > 
> > > Can you confirm there is no shared rabbitmq or redis between the new and old?  
> > > Also when you did the move, is it possible there is some stale dns?  
> > > (most processes, sensu included, don't re-resolve and re-connect, you  
> > > have to restart sensu clients to get them to point to a new rabbitmq,  
> > > etc.)
> > > 
> > > > On Tue, May 17, 2016 at 10:13 AM, Stuart Cracraft \<smcracraft@gmail.com\> wrote:  
> > > > Hi, I have keepalive's configured on all my nodes on one Sensu server.
> > > > 
> > > > They used to be pointing at the other Sensu server.
> > > > 
> > > > My problem is that even though I've API deleted the nodes from the old Sensu  
> > > > server, they keep getting rediscovered and reported as keepalive failures.
> > > > 
> > > > Has anyone seen this?

---

<div class="post-metadata">

**Author:** ![Stuart\_Cracraft1](https://avatars.discourse-cdn.com/v4/letter/s/2bfe46/32.png) [@Stuart\_Cracraft1](https://discourse.sensu.io/u/Stuart_Cracraft1)\
**Post date:** [May 18, 2016, 3:04am UTC](https://discourse.sensu.io/t/cannot-get-rid-of-keepalives/601/5 "2016-05-18T03:04:52Z")

</div>

Okay, thanks, I’ll look into it tomorrow.

Currently involved in, ironically, a completely separate non-Sensu Redis outage.

For now the Sensu keepalive’s are filtered and sequestered away from people.

> **···**
>
> > On May 17, 2016, at 7:24 PM, Kyle Anderson \<kyle@xkyle.com\> wrote:
> > 
> > Are you really sure you don't use redis? It is the only supported  
> > datastore of Sensu.  
> > [https://sensuapp.org/docs/latest/data-store](https://sensuapp.org/docs/latest/data-store)
> > 
> > I think it is very likely these two servers are sharing a redis.
> > 
> > Well keep in mind that the sensu client is pretty much always  
> > re-registering itself, so if you restart the client \*after\* you delete  
> > it from the api, it may re-register during that race.  
> > You should probably restart the client first, allow it to connect to  
> > the new rabbitmq, and \*then\* delete the client from the old server.
> > 
> > On Tue, May 17, 2016 at 7:19 PM, Stuart Cracraft \<smcracraft@me.com\> wrote:
> > 
> > > No shared RabbitMQ. One in each server.
> > > 
> > > No shared Redis either. We use RabbitMQ.
> > > 
> > > Also restarted Sensu client after API node delete.
> > > 
> > > Bizarre.
> > > 
> > > > On May 17, 2016, at 7:14 PM, Kyle Anderson \<kyle@xkyle.com\> wrote:
> > > > 
> > > > Can you confirm there is no shared rabbitmq or redis between the new and old?  
> > > > Also when you did the move, is it possible there is some stale dns?  
> > > > (most processes, sensu included, don't re-resolve and re-connect, you  
> > > > have to restart sensu clients to get them to point to a new rabbitmq,  
> > > > etc.)
> > > > 
> > > > > On Tue, May 17, 2016 at 10:13 AM, Stuart Cracraft \<smcracraft@gmail.com\> wrote:  
> > > > > Hi, I have keepalive's configured on all my nodes on one Sensu server.
> > > > > 
> > > > > They used to be pointing at the other Sensu server.
> > > > > 
> > > > > My problem is that even though I've API deleted the nodes from the old Sensu  
> > > > > server, they keep getting rediscovered and reported as keepalive failures.
> > > > > 
> > > > > Has anyone seen this?