# How to properly handle keepalive and AWS EC2 start/stop instances
**URL:** https://discourse.sensu.io/t/how-to-properly-handle-keepalive-and-aws-ec2-start-stop-instances/516
**Category:** Sensu Classic (EOL)
**Created:** [December 11, 2015, 12:06pm UTC](https://discourse.sensu.io/t/how-to-properly-handle-keepalive-and-aws-ec2-start-stop-instances/516 "2015-12-11T12:06:22Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Jonathan\_Ballet](https://discourse.sensu.io/u/Jonathan_Ballet)
#### Post date: [December 11, 2015, 12:06pm UTC](https://discourse.sensu.io/t/how-to-properly-handle-keepalive-and-aws-ec2-start-stop-instances/516/1 "2015-12-11T12:06:22Z")
Hi,
we have some fairly large (and expensive) instances on AWS which we use only 6 to 10 hours each day to run some batch computation.
We are in the process of starting/stopping the instances at the times when we want the computation to be done to save a bit on the bill.
These instances are monitored by Sensu and each of them run the Sensu client, which pings back to the server every once in a while with the keepalive function. If we just shutdown the instances for several hours, we will have keepalives warnings popping up in our reporting system. How should I approach this kind of behavior?
Ideally, I would like to say something like : “I know this instance is supposed to run every days at midnight, don’t produce any keepalive (or other) warnings until the moment I stop and next midnight”, but I’m not sure how I should do that. Creating a stash via the Sensu API server?
Do you have any other possible solution to this problem?
Thanks!
Jonathan
---
### Author:  [@Bryan\_Brandau](https://discourse.sensu.io/u/Bryan_Brandau)
#### Post date: [December 11, 2015, 1:47pm UTC](https://discourse.sensu.io/t/how-to-properly-handle-keepalive-and-aws-ec2-start-stop-instances/516/2 "2015-12-11T13:47:49Z")
It is similar logic that is done in a aws decommission handler. You’ll want to check for stopped instances and remove the client from sensu. When they come back up they will register again and everything will be happy.
For an overview of logic that I’m talking about, see here: [http://www.ragedsyscoder.com/blog/2014/01/14/sensu-automated-decommission-of-clients/](http://www.ragedsyscoder.com/blog/2014/01/14/sensu-automated-decommission-of-clients/)
-Bryan
> **···**
>
> On Fri, Dec 11, 2015 at 6:06 AM, Jonathan Ballet [jon@multani.info](mailto:jon@multani.info) wrote:
>
> > Hi,
> >
> > we have some fairly large (and expensive) instances on AWS which we use only 6 to 10 hours each day to run some batch computation.
> > We are in the process of starting/stopping the instances at the times when we want the computation to be done to save a bit on the bill.
> >
> > These instances are monitored by Sensu and each of them run the Sensu client, which pings back to the server every once in a while with the keepalive function. If we just shutdown the instances for several hours, we will have keepalives warnings popping up in our reporting system. How should I approach this kind of behavior?
> >
> > Ideally, I would like to say something like : “I know this instance is supposed to run every days at midnight, don’t produce any keepalive (or other) warnings until the moment I stop and next midnight”, but I’m not sure how I should do that. Creating a stash via the Sensu API server?
> >
> > Do you have any other possible solution to this problem?
> >
> > Thanks!
> >
> > Jonathan
---
### Author:  [@matty\_jones](https://discourse.sensu.io/u/matty_jones)
#### Post date: [December 11, 2015, 2:09pm UTC](https://discourse.sensu.io/t/how-to-properly-handle-keepalive-and-aws-ec2-start-stop-instances/516/3 "2015-12-11T14:09:45Z")
Write an init script that upon graceful shutdown makes an api call to pull the client from monitoring and create a stash. Then modify then keep alive handler to check for this stash before sending an alert.
The stash is used solely to prevent a race condition where a client would get added back in after it was pulled. I time it fairly close, we have the kill script kicked at 70 and sensu-client gets pulled at 75 I believe.
> **···**
>
> On Fri, Dec 11, 2015 at 6:06 AM, Jonathan Ballet [jon@multani.info](mailto:jon@multani.info) wrote:
>
> > Hi,
> >
> > we have some fairly large (and expensive) instances on AWS which we use only 6 to 10 hours each day to run some batch computation.
> > We are in the process of starting/stopping the instances at the times when we want the computation to be done to save a bit on the bill.
> >
> > These instances are monitored by Sensu and each of them run the Sensu client, which pings back to the server every once in a while with the keepalive function. If we just shutdown the instances for several hours, we will have keepalives warnings popping up in our reporting system. How should I approach this kind of behavior?
> >
> > Ideally, I would like to say something like : “I know this instance is supposed to run every days at midnight, don’t produce any keepalive (or other) warnings until the moment I stop and next midnight”, but I’m not sure how I should do that. Creating a stash via the Sensu API server?
> >
> > Do you have any other possible solution to this problem?
> >
> > Thanks!
> >
> > Jonathan
---
### Author:  [@Jonathan\_Ballet](https://discourse.sensu.io/u/Jonathan_Ballet)
#### Post date: [December 13, 2015, 6:54pm UTC](https://discourse.sensu.io/t/how-to-properly-handle-keepalive-and-aws-ec2-start-stop-instances/516/4 "2015-12-13T18:54:04Z")
Hi Bryan,
Thanks for your answer! I came on your website already when I was looking for an answer for this problem, but I was looking for something hopefully less radical than removing the client from Pdmpt itself. It is something which is going to happen fairly frequently (everyday, for a couple of machines at first), and I was more looking for a way to say to Sensu :“it’s OK if this machine is off for the moment, don’t get all stressed about it”.
I’ll still keep an eye on your article as it may help me to program something!
Jonathan
Jonathan
> **···**
>
> On Fri, Dec 11, 2015 at 6:06 AM, Jonathan Ballet [jon@multani.info](mailto:jon@multani.info) wrote:
>
> > Hi,
> >
> > we have some fairly large (and expensive) instances on AWS which we use only 6 to 10 hours each day to run some batch computation.
> > We are in the process of starting/stopping the instances at the times when we want the computation to be done to save a bit on the bill.
> >
> > These instances are monitored by Sensu and each of them run the Sensu client, which pings back to the server every once in a while with the keepalive function. If we just shutdown the instances for several hours, we will have keepalives warnings popping up in our reporting system. How should I approach this kind of behavior?
> >
> > Ideally, I would like to say something like : “I know this instance is supposed to run every days at midnight, don’t produce any keepalive (or other) warnings until the moment I stop and next midnight”, but I’m not sure how I should do that. Creating a stash via the Sensu API server?
> >
> > Do you have any other possible solution to this problem?
> >
> > Thanks!
> >
> > Jonathan
---
### Author:  [@Jonathan\_Ballet](https://discourse.sensu.io/u/Jonathan_Ballet)
#### Post date: [December 13, 2015, 7:01pm UTC](https://discourse.sensu.io/t/how-to-properly-handle-keepalive-and-aws-ec2-start-stop-instances/516/5 "2015-12-13T19:01:17Z")
Hi Matty,
Thanks for your answer! Have you tried to write a solution where you were having custom keep ajgtdp sent by the client, like “I should be sending the next keep alive in 20 seconds”?
I’m not sure I understand the race condition you are talking about, would you care to explain?
Also, when you speak about 70 and 75, are you referring to the init script number which removes the stash and starts the Sensu client?
Thanks!
Jonathan
> **···**
>
> On Fri, Dec 11, 2015 at 6:06 AM, Jonathan Ballet [jon@multani.info](mailto:jon@multani.info) wrote:
>
> > Hi,
> >
> > we have some fairly large (and expensive) instances on AWS which we use only 6 to 10 hours each day to run some batch computation.
> > We are in the process of starting/stopping the instances at the times when we want the computation to be done to save a bit on the bill.
> >
> > These instances are monitored by Sensu and each of them run the Sensu client, which pings back to the server every once in a while with the keepalive function. If we just shutdown the instances for several hours, we will have keepalives warnings popping up in our reporting system. How should I approach this kind of behavior?
> >
> > Ideally, I would like to say something like : “I know this instance is supposed to run every days at midnight, don’t produce any keepali ve (or other) warnings until the moment I stop and next midnight”, but I’m not sure how I should do that. Creating a stash via the Sensu API server?
> >
> > Do you have any other possible solution to this problem?
> >
> > Thanks!
> >
> > Jonathan
---
### Author:  [@Matt\_Jones](https://discourse.sensu.io/u/Matt_Jones)
#### Post date: [December 13, 2015, 7:17pm UTC](https://discourse.sensu.io/t/how-to-properly-handle-keepalive-and-aws-ec2-start-stop-instances/516/6 "2015-12-13T19:17:28Z")
I have not tried doing something like that.
The race condition would be something along the lines of the following, with the being the init script number which refers to the sequence that is gets executed in.
SHUTDOWN ORDER:
\<70\> send remove host to sensu
\<70\> send api call to create stash saying client has been removed
\<75\> shutdown sensu client
The reason I have the stash is that after I send the call to remove the host \<70\>, the sensu-client still has time to send another keep-alive. If that keep-alive comes in then the client will get added back in and we would thus have an alert on a “phantom” host.
By creating the stash and modifying the keepalive handler, before an alert is sent it will check the stashes to see if the host has be given a “remove” status and thus don’t alert on it as it is being shutdown gracefully.
Here are the modified [handlers](https://github.com/yieldbot/sensu-yieldbot-plugins/tree/master/handlers/other)
This is what we use to [send the call](https://github.com/yieldbot/sensu-yieldbot-plugins/blob/master/plugins/sensu/sensu-socket-client.rb)from the init script to the server
Here is a copy of the [kill script](https://gist.github.com/mattyjones/79e018a65ee8d81a8d41) that we drop on machines
> **···**
>
> On Sun, Dec 13, 2015 at 2:01 PM, Jonathan Ballet [jon@multani.info](mailto:jon@multani.info) wrote:
>
> > Hi Matty,
> >
> > Thanks for your answer! Have you tried to write a solution where you were having custom keep ajgtdp sent by the client, like “I should be sending the next keep alive in 20 seconds”?
> >
> > I’m not sure I understand the race condition you are talking about, would you care to explain?
> >
> > Also, when you speak about 70 and 75, are you referring to the init script number which removes the stash and starts the Sensu client?
> >
> > Thanks!
> >
> > Jonathan
> >
> > On December 11, 2015 3:09:45 PM CET, matty jones [urlugal@gmail.com](mailto:urlugal@gmail.com) wrote:
> >
> > > Write an init script that upon graceful shutdown makes an api call to pull the client from monitoring and create a stash. Then modify then keep alive handler to check for this stash before sending an alert.
>
> > > The stash is used solely to prevent a race condition where a client would get added back in after it was pulled. I time it fairly close, we have the kill script kicked at 70 and sensu-client gets pulled at 75 I believe.
>
> > > On Dec 11, 2015 8:47 AM, “Bryan Brandau” [agent462@gmail.com](mailto:agent462@gmail.com) wrote:
> > >
> > > > It is similar logic that is done in a aws decommission handler. You’ll want to check for stopped instances and remove the client from sensu. When they come back up they will register again and everything will be happy.
>
> > > > For an overview of logic that I’m talking about, see here: [http://www.ragedsyscoder.com/blog/2014/01/14/sensu-automated-decommission-of-clients/](http://www.ragedsyscoder.com/blog/2014/01/14/sensu-automated-decommission-of-clients/)
>
> > > >
>
> > > > -Bryan
>
> –
>
> On Fri, Dec 11, 2015 at 6:06 AM, Jonathan Ballet [jon@multani.info](mailto:jon@multani.info) wrote:
>
> > Hi,
> >
> > we have some fairly large (and expensive) instances on AWS which we use only 6 to 10 hours each day to run some batch computation.
> > We are in the process of starting/stopping the instances at the times when we want the computation to be done to save a bit on the bill.
> >
> > These instances are monitored by Sensu and each of them run the Sensu client, which pings back to the server every once in a while with the keepalive function. If we just shutdown the instances for several hours, we will have keepalives warnings popping up in our reporting system. How should I approach this kind of behavior?
> >
> > Ideally, I would like to say something like : “I know this instance is supposed to run every days at midnight, don’t produce any keepali ve (or other) warnings until the moment I stop and next midnight”, but I’m not sure how I should do that. Creating a stash via the Sensu API server?
> >
> > Do you have any other possible solution to this problem?
> >
> > Thanks!
> >
> > Jonathan
>
> Matt Jones @DevopsMatt
>
> Infrastructure Engineer - [Yieldbot Inc.](http://yieldbot.com/)
>
> Core Contributor - [Sensu Plugins](http://sensu-plugins.github.io/)
>
> Co-Organizer - [Boston Infrastructure Coders](http://www.meetup.com/Boston-Infrastructure-Coders/)
>
> [https://linkedin.com/in/mattyjones](https://linkedin.com/in/mattyjones)
---
### Author:  [@Jonathan\_Ballet](https://discourse.sensu.io/u/Jonathan_Ballet)
#### Post date: [December 13, 2015, 10:33pm UTC](https://discourse.sensu.io/t/how-to-properly-handle-keepalive-and-aws-ec2-start-stop-instances/516/7 "2015-12-13T22:33:05Z")
Hi Matt,
OK, I understand better the race condition now 🙂
I'll have a look at your scripts and see if I can do something with them, they seem to be nicely written and I can most probably learn a lot from them, thanks!
Jonathan
> **···**
>
> On 12/13/2015 08:17 PM, Matt Jones wrote:
>
> > I have not tried doing something like that.
> >
> > The race condition would be something along the lines of the following,
> > with the \ being the init script number which refers to the
> > sequence that is gets executed in.
> >
> > SHUTDOWN ORDER:
> >
> > \<70\> send remove host to sensu
> > \<70\> send api call to create stash saying client has been removed
> > \<75\> shutdown sensu client
> >
> > The reason I have the stash is that after I send the call to remove the
> > host \<70\>, the sensu-client still has time to send another keep-alive.
> > If that keep-alive comes in then the client will get added back in and
> > we would thus have an alert on a "phantom" host.
> >
> > By creating the stash and modifying the keepalive handler, before an
> > alert is sent it will check the stashes to see if the host has be given
> > a "remove" status and thus don't alert on it as it is being shutdown
> > gracefully.
> >
> > Here are the modified handlers
> > \<[https://github.com/yieldbot/sensu-yieldbot-plugins/tree/master/handlers/other>](https://github.com/yieldbot/sensu-yieldbot-plugins/tree/master/handlers/other>);
> >
> > This is what we use to send the call
> > \<[https://github.com/yieldbot/sensu-yieldbot-plugins/blob/master/plugins/sensu/sensu-socket-client.rb>from](https://github.com/yieldbot/sensu-yieldbot-plugins/blob/master/plugins/sensu/sensu-socket-client.rb>from)
> > the init script to the server
> >
> > Here is a copy of the kill script
> > \<[https://gist.github.com/mattyjones/79e018a65ee8d81a8d41>](https://gist.github.com/mattyjones/79e018a65ee8d81a8d41>); that we drop
> > on machines
> >
> > On Sun, Dec 13, 2015 at 2:01 PM, Jonathan Ballet \ \\> wrote:
> >
> > Hi Matty,
> >
> > Thanks for your answer! Have you tried to write a solution where you
> > were having custom keep ajgtdp sent by the client, like "I should be
> > sending the next keep alive in 20 seconds"?
> > I'm not sure I understand the race condition you are talking about,
> > would you care to explain?
> > Also, when you speak about 70 and 75, are you referring to the init
> > script number which removes the stash and starts the Sensu client?
> >
> > Thanks!
> >
> > Jonathan
> >
> > On December 11, 2015 3:09:45 PM CET, matty jones \ \\> wrote:
> >
> > Write an init script that upon graceful shutdown makes an api
> > call to pull the client from monitoring and create a stash. Then
> > modify then keep alive handler to check for this stash before
> > sending an alert.
> >
> > The stash is used solely to prevent a race condition where a
> > client would get added back in after it was pulled. I time it
> > fairly close, we have the kill script kicked at 70 and
> > sensu-client gets pulled at 75 I believe.
> >
> > On Dec 11, 2015 8:47 AM, "Bryan Brandau" \ \\> wrote:
> >
> > It is similar logic that is done in a aws decommission
> > handler. You’ll want to check for stopped instances and
> > remove the client from sensu. When they come back up they
> > will register again and everything will be happy.
> >
> > For an overview of logic that I’m talking about, see here:
> > [http://www.ragedsyscoder.com/blog/2014/01/14/sensu-automated-decommission-of-clients/](http://www.ragedsyscoder.com/blog/2014/01/14/sensu-automated-decommission-of-clients/)
> >
> > -Bryan
> >
> > On Fri, Dec 11, 2015 at 6:06 AM, Jonathan Ballet \> \\> wrote:
> >
> > Hi,
> >
> > we have some fairly large (and expensive) instances on
> > AWS which we use only 6 to 10 hours each day to run some
> > batch computation.
> > We are in the process of starting/stopping the instances
> > at the times when we want the computation to be done to
> > save a bit on the bill.
> >
> > These instances are monitored by Sensu and each of them
> > run the Sensu client, which pings back to the server
> > every once in a while with the keepalive function. If we
> > just shutdown the instances for several hours, we will
> > have keepalives warnings popping up in our reporting
> > system. How should I approach this kind of behavior?
> >
> > Ideally, I would like to say something like : "I know
> > this instance is supposed to run every days at midnight,
> > don't produce any keepali ve (or other) warnings until
> > the moment I stop and next midnight", but I'm not sure
> > how I should do that. Creating a stash via the Sensu API
> > server?
> >
> > Do you have any other possible solution to this problem?
> >
> > Thanks!
> >
> > Jonathan
> >
> > --
> > Matt Jones @DevopsMatt
> > Infrastructure Engineer - Yieldbot Inc. \<[BIGLUCK8: Slot Server Thailand Gacor Situs Slot 88 Gampang Maxwin](http://yieldbot.com/>);
> > Core Contributor - Sensu Plugins \<[http://sensu-plugins.github.io/>](http://sensu-plugins.github.io/>);
> > Co-Organizer - Boston Infrastructure Coders
> > \<[http://www.meetup.com/Boston-Infrastructure-Coders/>](http://www.meetup.com/Boston-Infrastructure-Coders/>);
> > [https://linkedin.com/in/mattyjones](https://linkedin.com/in/mattyjones)