Deregister client automatically <possible solution>

Sensu Classic (EOL)
1

Background/Problem:

I use vagrant a lot for testing so when I spin up a vagrant box, using Chef, it will register itself with my sensu environment. When I kill the vagrant box I need to go to the dashboard and remove the client, if I remember too. At the moment I don’t want to expose the sensu-api beyond the api server itself, yet I want a way to deregister a client from sensu automatically. I also don’t want every developer and user out there to have the ability to resolve events either or issue api commands at will.

Proposed Solution:

An init script on the client side runs when a graceful shutdown is performed, this script calls a ruby plugin that drops a check-result onto port 3030 of the local machine for the client to deliver to the server. The “output” of this result can be any api command, in this case I set the output to delete, example:

‘{ “name”: “api_call”, “output”: “delete”, “process”: “init”, “user”: “root”, “handler”: “sensu-api-handler”}’

A handler on the server is set as a pipe and sends the check-result to an “api-handler”. This handler then evaluates the output string to decide what to do. In this case it sees delete. The handler then, using the api, creates a stash containing the following:

{“path”: “api_call”/

“content”: {

node:

user: default is sensu

process: default is sensu

api_cmd: “delete”

timestamp: Time.now

}}

Forgive any syntax errors, thats what linters are for. Then, using the api as well, a delete command is issued to the api-server and removes that client from the dashboard.

This stash is created so that with modification, the handlers check here for a delete value and if so drop the check-result as the machine will be going down cleanly soon.

For cleanup, I have a check that runs every 60 seconds and removes stashes with a value of delete, this prevents Sensu from dropping valid results do to a stash.

I am aware that there is a way to expire the stashes automatically, I just haven’t gotten that far yet. In a perfect world I wouldn’t need to do this. The client, upon graceful shutdown, would issue an api delete command and remove itself, at some point I will look at that. Also, it would be nice if the client was able to do some of the above steps naturally. Possibly by picking up a key/value pair in a check result and knowing that this is an api command.

I understand there is/would be some duplicate effort involved here. The main thought behind this is to leverage the check-result Q to deliver api commands to the api server, thereby exposing one less machine, or distributing one less key/credentials to the environment. A method such as this may also be a way to control or mange in a small capacity appliances by having a dedicated sensu-client be responsible for accepting traps from a group of appliances and issuing api commands on behalf of the appliance.

Just figured I would throw this out there

2

Why not have the sensu-api-handler just remove the client immediately,
and skip the stash part?
It could be a simple bash script that just runs sensu-cli client
delete $cilent_name

Also, client side you don't need any fancy ruby script, just bash is
fine if you want:
echo '{ "name": "api_call", "output": "delete", "process": "init",
"user": "root", "handler": "sensu-api-handler"}' >
/dev/tcp/localhost/3030

You will need a non-zero status code to get your handler to pick up on it.

Just make sure your init scripts are ordered correctly, I have
something similar in production and getting the shutdown sequence
ordered correctly across different distros and init systems is hard.
My thing does a sensu-cli client delete on shutdown, level 0 (I
distribute credentials to the clients), and I add a 5 minute silence
stash on reboot (6).

···

On Fri, Nov 14, 2014 at 7:40 AM, matty jones <urlugal@gmail.com> wrote:

Background/Problem:

I use vagrant a lot for testing so when I spin up a vagrant box, using Chef,
it will register itself with my sensu environment. When I kill the vagrant
box I need to go to the dashboard and remove the client, if I remember too.
At the moment I don't want to expose the sensu-api beyond the api server
itself, yet I want a way to deregister a client from sensu automatically. I
also don't want every developer and user out there to have the ability to
resolve events either or issue api commands at will.

Proposed Solution:

An init script on the client side runs when a graceful shutdown is
performed, this script calls a ruby plugin that drops a check-result onto
port 3030 of the local machine for the client to deliver to the server. The
"output" of this result can be any api command, in this case I set the
output to delete, example:

'{ "name": "api_call", "output": "delete", "process": "init", "user":
"root", "handler": "sensu-api-handler"}'

A handler on the server is set as a pipe and sends the check-result to an
"api-handler". This handler then evaluates the output string to decide what
to do. In this case it sees delete. The handler then, using the api,
creates a stash containing the following:

{"path": "api_call"/<sensu-client>
"content": {
                    node: <sensu-client>
                    user: <user making the call> default is sensu
                    process: <process making the call> default is sensu
                    api_cmd: "delete"
                    timestamp: Time.now
}}

Forgive any syntax errors, thats what linters are for. Then, using the api
as well, a delete command is issued to the api-server and removes that
client from the dashboard.

This stash is created so that with modification, the handlers check here for
a delete value and if so drop the check-result as the machine will be going
down cleanly soon.

For cleanup, I have a check that runs every 60 seconds and removes stashes
with a value of delete, this prevents Sensu from dropping valid results do
to a stash.

I am aware that there is a way to expire the stashes automatically, I just
haven't gotten that far yet. In a perfect world I wouldn't need to do this.
The client, upon graceful shutdown, would issue an api delete command and
remove itself, at some point I will look at that. Also, it would be nice if
the client was able to do some of the above steps naturally. Possibly by
picking up a key/value pair in a check result and knowing that this is an
api command.

I understand there is/would be some duplicate effort involved here. The
main thought behind this is to leverage the check-result Q to deliver api
commands to the api server, thereby exposing one less machine, or
distributing one less key/credentials to the environment. A method such as
this may also be a way to control or mange in a small capacity appliances by
having a dedicated sensu-client be responsible for accepting traps from a
group of appliances and issuing api commands on behalf of the appliance.

Just figured I would throw this out there

3

Why not have the sensu-api-handler just remove the client immediately,
and skip the stash part?

The stash part is necessary to avoid a race condition where a keep alive
comes in after the deregister has been sent.

It could be a simple bash script that just runs sensu-cli client
delete $cilent_name

The sensu-cli depends upon the sensu-api, to my understanding of the code

it is simply a wrapper, and in the case of my environment, the sensu-api is
not exposed beyond the sensu server. If you mean on the server side, I
prefer Ruby just to keep things all the same language, it also makes it
simplier, for my environment anyway, to write tests for and manage.

Also, client side you don't need any fancy ruby script, just bash is
fine if you want:
echo '{ "name": "api_call", "output": "delete", "process": "init",
"user": "root", "handler": "sensu-api-handler"}' >
/dev/tcp/localhost/3030

I don't want to do anything with the init script but call a Ruby script,
makes it easier to template with Chef. I want the power/flexibility of
Ruby as well, who knows what I will want to do later on with this script so
I want a separate script to handle the json generation.. At some point in
time the client side will be able to handle any api calls, so I wrote and
designed it to account for this. I don't believe in writing one-offs it I
can extrapolate the code out to something useful but still retain a sense
of scope I will. The delete call was the one I needed right away so that
is what I have been testing with the last few days.

You will need a non-zero status code to get your handler to pick up on it.

Just make sure your init scripts are ordered correctly, I have
something similar in production and getting the shutdown sequence
ordered correctly across different distros and init systems is hard.
My thing does a sensu-cli client delete on shutdown, level 0 (I
distribute credentials to the clients), and I add a 5 minute silence
stash on reboot (6).

The default client shutdown script is at 74 I believe, I set this script

to 70 but I am still going to create the stash as I don't like being nearly
certain, or leaving things to chance such as network latency or whatnot. I
want it to be absolute, and the only two ways in my mind to do that were to
create a delete stash or to modify the client to do this itself as a last
dying act (next on my list of things to not get done this year).

Thanks for the input

···

On Fri, Nov 14, 2014 at 11:13 PM, Kyle Anderson <kyle@xkyle.com> wrote:

On Fri, Nov 14, 2014 at 7:40 AM, matty jones <urlugal@gmail.com> wrote:
> Background/Problem:
>
> I use vagrant a lot for testing so when I spin up a vagrant box, using
Chef,
> it will register itself with my sensu environment. When I kill the
vagrant
> box I need to go to the dashboard and remove the client, if I remember
too.
> At the moment I don't want to expose the sensu-api beyond the api server
> itself, yet I want a way to deregister a client from sensu
automatically. I
> also don't want every developer and user out there to have the ability to
> resolve events either or issue api commands at will.
>
> Proposed Solution:
>
> An init script on the client side runs when a graceful shutdown is
> performed, this script calls a ruby plugin that drops a check-result onto
> port 3030 of the local machine for the client to deliver to the server.
The
> "output" of this result can be any api command, in this case I set the
> output to delete, example:
>
> '{ "name": "api_call", "output": "delete", "process": "init", "user":
> "root", "handler": "sensu-api-handler"}'
>
> A handler on the server is set as a pipe and sends the check-result to an
> "api-handler". This handler then evaluates the output string to decide
what
> to do. In this case it sees delete. The handler then, using the api,
> creates a stash containing the following:
>
> {"path": "api_call"/<sensu-client>
> "content": {
> node: <sensu-client>
> user: <user making the call> default is sensu
> process: <process making the call> default is sensu
> api_cmd: "delete"
> timestamp: Time.now
> }}
>
> Forgive any syntax errors, thats what linters are for. Then, using the
api
> as well, a delete command is issued to the api-server and removes that
> client from the dashboard.
>
> This stash is created so that with modification, the handlers check here
for
> a delete value and if so drop the check-result as the machine will be
going
> down cleanly soon.
>
> For cleanup, I have a check that runs every 60 seconds and removes
stashes
> with a value of delete, this prevents Sensu from dropping valid results
do
> to a stash.
>
> I am aware that there is a way to expire the stashes automatically, I
just
> haven't gotten that far yet. In a perfect world I wouldn't need to do
this.
> The client, upon graceful shutdown, would issue an api delete command and
> remove itself, at some point I will look at that. Also, it would be
nice if
> the client was able to do some of the above steps naturally. Possibly by
> picking up a key/value pair in a check result and knowing that this is an
> api command.
>
> I understand there is/would be some duplicate effort involved here. The
> main thought behind this is to leverage the check-result Q to deliver api
> commands to the api server, thereby exposing one less machine, or
> distributing one less key/credentials to the environment. A method such
as
> this may also be a way to control or mange in a small capacity
appliances by
> having a dedicated sensu-client be responsible for accepting traps from a
> group of appliances and issuing api commands on behalf of the appliance.
>
> Just figured I would throw this out there

4

Matt, did you figure out how to go about this. I am also looking for something similar.

Thanks

···

On Friday, November 14, 2014 at 8:28:31 PM UTC-8, Matt Jones wrote:

On Fri, Nov 14, 2014 at 11:13 PM, Kyle Anderson ky...@xkyle.com wrote:

Why not have the sensu-api-handler just remove the client immediately,

and skip the stash part?

The stash part is necessary to avoid a race condition where a keep alive comes in after the deregister has been sent.

It could be a simple bash script that just runs sensu-cli client

delete $cilent_name

The sensu-cli depends upon the sensu-api, to my understanding of the code it is simply a wrapper, and in the case of my environment, the sensu-api is not exposed beyond the sensu server. If you mean on the server side, I prefer Ruby just to keep things all the same language, it also makes it simplier, for my environment anyway, to write tests for and manage.

Also, client side you don’t need any fancy ruby script, just bash is

fine if you want:

echo '{ “name”: “api_call”, “output”: “delete”, “process”: “init”,

“user”: “root”, “handler”: “sensu-api-handler”}’ >

/dev/tcp/localhost/3030

I don’t want to do anything with the init script but call a Ruby script, makes it easier to template with Chef. I want the power/flexibility of Ruby as well, who knows what I will want to do later on with this script so I want a separate script to handle the json generation… At some point in time the client side will be able to handle any api calls, so I wrote and designed it to account for this. I don’t believe in writing one-offs it I can extrapolate the code out to something useful but still retain a sense of scope I will. The delete call was the one I needed right away so that is what I have been testing with the last few days.

You will need a non-zero status code to get your handler to pick up on it.

Just make sure your init scripts are ordered correctly, I have

something similar in production and getting the shutdown sequence

ordered correctly across different distros and init systems is hard.

My thing does a sensu-cli client delete on shutdown, level 0 (I

distribute credentials to the clients), and I add a 5 minute silence

stash on reboot (6).

The default client shutdown script is at 74 I believe, I set this script to 70 but I am still going to create the stash as I don’t like being nearly certain, or leaving things to chance such as network latency or whatnot. I want it to be absolute, and the only two ways in my mind to do that were to create a delete stash or to modify the client to do this itself as a last dying act (next on my list of things to not get done this year).

Thanks for the input

On Fri, Nov 14, 2014 at 7:40 AM, matty jones url...@gmail.com wrote:

Background/Problem:

I use vagrant a lot for testing so when I spin up a vagrant box, using Chef,

it will register itself with my sensu environment. When I kill the vagrant

box I need to go to the dashboard and remove the client, if I remember too.

At the moment I don’t want to expose the sensu-api beyond the api server

itself, yet I want a way to deregister a client from sensu automatically. I

also don’t want every developer and user out there to have the ability to

resolve events either or issue api commands at will.

Proposed Solution:

An init script on the client side runs when a graceful shutdown is

performed, this script calls a ruby plugin that drops a check-result onto

port 3030 of the local machine for the client to deliver to the server. The

“output” of this result can be any api command, in this case I set the

output to delete, example:

'{ “name”: “api_call”, “output”: “delete”, “process”: “init”, “user”:

“root”, “handler”: “sensu-api-handler”}’

A handler on the server is set as a pipe and sends the check-result to an

“api-handler”. This handler then evaluates the output string to decide what

to do. In this case it sees delete. The handler then, using the api,

creates a stash containing the following:

{“path”: “api_call”/

“content”: {

                node: <sensu-client>

                user: <user making the call> default is sensu

                process: <process making the call> default is sensu

                api_cmd: "delete"

                timestamp: Time.now

}}

Forgive any syntax errors, thats what linters are for. Then, using the api

as well, a delete command is issued to the api-server and removes that

client from the dashboard.

This stash is created so that with modification, the handlers check here for

a delete value and if so drop the check-result as the machine will be going

down cleanly soon.

For cleanup, I have a check that runs every 60 seconds and removes stashes

with a value of delete, this prevents Sensu from dropping valid results do

to a stash.

I am aware that there is a way to expire the stashes automatically, I just

haven’t gotten that far yet. In a perfect world I wouldn’t need to do this.

The client, upon graceful shutdown, would issue an api delete command and

remove itself, at some point I will look at that. Also, it would be nice if

the client was able to do some of the above steps naturally. Possibly by

picking up a key/value pair in a check result and knowing that this is an

api command.

I understand there is/would be some duplicate effort involved here. The

main thought behind this is to leverage the check-result Q to deliver api

commands to the api server, thereby exposing one less machine, or

distributing one less key/credentials to the environment. A method such as

this may also be a way to control or mange in a small capacity appliances by

having a dedicated sensu-client be responsible for accepting traps from a

group of appliances and issuing api commands on behalf of the appliance.

Just figured I would throw this out there

5

It looks like sensu client will get native deregister support in the future. The roadmap scheduled it to 0.21.

6

GREAT!!!

···

Em quarta-feira, 22 de julho de 2015 15:27:32 UTC-3, Philipp H escreveu:

It looks like sensu client will get native deregister support in the future. The roadmap scheduled it to 0.21.