Questions regarding check extensions


#1

We have about 15 checks running on each of our hosts, most of them are metric checks that run every 10s, this is causing alot of sensu-clients instances constantly spawned on the hosts.. i am wondering if :

1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks extension file? - if we do it this way it might also pose an issue with the extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on the servers.

Thanks,

Amr


#2

Any insight on this please? :slight_smile:

···

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are metric checks that run every 10s, this is causing alot of sensu-clients instances constantly spawned on the hosts… i am wondering if :
1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks extension file? - if we do it this way it might also pose an issue with the extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on the servers.

Thanks,

Amr


#3

I am about to look at porting some handlers (not checks) of mine into
extensions for performance reasons.

1. I'm assuming your metric checks are in ruby? Have you looked at the
other examples of this yet?
https://github.com/sensu/sensu-community-plugins/tree/master/extensions/checks

2. As far as I understand check extensions, are executed close to
where normal checks are:


I don't think there is any additional downside here, only a speed
improvement from not forking.

3. If you were *not* doing check extensions, then I would recommend
that. Since you are doing check extensions, I don't see a benefit of
executing them all together like that unless your metrics are "all or
nothing".

···

On Mon, Aug 4, 2014 at 7:12 PM, Amr Ali <amrenator@gmail.com> wrote:

Any insight on this please? :slight_smile:

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are
metric checks that run every 10s, this is causing alot of sensu-clients
instances constantly spawned on the hosts.. i am wondering if :

1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions
running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks
extension file? - if we do it this way it might also pose an issue with the
extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on
the servers.

Thanks,

Amr


#4

Thanks,

i am in the process of transforming the checks to extensions, most of them are checks from the community plugins, so will hopefully be sharing them once they are done.

will note the performance impact after porting some of the checks and testing them.

···

On Tuesday, August 5, 2014 4:51:01 PM UTC+2, Kyle Anderson wrote:

I am about to look at porting some handlers (not checks) of mine into

extensions for performance reasons.

  1. I’m assuming your metric checks are in ruby? Have you looked at the

other examples of this yet?

https://github.com/sensu/sensu-community-plugins/tree/master/extensions/checks

  1. As far as I understand check extensions, are executed close to

where normal checks are:

https://github.com/sensu/sensu/blob/c81c50a54bb746cfc6be952e75ed5ffcda2ccfb1/lib/sensu/client.rb#L144-L156

I don’t think there is any additional downside here, only a speed

improvement from not forking.

  1. If you were not doing check extensions, then I would recommend

that. Since you are doing check extensions, I don’t see a benefit of

executing them all together like that unless your metrics are "all or

nothing".

On Mon, Aug 4, 2014 at 7:12 PM, Amr Ali amre...@gmail.com wrote:

Any insight on this please? :slight_smile:

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are

metric checks that run every 10s, this is causing alot of sensu-clients

instances constantly spawned on the hosts… i am wondering if :

1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions

running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks

extension file? - if we do it this way it might also pose an issue with the

extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on

the servers.

Thanks,

Amr


#5

Be sure that your extensions do not block the reactor, you must use EventMachine libraries, as they run within the Sensu event loop. Extensions can impact the client’s performance and errors will crash it.

···

On Aug 5, 2014 12:54 PM, “Amr Ali” amrenator@gmail.com wrote:

Thanks,

i am in the process of transforming the checks to extensions, most of them are checks from the community plugins, so will hopefully be sharing them once they are done.

will note the performance impact after porting some of the checks and testing them.

On Tuesday, August 5, 2014 4:51:01 PM UTC+2, Kyle Anderson wrote:

I am about to look at porting some handlers (not checks) of mine into
extensions for performance reasons.

  1. I’m assuming your metric checks are in ruby? Have you looked at the
    other examples of this yet?
    https://github.com/sensu/sensu-community-plugins/tree/master/extensions/checks

  2. As far as I understand check extensions, are executed close to
    where normal checks are:
    https://github.com/sensu/sensu/blob/c81c50a54bb746cfc6be952e75ed5ffcda2ccfb1/lib/sensu/client.rb#L144-L156

I don’t think there is any additional downside here, only a speed
improvement from not forking.

  1. If you were not doing check extensions, then I would recommend
    that. Since you are doing check extensions, I don’t see a benefit of
    executing them all together like that unless your metrics are “all or
    nothing”.

On Mon, Aug 4, 2014 at 7:12 PM, Amr Ali amre...@gmail.com wrote:

Any insight on this please? :slight_smile:

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are
metric checks that run every 10s, this is causing alot of sensu-clients
instances constantly spawned on the hosts… i am wondering if :

1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions
running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks
extension file? - if we do it this way it might also pose an issue with the
extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on
the servers.

Thanks,

Amr


#6

So in this case and that was my main question, is it better to leave things as they are and not worry about forked processes?

or will using extensions be better performance wise? i’m kinda skeptical about this my understanding is it’d be ok to run a couple of check extensions but doing 10+ would actually be worse than leaving them as normal checks, is that right?

···

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are metric checks that run every 10s, this is causing alot of sensu-clients instances constantly spawned on the hosts… i am wondering if :
1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks extension file? - if we do it this way it might also pose an issue with the extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on the servers.

Thanks,

Amr


#7

Using check extensions will be better performance-wise, it is just
more dangerous because of what @portertech just described.
Depending on how much you value your cpu, it may or may not be worth it.

···

On Tue, Aug 5, 2014 at 1:02 PM, Amr Ali <amrenator@gmail.com> wrote:

So in this case and that was my main question, is it better to leave things
as they are and not worry about forked processes?

or will using extensions be better performance wise? i'm kinda skeptical
about this my understanding is it'd be ok to run a couple of check
extensions but doing 10+ would actually be worse than leaving them as normal
checks, is that right?

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are
metric checks that run every 10s, this is causing alot of sensu-clients
instances constantly spawned on the hosts.. i am wondering if :

1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions
running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks
extension file? - if we do it this way it might also pose an issue with the
extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on
the servers.

Thanks,

Amr