Questions regarding check extensions

Amr_Ali · July 30, 2014, 5:43am

We have about 15 checks running on each of our hosts, most of them are metric checks that run every 10s, this is causing alot of sensu-clients instances constantly spawned on the hosts.. i am wondering if :

1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks extension file? - if we do it this way it might also pose an issue with the extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on the servers.

Thanks,

Amr

Amr_Ali · August 5, 2014, 2:12am

Any insight on this please?

···

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are metric checks that run every 10s, this is causing alot of sensu-clients instances constantly spawned on the hosts… i am wondering if :
1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks extension file? - if we do it this way it might also pose an issue with the extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on the servers.

Thanks,

Amr

Kyle_Anderson · August 5, 2014, 2:50pm

I am about to look at porting some handlers (not checks) of mine into
extensions for performance reasons.

1. I'm assuming your metric checks are in ruby? Have you looked at the
other examples of this yet?
https://github.com/sensu/sensu-community-plugins/tree/master/extensions/checks

2. As far as I understand check extensions, are executed close to
where normal checks are:

github.com

sensu/sensu/blob/c81c50a54bb746cfc6be952e75ed5ffcda2ccfb1/lib/sensu/client.rb#L144-L156


      
              execute_check_command(check)
            elsif @safe_mode
              check[:output] = 'Check is not locally defined (safe mode)'
              check[:status] = 3
              check[:handle] = false
              check[:executed] = Time.now.to_i
              publish_result(check)
            else
              execute_check_command(check)
            end
          else
            if @extensions.check_exists?(check[:name])
              run_check_extension(check)

I don't think there is any additional downside here, only a speed
improvement from not forking.

3. If you were *not* doing check extensions, then I would recommend
that. Since you are doing check extensions, I don't see a benefit of
executing them all together like that unless your metrics are "all or
nothing".

···

On Mon, Aug 4, 2014 at 7:12 PM, Amr Ali <amrenator@gmail.com> wrote:

Any insight on this please?

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are
metric checks that run every 10s, this is causing alot of sensu-clients
instances constantly spawned on the hosts.. i am wondering if :

1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions
running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks
extension file? - if we do it this way it might also pose an issue with the
extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on
the servers.

Thanks,

Amr

Amr_Ali · August 5, 2014, 7:54pm

Thanks,

i am in the process of transforming the checks to extensions, most of them are checks from the community plugins, so will hopefully be sharing them once they are done.

will note the performance impact after porting some of the checks and testing them.

···

On Tuesday, August 5, 2014 4:51:01 PM UTC+2, Kyle Anderson wrote:

I am about to look at porting some handlers (not checks) of mine into

extensions for performance reasons.

I’m assuming your metric checks are in ruby? Have you looked at the

other examples of this yet?

https://github.com/sensu/sensu-community-plugins/tree/master/extensions/checks

As far as I understand check extensions, are executed close to

where normal checks are:

https://github.com/sensu/sensu/blob/c81c50a54bb746cfc6be952e75ed5ffcda2ccfb1/lib/sensu/client.rb#L144-L156

I don’t think there is any additional downside here, only a speed

improvement from not forking.

If you were not doing check extensions, then I would recommend

that. Since you are doing check extensions, I don’t see a benefit of

executing them all together like that unless your metrics are "all or

nothing".

On Mon, Aug 4, 2014 at 7:12 PM, Amr Ali amre...@gmail.com wrote:

Any insight on this please?

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are

metric checks that run every 10s, this is causing alot of sensu-clients

instances constantly spawned on the hosts… i am wondering if :

1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions

running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks

extension file? - if we do it this way it might also pose an issue with the

extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on

the servers.

Thanks,

Amr

Sean_Porter · August 5, 2014, 7:57pm

Be sure that your extensions do not block the reactor, you must use EventMachine libraries, as they run within the Sensu event loop. Extensions can impact the client’s performance and errors will crash it.

···

On Aug 5, 2014 12:54 PM, “Amr Ali” amrenator@gmail.com wrote:

Thanks,

i am in the process of transforming the checks to extensions, most of them are checks from the community plugins, so will hopefully be sharing them once they are done.

will note the performance impact after porting some of the checks and testing them.

On Tuesday, August 5, 2014 4:51:01 PM UTC+2, Kyle Anderson wrote:

I am about to look at porting some handlers (not checks) of mine into
extensions for performance reasons.

I’m assuming your metric checks are in ruby? Have you looked at the
other examples of this yet?
https://github.com/sensu/sensu-community-plugins/tree/master/extensions/checks

As far as I understand check extensions, are executed close to
where normal checks are:
https://github.com/sensu/sensu/blob/c81c50a54bb746cfc6be952e75ed5ffcda2ccfb1/lib/sensu/client.rb#L144-L156

I don’t think there is any additional downside here, only a speed
improvement from not forking.

If you were not doing check extensions, then I would recommend
that. Since you are doing check extensions, I don’t see a benefit of
executing them all together like that unless your metrics are “all or
nothing”.

On Mon, Aug 4, 2014 at 7:12 PM, Amr Ali amre...@gmail.com wrote:

Any insight on this please?

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are
metric checks that run every 10s, this is causing alot of sensu-clients
instances constantly spawned on the hosts… i am wondering if :

1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions
running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks
extension file? - if we do it this way it might also pose an issue with the
extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on
the servers.

Thanks,

Amr

Amr_Ali · August 5, 2014, 8:02pm

So in this case and that was my main question, is it better to leave things as they are and not worry about forked processes?

or will using extensions be better performance wise? i’m kinda skeptical about this my understanding is it’d be ok to run a couple of check extensions but doing 10+ would actually be worse than leaving them as normal checks, is that right?

···

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are metric checks that run every 10s, this is causing alot of sensu-clients instances constantly spawned on the hosts… i am wondering if :
1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks extension file? - if we do it this way it might also pose an issue with the extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on the servers.

Thanks,

Amr

Kyle_Anderson · August 6, 2014, 1:12am

Using check extensions will be better performance-wise, it is just
more dangerous because of what @portertech just described.
Depending on how much you value your cpu, it may or may not be worth it.

···

On Tue, Aug 5, 2014 at 1:02 PM, Amr Ali <amrenator@gmail.com> wrote:

So in this case and that was my main question, is it better to leave things
as they are and not worry about forked processes?

or will using extensions be better performance wise? i'm kinda skeptical
about this my understanding is it'd be ok to run a couple of check
extensions but doing 10+ would actually be worse than leaving them as normal
checks, is that right?

On Wednesday, July 30, 2014 7:43:48 AM UTC+2, Amr Ali wrote:

We have about 15 checks running on each of our hosts, most of them are
metric checks that run every 10s, this is causing alot of sensu-clients
instances constantly spawned on the hosts.. i am wondering if :

1- would it be a good idea to port our metric checks to extension checks?

2- if we do is there a downside to having several check extensions
running at the same time, will this cause delays in checks execution?

3- depending on 2 should we combine all our checks into one checks
extension file? - if we do it this way it might also pose an issue with the
extension taking time to execute all the check functions it includes.

Looking for insight on the best way to handle this with minimal load on
the servers.

Thanks,

Amr

Topic		Replies	Views
sensu performance Sensu Classic (EOL)	1	632	November 6, 2018
Anyone successfully using check extensions in recent versions? (>= 0.13.0) Sensu Classic (EOL)	0	432	February 20, 2015
Check Extensions Sensu Classic (EOL)	2	466	November 22, 2018
Isolating checks Sensu Classic (EOL)	8	989	May 30, 2019
Sensu-client metadata around spawning checks Sensu Classic (EOL)	0	640	November 29, 2018

Questions regarding check extensions

Related topics