When to use a standalone check?


#1

Hi all,

I’m wondering when standalone checks are useful. Initially I thought it would increase availability such that clients could continue monitoring even if server is down. But it was not the case because handlers run on the server anyway.

One thing I’ve come up with is that they could be useful when the link between the server and the client is narrow. In this case, we might want to save bandwidth by stopping triggering check events from the server. The actual gain might be subtle, though.

So my question is: when are standalone checks useful?

Thanks,

Mitsutoshi


#2

Standalone checks are for special snowflakes, IMO. So you have swaths of servers you monitor more or less the same, for those the subscription model is perfect. But I use standalone checks for the janky old servers nobody’s touched in ages, that’s always running at 60% cpu and whatnot. You still want to monitor it, but you don’t want to create a centralized config only for this one server.

My 2¢ :slight_smile:

Mat

···

On Thu, Nov 28, 2013 at 12:04 AM, Mitsutoshi Aoe maoe@foldr.in wrote:

Hi all,

I’m wondering when standalone checks are useful. Initially I thought it would increase availability such that clients could continue monitoring even if server is down. But it was not the case because handlers run on the server anyway.

One thing I’ve come up with is that they could be useful when the link between the server and the client is narrow. In this case, we might want to save bandwidth by stopping triggering check events from the server. The actual gain might be subtle, though.

So my question is: when are standalone checks useful?

Thanks,

Mitsutoshi

I’m the founder of Rock Solid Ops, a web operations and development consultancy.

I’m also the organizer of DevOpsMtl, a monthly event to discuss the challenges, tools and the culture surrounding the movement.

My main fields of expertise are web development with Ruby on Rails, DevOps and a bit of mobile development. If you need help scaling, monitoring, securing or managing your web infrastructure (Rails or not), get in touch!

Connect with me and read testimonials on my LinkedIn, follow me on twitter @webmat, or check out my blog at programblings.com.


#3

Hi Mathieu,
Thanks for sharing your experience!

Given that we use Chef/Puupet to configure Sensu, why wouldn't you
like to put the irregular config into the sensu server?

Regards,

Mitsutoshi

Mitsutoshi Aoe
maoe@foldr.in

···

2013/11/28 Mathieu Martin <webmat@gmail.com>:

Standalone checks are for special snowflakes, IMO. So you have swaths of
servers you monitor more or less the same, for those the subscription model
is perfect. But I use standalone checks for the janky old servers nobody's
touched in ages, that's always running at 60% cpu and whatnot. You still
want to monitor it, but you don't want to create a centralized config only
for this one server.

My 2¢ :slight_smile:

Mat

On Thu, Nov 28, 2013 at 12:04 AM, Mitsutoshi Aoe <maoe@foldr.in> wrote:

Hi all,

I'm wondering when standalone checks are useful. Initially I thought it
would increase availability such that clients could continue monitoring even
if server is down. But it was not the case because handlers run on the
server anyway.

One thing I've come up with is that they could be useful when the link
between the server and the client is narrow. In this case, we might want to
save bandwidth by stopping triggering check events from the server. The
actual gain might be subtle, though.

So my question is: when are standalone checks useful?

Thanks,

Mitsutoshi

--

I'm the founder of Rock Solid Ops, a web operations and development
consultancy.

I'm also the organizer of DevOpsMtl, a monthly event to discuss the
challenges, tools and the culture surrounding the movement.

My main fields of expertise are web development with Ruby on Rails, DevOps
and a bit of mobile development. If you need help scaling, monitoring,
securing or managing your web infrastructure (Rails or not), get in touch!

Connect with me and read testimonials on my LinkedIn, follow me on twitter
@webmat, or check out my blog at programblings.com.


#4

Hi Mitsutoshi,

Fair point. Here’s how I see it. The snowflake config is still ideally configured by the config management tool, I agree. But instead it would create the config file directly on the snowflake, as a standalone check.

I’m not a fan of having it on the Sensu server as one of the subscriptions, because, suppose I have a “mysql” subscription for all my up to date DB servers. Then out of my bunch of servers, I have 2 old MySQL snowflakes that have different oddities going for them. I’d have to create a “mysql-oldapp1” subscription only for one of the servers, and another “mysql-oldapp2” subscription for the other server. That just seems like useless chatter happening on RabbitMQ.

And actually it’s good that you mention config management tools. I haven’t been in that situation recently, but sometimes the snowflakes aren’t even being configured with a tool like Chef… One more case where standalone checks can be useful: a manual Sensu setup :slight_smile:

HTH,

Mat

···

On Thu, Nov 28, 2013 at 7:36 PM, Mitsutoshi Aoe maoe@foldr.in wrote:

Hi Mathieu,

Thanks for sharing your experience!

Given that we use Chef/Puupet to configure Sensu, why wouldn’t you

like to put the irregular config into the sensu server?

Regards,

Mitsutoshi

Mitsutoshi Aoe

maoe@foldr.in

2013/11/28 Mathieu Martin webmat@gmail.com:

Standalone checks are for special snowflakes, IMO. So you have swaths of

servers you monitor more or less the same, for those the subscription model

is perfect. But I use standalone checks for the janky old servers nobody’s

touched in ages, that’s always running at 60% cpu and whatnot. You still

want to monitor it, but you don’t want to create a centralized config only

for this one server.

My 2¢ :slight_smile:

Mat

On Thu, Nov 28, 2013 at 12:04 AM, Mitsutoshi Aoe maoe@foldr.in wrote:

Hi all,

I’m wondering when standalone checks are useful. Initially I thought it

would increase availability such that clients could continue monitoring even

if server is down. But it was not the case because handlers run on the

server anyway.

One thing I’ve come up with is that they could be useful when the link

between the server and the client is narrow. In this case, we might want to

save bandwidth by stopping triggering check events from the server. The

actual gain might be subtle, though.

So my question is: when are standalone checks useful?

Thanks,

Mitsutoshi

I’m the founder of Rock Solid Ops, a web operations and development

consultancy.

I’m also the organizer of DevOpsMtl, a monthly event to discuss the

challenges, tools and the culture surrounding the movement.

My main fields of expertise are web development with Ruby on Rails, DevOps

and a bit of mobile development. If you need help scaling, monitoring,

securing or managing your web infrastructure (Rails or not), get in touch!

Connect with me and read testimonials on my LinkedIn, follow me on twitter

@webmat, or check out my blog at programblings.com.

I’m the founder of Rock Solid Ops, a web operations and development consultancy.

I’m also the organizer of DevOpsMtl, a monthly event to discuss the challenges, tools and the culture surrounding the movement.

My main fields of expertise are web development with Ruby on Rails, DevOps and a bit of mobile development. If you need help scaling, monitoring, securing or managing your web infrastructure (Rails or not), get in touch!

Connect with me and read testimonials on my LinkedIn, follow me on twitter @webmat, or check out my blog at programblings.com.


#5

Hi Mathieu,

Fair enough. You've convinced me.

Thanks,

Mitsutoshi

···

2013/11/29 Mathieu Martin <webmat@gmail.com>:

Hi Mitsutoshi,

Fair point. Here's how I see it. The snowflake config is still ideally
configured by the config management tool, I agree. But instead it would
create the config file directly on the snowflake, as a standalone check.

I'm not a fan of having it on the Sensu server as one of the subscriptions,
because, suppose I have a "mysql" subscription for all my up to date DB
servers. Then out of my bunch of servers, I have 2 old MySQL snowflakes that
have different oddities going for them. I'd have to create a "mysql-oldapp1"
subscription only for one of the servers, and another "mysql-oldapp2"
subscription for the other server. That just seems like useless chatter
happening on RabbitMQ.

And actually it's good that you mention config management tools. I haven't
been in that situation recently, but sometimes the snowflakes aren't even
being configured with a tool like Chef... One more case where standalone
checks can be useful: a manual Sensu setup :slight_smile:

HTH,

Mat

On Thu, Nov 28, 2013 at 7:36 PM, Mitsutoshi Aoe <maoe@foldr.in> wrote:

Hi Mathieu,
Thanks for sharing your experience!

Given that we use Chef/Puupet to configure Sensu, why wouldn't you
like to put the irregular config into the sensu server?

Regards,

Mitsutoshi

Mitsutoshi Aoe
maoe@foldr.in

2013/11/28 Mathieu Martin <webmat@gmail.com>:
> Standalone checks are for special snowflakes, IMO. So you have swaths of
> servers you monitor more or less the same, for those the subscription
> model
> is perfect. But I use standalone checks for the janky old servers
> nobody's
> touched in ages, that's always running at 60% cpu and whatnot. You still
> want to monitor it, but you don't want to create a centralized config
> only
> for this one server.
>
> My 2¢ :slight_smile:
>
> Mat
>
>
> On Thu, Nov 28, 2013 at 12:04 AM, Mitsutoshi Aoe <maoe@foldr.in> wrote:
>>
>> Hi all,
>>
>> I'm wondering when standalone checks are useful. Initially I thought it
>> would increase availability such that clients could continue monitoring
>> even
>> if server is down. But it was not the case because handlers run on the
>> server anyway.
>>
>> One thing I've come up with is that they could be useful when the link
>> between the server and the client is narrow. In this case, we might
>> want to
>> save bandwidth by stopping triggering check events from the server. The
>> actual gain might be subtle, though.
>>
>> So my question is: when are standalone checks useful?
>>
>> Thanks,
>>
>> Mitsutoshi
>>
>
>
>
> --
>
> I'm the founder of Rock Solid Ops, a web operations and development
> consultancy.
>
> I'm also the organizer of DevOpsMtl, a monthly event to discuss the
> challenges, tools and the culture surrounding the movement.
>
> My main fields of expertise are web development with Ruby on Rails,
> DevOps
> and a bit of mobile development. If you need help scaling, monitoring,
> securing or managing your web infrastructure (Rails or not), get in
> touch!
>
> Connect with me and read testimonials on my LinkedIn, follow me on
> twitter
> @webmat, or check out my blog at programblings.com.

--

I'm the founder of Rock Solid Ops, a web operations and development
consultancy.

I'm also the organizer of DevOpsMtl, a monthly event to discuss the
challenges, tools and the culture surrounding the movement.

My main fields of expertise are web development with Ruby on Rails, DevOps
and a bit of mobile development. If you need help scaling, monitoring,
securing or managing your web infrastructure (Rails or not), get in touch!

Connect with me and read testimonials on my LinkedIn, follow me on twitter
@webmat, or check out my blog at programblings.com.


#6