Hi,
I’m wanting to use sensu in a multi-tenant environment. Given that use case, how is it that one can list all the events specific to a particular tenant.
Also, is it possible to have thresholds defined for external results sent to sensu on port 3030. More like can i specify the thresholds for specific checks via a check definition on the client/server.
Thanks,
Akshay
Hi,
I'm wanting to use sensu in a multi-tenant environment. Given that use case,
how is it that one can list all the events specific to a particular tenant.
There is no built-in functionality for sensu for that, and probably won't be.
However, you can add arbitrary key values into the event data and use
that to filter based on different tenants.
We use that here:
https://github.com/Yelp/monitoring_check
Once this tenant data is in the event json, you can write your own
tools to filter.
(sensu-cli event list -f json | jq ".tenant = 'your tenant'")
You also might want to look at http://flapjack.io/
As it is designed to be multi-tenant, and Sensu can feed it.
Also, is it possible to have thresholds defined for external results sent to
sensu on port 3030. More like can i specify the thresholds for specific
checks via a check definition on the client/server.
What kind of thresholds?
If it is data for the handler to consume (like occurences/refresh), then Yes.
If it is for keepalive thresholds, then No (but there is a PR
https://github.com/sensu/sensu/pull/910\)
Yes, you can specify specific check thresholds (occurences/refresh) by
modifying the check definition.
If you are doing this on the client, it must be a standalone check.
(otherwise the check as defined on the server will take precedence.)
On Tue, Mar 31, 2015 at 11:30 AM, Akshay Kapoor <akshay.anilkapoor@gmail.com> wrote:
Thanks Kyle for your explanation. I have a couple of questions based on what you’ve said. It is a bit length as i tried being as verbose as i can so as to avoid confusion.
In the sensu-handlers project of yelp, if i understand correctly, the team-name is the tenant-name per say for the multi-tenant model. So the client-name remains the sensu-client name and the team-name gets added to the event data
How can you list all events based on per tenant from the currently exposed sensu APIs or you guys use something else for that?
Based on my use-case which is to be able to list out all the events per tenant and query the same, looking at the sensu code i figured the following,
For the multi-tenant i believe, since redis stores the key:value pair as client-name: where client name is the sensu-client hostname. To have for example all active events from each tenant and to be able to use the existing APIs exposed by sensu (because that would be needed by some application in my case, to get all the events wrt tenant) i suppose i would need to store them with tenant-name: (Not sure if i could do that without requiring to fork sensu and manage stuff [maintain, rebase sensu for all new versions)] by myself, probably that is not the desired approach i think)
Regarding the external check-results to be sent to sensu, i wanted to confirm if what i’m thinking is correct and just want you to confirm for correctness
I have some events in logstash which needs to be sent out to sensu for raising a notification. For this i send it to the localhost port 3030 that sensu-client is listening on (Not sure if this is the best approach to get events from logs; maybe you could suggest improvements if any).
When this is sent out to sensu, it sends a notification using a handler as defined in the event in logstash (as there is no check definition defined on the sensu-server for this, i cannot add threshold values for the handler to use, may be i can just set that in the logstash event to have it available as a part of the event data)
So is it possible to have check-definitions on the sensu-server or client for external results sent to the sensu-client on port 3030 (because i’m trying to get in events from logstash)
Thanks,
Akshay
P.S - Sorry for being so verbose, it’s because of my little knowledge on sensu (trying to catch up on it with help from people like you)
On Tue, Mar 31, 2015 at 6:48 PM, Kyle Anderson kyle@xkyle.com wrote:
On Tue, Mar 31, 2015 at 11:30 AM, Akshay Kapoor
akshay.anilkapoor@gmail.com wrote:
Hi,
I’m wanting to use sensu in a multi-tenant environment. Given that use case,
how is it that one can list all the events specific to a particular tenant.
There is no built-in functionality for sensu for that, and probably won’t be.
However, you can add arbitrary key values into the event data and use
that to filter based on different tenants.
We use that here:
https://github.com/Yelp/sensu_handlers
https://github.com/Yelp/monitoring_check
Once this tenant data is in the event json, you can write your own
tools to filter.
(sensu-cli event list -f json | jq “.tenant = ‘your tenant’”)
You also might want to look at http://flapjack.io/
As it is designed to be multi-tenant, and Sensu can feed it.
Also, is it possible to have thresholds defined for external results sent to
sensu on port 3030. More like can i specify the thresholds for specific
checks via a check definition on the client/server.
What kind of thresholds?
If it is data for the handler to consume (like occurences/refresh), then Yes.
If it is for keepalive thresholds, then No (but there is a PR
https://github.com/sensu/sensu/pull/910)
Yes, you can specify specific check thresholds (occurences/refresh) by
modifying the check definition.
If you are doing this on the client, it must be a standalone check.
(otherwise the check as defined on the server will take precedence.)
Thanks Kyle for your explanation. I have a couple of questions based on what
you've said. It is a bit length as i tried being as verbose as i can so as
to avoid confusion.In the sensu-handlers project of yelp, if i understand correctly, the
team-name is the tenant-name per say for the multi-tenant model. So the
client-name remains the sensu-client name and the team-name gets added to
the event data
How can you list all events based on per tenant from the currently exposed
sensu APIs or you guys use something else for that?
Correct.
We just use uchiwa and type in the team name in the search box.
Based on my use-case which is to be able to list out all the events per
tenant and query the same, looking at the sensu code i figured the
following,For the multi-tenant i believe, since redis stores the key:value pair as
client-name:<value> where client name is the sensu-client hostname. To have
for example all active events from each tenant and to be able to use the
existing APIs exposed by sensu (because that would be needed by some
application in my case, to get all the events wrt tenant) i suppose i would
need to store them with tenant-name:<value> (Not sure if i could do that
without requiring to fork sensu and manage stuff [maintain, rebase sensu for
all new versions)] by myself, probably that is not the desired approach i
think)
You can add tenant-name right now without forking Sensu.
I strongly recommend just using the flexibility of being able to
provide arbitrary key-value pairs to inject the data you need.
Then, I think it would be easiest to write tools around that, or heck
just an http frontend that wraps the api as you need:
# http://some-sensu-api-filter-service?tenant=foo
raw_api_json = fetch_sensu_event_api()
for event in raw_api_json:
if event['tenant'] == tenant #foo here
print event
Then you can use standard dashboards and get a customized view?
Obviously this is simplifying things.
Also flapjack.io does some of this.
Regarding the external check-results to be sent to sensu, i wanted to
confirm if what i'm thinking is correct and just want you to confirm for
correctnessI have some events in logstash which needs to be sent out to sensu for
raising a notification. For this i send it to the localhost port 3030 that
sensu-client is listening on (Not sure if this is the best approach to get
events from logs; maybe you could suggest improvements if any).When this is sent out to sensu, it sends a notification using a handler as
defined in the event in logstash (as there is no check definition defined on
the sensu-server for this, i cannot add threshold values for the handler to
use, may be i can just set that in the logstash event to have it available
as a part of the event data)So is it possible to have check-definitions on the sensu-server or client
for external results sent to the sensu-client on port 3030 (because i'm
trying to get in events from logstash)
Yes, you can send any arbitrary key-value pair to port 3030, just like
a check definition. (and yes, without a check definition, sensu
doesn't care)
Including tenant, refresh, occurrences. In Yelp's case, we do "team",
"page", "ticket", etc.
Isn't Sensu liberating? You can just like, "send alerts" for whatever? 
On Wed, Apr 1, 2015 at 12:11 PM, Akshay Kapoor <akshay.anilkapoor@gmail.com> wrote:
Thanks,
AkshayP.S - Sorry for being so verbose, it's because of my little knowledge on
sensu (trying to catch up on it with help from people like you)On Tue, Mar 31, 2015 at 6:48 PM, Kyle Anderson <kyle@xkyle.com> wrote:
On Tue, Mar 31, 2015 at 11:30 AM, Akshay Kapoor >> <akshay.anilkapoor@gmail.com> wrote:
> Hi,
>
> I'm wanting to use sensu in a multi-tenant environment. Given that use
> case,
> how is it that one can list all the events specific to a particular
> tenant.
There is no built-in functionality for sensu for that, and probably won't
be.
However, you can add arbitrary key values into the event data and use
that to filter based on different tenants.
We use that here:
GitHub - Yelp/sensu_handlers: Custom Sensu Handlers to support a multi-tenant environment, allowing checks themselves to emit the type of handler behavior they need in the event json
https://github.com/Yelp/monitoring_checkOnce this tenant data is in the event json, you can write your own
tools to filter.
(sensu-cli event list -f json | jq ".tenant = 'your tenant'")You also might want to look at http://flapjack.io/
As it is designed to be multi-tenant, and Sensu can feed it.> Also, is it possible to have thresholds defined for external results
> sent to
> sensu on port 3030. More like can i specify the thresholds for specific
> checks via a check definition on the client/server.What kind of thresholds?
If it is data for the handler to consume (like occurences/refresh), then
Yes.
If it is for keepalive thresholds, then No (but there is a PR
https://github.com/sensu/sensu/pull/910\)Yes, you can specify specific check thresholds (occurences/refresh) by
modifying the check definition.
If you are doing this on the client, it must be a standalone check.
(otherwise the check as defined on the server will take precedence.)
Kyle thanks, that makes things a bit clear to me. But still don’t seem to understand one thing from this which is how things will be stored in redis. Apologies Kyle if i’m repeating a few things as i just want to be able to understand clearly.
Just for the sake of discussion. I have an event generated from one of the tenant which gets discovered by logstash and is sent to sensu along with the tenant-name/team-name in the result.
This would create an entry in redis with event: (check-name would be tenant-name/team-name) if this tenant is not present in redis. This can be verified by looking at the following code lines in sensu,
https://github.com/sensu/sensu/blob/master/lib/sensu/server/process.rb#L303-319
Now, if another event comes from the same tenant, the way sensu would process it would be like,
since event: will already be present from the first event…it would just update the the event by increasing the number of occurrences. (That is because the field is already present in the hash event:)
One way to get around this i feel would be to have the check-names be a combination of tenant-name:event-name
But now when i think i have to get all active events for a particular tenant, i’m not sure how can i query it in redis to view the same.
I was playing around with redis-cli to see how can i filter events based on tenant or team-name, but with no luck.
I hope i’m making sense.
Appreciate your time and help!
Thanks
On Thu, Apr 2, 2015 at 8:36 AM, Kyle Anderson kyle@xkyle.com wrote:
On Wed, Apr 1, 2015 at 12:11 PM, Akshay Kapoor
akshay.anilkapoor@gmail.com wrote:
Thanks Kyle for your explanation. I have a couple of questions based on what
you’ve said. It is a bit length as i tried being as verbose as i can so as
to avoid confusion.
In the sensu-handlers project of yelp, if i understand correctly, the
team-name is the tenant-name per say for the multi-tenant model. So the
client-name remains the sensu-client name and the team-name gets added to
the event data
How can you list all events based on per tenant from the currently exposed
sensu APIs or you guys use something else for that?
Correct.
We just use uchiwa and type in the team name in the search box.
Based on my use-case which is to be able to list out all the events per
tenant and query the same, looking at the sensu code i figured the
following,
For the multi-tenant i believe, since redis stores the key:value pair as
client-name: where client name is the sensu-client hostname. To have
for example all active events from each tenant and to be able to use the
existing APIs exposed by sensu (because that would be needed by some
application in my case, to get all the events wrt tenant) i suppose i would
need to store them with tenant-name: (Not sure if i could do that
without requiring to fork sensu and manage stuff [maintain, rebase sensu for
all new versions)] by myself, probably that is not the desired approach i
think)
You can add tenant-name right now without forking Sensu.
I strongly recommend just using the flexibility of being able to
provide arbitrary key-value pairs to inject the data you need.
Then, I think it would be easiest to write tools around that, or heck
just an http frontend that wraps the api as you need:
http://some-sensu-api-filter-service?tenant=foo
raw_api_json = fetch_sensu_event_api()
for event in raw_api_json:
if event['tenant'] == tenant #foo here print eventThen you can use standard dashboards and get a customized view?
Obviously this is simplifying things.
Also flapjack.io does some of this.
Regarding the external check-results to be sent to sensu, i wanted to
confirm if what i’m thinking is correct and just want you to confirm for
correctness
I have some events in logstash which needs to be sent out to sensu for
raising a notification. For this i send it to the localhost port 3030 that
sensu-client is listening on (Not sure if this is the best approach to get
events from logs; maybe you could suggest improvements if any).
When this is sent out to sensu, it sends a notification using a handler as
defined in the event in logstash (as there is no check definition defined on
the sensu-server for this, i cannot add threshold values for the handler to
use, may be i can just set that in the logstash event to have it available
as a part of the event data)
So is it possible to have check-definitions on the sensu-server or client
for external results sent to the sensu-client on port 3030 (because i’m
trying to get in events from logstash)
Yes, you can send any arbitrary key-value pair to port 3030, just like
a check definition. (and yes, without a check definition, sensu
doesn’t care)
Including tenant, refresh, occurrences. In Yelp’s case, we do “team”,
“page”, “ticket”, etc.
Isn’t Sensu liberating? You can just like, “send alerts” for whatever?
Thanks,
Akshay
P.S - Sorry for being so verbose, it’s because of my little knowledge on
sensu (trying to catch up on it with help from people like you)
On Tue, Mar 31, 2015 at 6:48 PM, Kyle Anderson kyle@xkyle.com wrote:
On Tue, Mar 31, 2015 at 11:30 AM, Akshay Kapoor
akshay.anilkapoor@gmail.com wrote:
Hi,
I’m wanting to use sensu in a multi-tenant environment. Given that use
case,
how is it that one can list all the events specific to a particular
tenant.
There is no built-in functionality for sensu for that, and probably won’t
be.
However, you can add arbitrary key values into the event data and use
that to filter based on different tenants.
We use that here:
Once this tenant data is in the event json, you can write your own
tools to filter.
(sensu-cli event list -f json | jq “.tenant = ‘your tenant’”)
You also might want to look at http://flapjack.io/
As it is designed to be multi-tenant, and Sensu can feed it.
Also, is it possible to have thresholds defined for external results
sent to
sensu on port 3030. More like can i specify the thresholds for specific
checks via a check definition on the client/server.
What kind of thresholds?
If it is data for the handler to consume (like occurences/refresh), then
Yes.
If it is for keepalive thresholds, then No (but there is a PR
Yes, you can specify specific check thresholds (occurences/refresh) by
modifying the check definition.
If you are doing this on the client, it must be a standalone check.
(otherwise the check as defined on the server will take precedence.)
Yes, if you are expecting check-name collisions on a host, namespacing
the check names by tenant sounds pretty sane.
I don't think you can use "/" as a separator, but yea, ":" or "-" or something.
I would still inject the team-name and tenant-name into the event json
to make parsing easier, and use this data as the "canonical" tags for
which events belong to who.
I would not bother with the redis-cli, to me that is too low-level.
Instead use the sensu-cli. You can use grep, or jq for simple
filtering while you are exploring the schema.
It would be something like
sensu-cli event list -f json | jq '. | select(.tenant == "tenant_1")'
On Thu, Apr 2, 2015 at 7:23 PM, Akshay Kapoor <akshay.anilkapoor@gmail.com> wrote:
Kyle thanks, that makes things a bit clear to me. But still don't seem to
understand one thing from this which is how things will be stored in redis.
Apologies Kyle if i'm repeating a few things as i just want to be able to
understand clearly.Just for the sake of discussion. I have an event generated from one of the
tenant which gets discovered by logstash and is sent to sensu along with the
tenant-name/team-name in the result.This would create an entry in redis with event:<client-name> <check-name>
(check-name would be tenant-name/team-name) if this tenant is not present in
redis. This can be verified by looking at the following code lines in sensu,https://github.com/sensu/sensu/blob/master/lib/sensu/server/process.rb#L303-319
Now, if another event comes from the same tenant, the way sensu would
process it would be like,
since event:<client-name> <tenant-name> will already be present from the
first event..it would just update the the event by increasing the number of
occurrences. (That is because the field <tenant-name> is already present in
the hash event:<client-name>)One way to get around this i feel would be to have the check-names be a
combination of <tenant-name:event-name>But now when i think i have to get all active events for a particular
tenant, i'm not sure how can i query it in redis to view the same.I was playing around with redis-cli to see how can i filter events based on
tenant or team-name, but with no luck.I hope i'm making sense.
Appreciate your time and help!
Thanks
On Thu, Apr 2, 2015 at 8:36 AM, Kyle Anderson <kyle@xkyle.com> wrote:
On Wed, Apr 1, 2015 at 12:11 PM, Akshay Kapoor >> <akshay.anilkapoor@gmail.com> wrote:
> Thanks Kyle for your explanation. I have a couple of questions based on
> what
> you've said. It is a bit length as i tried being as verbose as i can so
> as
> to avoid confusion.
>
> In the sensu-handlers project of yelp, if i understand correctly, the
> team-name is the tenant-name per say for the multi-tenant model. So the
> client-name remains the sensu-client name and the team-name gets added
> to
> the event data
> How can you list all events based on per tenant from the currently
> exposed
> sensu APIs or you guys use something else for that?
Correct.
We just use uchiwa and type in the team name in the search box.> Based on my use-case which is to be able to list out all the events per
> tenant and query the same, looking at the sensu code i figured the
> following,
>
> For the multi-tenant i believe, since redis stores the key:value pair as
> client-name:<value> where client name is the sensu-client hostname. To
> have
> for example all active events from each tenant and to be able to use the
> existing APIs exposed by sensu (because that would be needed by some
> application in my case, to get all the events wrt tenant) i suppose i
> would
> need to store them with tenant-name:<value> (Not sure if i could do that
> without requiring to fork sensu and manage stuff [maintain, rebase sensu
> for
> all new versions)] by myself, probably that is not the desired approach
> i
> think)
You can add tenant-name right now without forking Sensu.
I strongly recommend just using the flexibility of being able to
provide arbitrary key-value pairs to inject the data you need.
Then, I think it would be easiest to write tools around that, or heck
just an http frontend that wraps the api as you need:# http://some-sensu-api-filter-service?tenant=foo
raw_api_json = fetch_sensu_event_api()
for event in raw_api_json:
if event['tenant'] == tenant #foo here
print eventThen you can use standard dashboards and get a customized view?
Obviously this is simplifying things.
Also flapjack.io does some of this.>
>
> Regarding the external check-results to be sent to sensu, i wanted to
> confirm if what i'm thinking is correct and just want you to confirm for
> correctness
>
> I have some events in logstash which needs to be sent out to sensu for
> raising a notification. For this i send it to the localhost port 3030
> that
> sensu-client is listening on (Not sure if this is the best approach to
> get
> events from logs; maybe you could suggest improvements if any).
>
> When this is sent out to sensu, it sends a notification using a handler
> as
> defined in the event in logstash (as there is no check definition
> defined on
> the sensu-server for this, i cannot add threshold values for the handler
> to
> use, may be i can just set that in the logstash event to have it
> available
> as a part of the event data)
>
> So is it possible to have check-definitions on the sensu-server or
> client
> for external results sent to the sensu-client on port 3030 (because i'm
> trying to get in events from logstash)Yes, you can send any arbitrary key-value pair to port 3030, just like
a check definition. (and yes, without a check definition, sensu
doesn't care)
Including tenant, refresh, occurrences. In Yelp's case, we do "team",
"page", "ticket", etc.Isn't Sensu liberating? You can just like, "send alerts" for whatever?
>
> Thanks,
> Akshay
>
> P.S - Sorry for being so verbose, it's because of my little knowledge on
> sensu (trying to catch up on it with help from people like you)
>
>
> On Tue, Mar 31, 2015 at 6:48 PM, Kyle Anderson <kyle@xkyle.com> wrote:
>>
>> On Tue, Mar 31, 2015 at 11:30 AM, Akshay Kapoor >> >> <akshay.anilkapoor@gmail.com> wrote:
>> > Hi,
>> >
>> > I'm wanting to use sensu in a multi-tenant environment. Given that
>> > use
>> > case,
>> > how is it that one can list all the events specific to a particular
>> > tenant.
>> There is no built-in functionality for sensu for that, and probably
>> won't
>> be.
>> However, you can add arbitrary key values into the event data and use
>> that to filter based on different tenants.
>> We use that here:
>> GitHub - Yelp/sensu_handlers: Custom Sensu Handlers to support a multi-tenant environment, allowing checks themselves to emit the type of handler behavior they need in the event json
>> https://github.com/Yelp/monitoring_check
>>
>> Once this tenant data is in the event json, you can write your own
>> tools to filter.
>> (sensu-cli event list -f json | jq ".tenant = 'your tenant'")
>>
>> You also might want to look at http://flapjack.io/
>> As it is designed to be multi-tenant, and Sensu can feed it.
>>
>>
>> > Also, is it possible to have thresholds defined for external results
>> > sent to
>> > sensu on port 3030. More like can i specify the thresholds for
>> > specific
>> > checks via a check definition on the client/server.
>>
>> What kind of thresholds?
>> If it is data for the handler to consume (like occurences/refresh),
>> then
>> Yes.
>> If it is for keepalive thresholds, then No (but there is a PR
>> https://github.com/sensu/sensu/pull/910\)
>>
>> Yes, you can specify specific check thresholds (occurences/refresh) by
>> modifying the check definition.
>> If you are doing this on the client, it must be a standalone check.
>> (otherwise the check as defined on the server will take precedence.)
>
>