Stash not working

Hello!

I’m having an issue with sensu stashes - I put something into a stash, via Uchiwa, and it says it’s stashed, but alerts are still firing.

I’ve checked the sensu-api.log, and I can see that the call is being made:

{“timestamp”:“2017-05-09T15:48:47.034799+1000”,“level”:“info”,“message”:“api response”,“request”:{“remote_address”:“127.0.0.1”,“user_agent”:“Go-http-client/1.1”,“method”:“POST”,“uri”:"/stashes",“query_string”:null,“body”:"{“dc”:“Sensu”,“path”:“silence/node01”,“content”:{“reason”:“testing, testing”,“source”:“uchiwa”,“timestamp”:1.494308963e+09,“username”:“admin”},“expire”:900}\n\n"},“status”:201,“content_length”:34,“time”:0.0}

It looks like it should be creating the stash properly, however I get no output whatsoever from “curl -s http://localhost:4567/stashes | jq .”.

This is a standalone sensu master, with sensu, redis, rabbitmq, and uchiwa all on the same master node, if that makes a difference.

If anybody could give me an idea of what’s going on here, I’d greatly appreciate it.

Thank you,

  • alan

Depending on what version of sensu you have and the exact setup, some handlers are not smart enough to respect silences like this.

But if you are getting literally no output, then yea it sounds like there is something wrong with the api.

Can you show a curl -v on the stashes endpoint? (the -s is for silent and is hurting your ability to see error codes)

And is it literally empty and not an empty array? ()

Don’t forget there is that 900 second expire time. Should be plenty of time to see something from the curl.

Another thing what would be helpful is looking at the log lines from sensu-server.log around the time that the handler is called.

···

On Mon, May 8, 2017 at 11:11 PM, alanpassel7@gmail.com wrote:

Hello!

I’m having an issue with sensu stashes - I put something into a stash, via Uchiwa, and it says it’s stashed, but alerts are still firing.

I’ve checked the sensu-api.log, and I can see that the call is being made:

{“timestamp”:“2017-05-09T15:48:47.034799+1000”,“level”:“info”,“message”:“api response”,“request”:{“remote_address”:“127.0.0.1”,“user_agent”:“Go-http-client/1.1”,“method”:“POST”,“uri”:"/stashes",“query_string”:null,“body”:"{“dc”:“Sensu”,“path”:“silence/node01”,“content”:{“reason”:“testing, testing”,“source”:“uchiwa”,“timestamp”:1.494308963e+09,“username”:“admin”},“expire”:900}\n\n"},“status”:201,“content_length”:34,“time”:0.0}

It looks like it should be creating the stash properly, however I get no output whatsoever from “curl -s http://localhost:4567/stashes | jq .”.

This is a standalone sensu master, with sensu, redis, rabbitmq, and uchiwa all on the same master node, if that makes a difference.

If anybody could give me an idea of what’s going on here, I’d greatly appreciate it.

Thank you,

  • alan

Hi Kyle, and thanks for the response.

First off, this is a little embarrassing, but using -v showed that I wasn’t authenticating. Derp.

So, now that I’m properly doing so, I’m getting the following output:

[{“path”:“silence/host”,“content”:{“reason”:“testing”,“source”:“uchiwa”,“timestamp”:1494376285.0,“username”:“admin”},“expire”:870}]

From the sensu-api log, I got this:

{“timestamp”:“2017-05-10T10:30:48.487156+1000”,“level”:“info”,“message”:“api response”,“request”:{“remote_address”:“127.0.0.1”,“user_agent”:“Go-http-client/1.1”,“method”:“POST”,“uri”:"/stashes",“query_string”:null,“body”:"{“dc”:“Sensu”,“path”:“silence/host”,“content”:{“reason”:“testing”,“source”:“uchiwa”,“timestamp”:1.494376285e+09,“username”:“admin”},“expire”:900}\n\n"},“status”:201,“content_length”:34,“time”:0.0}

Nothing appeared in the sensu-server log that matched up with this. Should I expect some output there?

We’re using the current, or very close to the current, version of Sensu for the master (0.29.0-7), and some of the clients are new, some are 0.20.3. But this worked flawlessly before we upgraded to the current one, and tbh that is what I’d expect of this. We are using some of our own plugins, but none of them are doing anything fancy, just if something meets this condition, fire off an alert.

We do use a non-standard handler for some nodes (Opsgenie), and the default for the rest, but I don’t see why that would interfere with silencing/stashing a node.

Thank you,

  • alan
···

On Wednesday, May 10, 2017 at 12:51:27 AM UTC+10, Kyle Anderson wrote:

Depending on what version of sensu you have and the exact setup, some handlers are not smart enough to respect silences like this.

But if you are getting literally no output, then yea it sounds like there is something wrong with the api.

Can you show a curl -v on the stashes endpoint? (the -s is for silent and is hurting your ability to see error codes)

And is it literally empty and not an empty array? ()

Don’t forget there is that 900 second expire time. Should be plenty of time to see something from the curl.

Another thing what would be helpful is looking at the log lines from sensu-server.log around the time that the handler is called.

On Mon, May 8, 2017 at 11:11 PM, alanp...@gmail.com wrote:

Hello!

I’m having an issue with sensu stashes - I put something into a stash, via Uchiwa, and it says it’s stashed, but alerts are still firing.

I’ve checked the sensu-api.log, and I can see that the call is being made:

{“timestamp”:“2017-05-09T15:48:47.034799+1000”,“level”:“info”,“message”:“api response”,“request”:{“remote_address”:“127.0.0.1”,“user_agent”:“Go-http-client/1.1”,“method”:“POST”,“uri”:"/stashes",“query_string”:null,“body”:"{“dc”:“Sensu”,“path”:“silence/node01”,“content”:{“reason”:“testing, testing”,“source”:“uchiwa”,“timestamp”:1.494308963e+09,“username”:“admin”},“expire”:900}\n\n"},“status”:201,“content_length”:34,“time”:0.0}

It looks like it should be creating the stash properly, however I get no output whatsoever from “curl -s http://localhost:4567/stashes | jq .”.

This is a standalone sensu master, with sensu, redis, rabbitmq, and uchiwa all on the same master node, if that makes a difference.

If anybody could give me an idea of what’s going on here, I’d greatly appreciate it.

Thank you,

  • alan

You can read about the changes to event filtering (silences, occurences, refresh, etc) with the newer version (>0.26) of sensu:

https://blog.sensuapp.org/deprecating-event-filtering-in-sensu-plugin-b60c7c500be3

I’ve been procrastinating in upgrading because I know it is will break some of my stuff.

https://github.com/Yelp/sensu_handlers/issues/102

In theory the sensu-server (not the handlers anymore) are supposed to be doing the filtering way early.

Is stashed-silence filtering just built in to the enterprise version?

https://sensuapp.org/docs/0.29/enterprise/filters/silence-stashes.html

I don’t know at this point :frowning:

Here are some more docs about this deprecation:

https://sensuapp.org/docs/0.29/reference/silencing.html#appendix-deprecated-stash-based-silencing

Is it possible you just need to upgrade uchiwa to >= 0.18?

···

On Tue, May 9, 2017 at 5:44 PM, alanpassel7@gmail.com wrote:

Hi Kyle, and thanks for the response.

First off, this is a little embarrassing, but using -v showed that I wasn’t authenticating. Derp.

So, now that I’m properly doing so, I’m getting the following output:

[{“path”:“silence/host”,“content”:{“reason”:“testing”,“source”:“uchiwa”,“timestamp”:1494376285.0,“username”:“admin”},“expire”:870}]

From the sensu-api log, I got this:

{“timestamp”:“2017-05-10T10:30:48.487156+1000”,“level”:“info”,“message”:“api response”,“request”:{“remote_address”:“127.0.0.1”,“user_agent”:“Go-http-client/1.1”,“method”:“POST”,“uri”:"/stashes",“query_string”:null,“body”:"{“dc”:“Sensu”,“path”:“silence/host”,“content”:{“reason”:“testing”,“source”:“uchiwa”,“timestamp”:1.494376285e+09,“username”:“admin”},“expire”:900}\n\n"},“status”:201,“content_length”:34,“time”:0.0}

Nothing appeared in the sensu-server log that matched up with this. Should I expect some output there?

We’re using the current, or very close to the current, version of Sensu for the master (0.29.0-7), and some of the clients are new, some are 0.20.3. But this worked flawlessly before we upgraded to the current one, and tbh that is what I’d expect of this. We are using some of our own plugins, but none of them are doing anything fancy, just if something meets this condition, fire off an alert.

We do use a non-standard handler for some nodes (Opsgenie), and the default for the rest, but I don’t see why that would interfere with silencing/stashing a node.

Thank you,

  • alan

On Wednesday, May 10, 2017 at 12:51:27 AM UTC+10, Kyle Anderson wrote:

Depending on what version of sensu you have and the exact setup, some handlers are not smart enough to respect silences like this.

But if you are getting literally no output, then yea it sounds like there is something wrong with the api.

Can you show a curl -v on the stashes endpoint? (the -s is for silent and is hurting your ability to see error codes)

And is it literally empty and not an empty array? ()

Don’t forget there is that 900 second expire time. Should be plenty of time to see something from the curl.

Another thing what would be helpful is looking at the log lines from sensu-server.log around the time that the handler is called.

On Mon, May 8, 2017 at 11:11 PM, alanp...@gmail.com wrote:

Hello!

I’m having an issue with sensu stashes - I put something into a stash, via Uchiwa, and it says it’s stashed, but alerts are still firing.

I’ve checked the sensu-api.log, and I can see that the call is being made:

{“timestamp”:“2017-05-09T15:48:47.034799+1000”,“level”:“info”,“message”:“api response”,“request”:{“remote_address”:“127.0.0.1”,“user_agent”:“Go-http-client/1.1”,“method”:“POST”,“uri”:"/stashes",“query_string”:null,“body”:"{“dc”:“Sensu”,“path”:“silence/node01”,“content”:{“reason”:“testing, testing”,“source”:“uchiwa”,“timestamp”:1.494308963e+09,“username”:“admin”},“expire”:900}\n\n"},“status”:201,“content_length”:34,“time”:0.0}

It looks like it should be creating the stash properly, however I get no output whatsoever from “curl -s http://localhost:4567/stashes | jq .”.

This is a standalone sensu master, with sensu, redis, rabbitmq, and uchiwa all on the same master node, if that makes a difference.

If anybody could give me an idea of what’s going on here, I’d greatly appreciate it.

Thank you,

  • alan

Ok, digging into that first link, looks like that answers my questions…they’ve changed how handlers are dealt with.

Thanks Kyle, I appreciate the help.

···

On Wednesday, May 10, 2017 at 12:36:13 PM UTC+10, Kyle Anderson wrote:

You can read about the changes to event filtering (silences, occurences, refresh, etc) with the newer version (>0.26) of sensu:

https://blog.sensuapp.org/deprecating-event-filtering-in-sensu-plugin-b60c7c500be3

I’ve been procrastinating in upgrading because I know it is will break some of my stuff.

https://github.com/Yelp/sensu_handlers/issues/102

In theory the sensu-server (not the handlers anymore) are supposed to be doing the filtering way early.

Is stashed-silence filtering just built in to the enterprise version?

https://sensuapp.org/docs/0.29/enterprise/filters/silence-stashes.html

I don’t know at this point :frowning:

Here are some more docs about this deprecation:

https://sensuapp.org/docs/0.29/reference/silencing.html#appendix-deprecated-stash-based-silencing

Is it possible you just need to upgrade uchiwa to >= 0.18?

On Tue, May 9, 2017 at 5:44 PM, alanp...@gmail.com wrote:

Hi Kyle, and thanks for the response.

First off, this is a little embarrassing, but using -v showed that I wasn’t authenticating. Derp.

So, now that I’m properly doing so, I’m getting the following output:

[{“path”:“silence/host”,“content”:{“reason”:“testing”,“source”:“uchiwa”,“timestamp”:1494376285.0,“username”:“admin”},“expire”:870}]

From the sensu-api log, I got this:

{“timestamp”:“2017-05-10T10:30:48.487156+1000”,“level”:“info”,“message”:“api response”,“request”:{“remote_address”:“127.0.0.1”,“user_agent”:“Go-http-client/1.1”,“method”:“POST”,“uri”:"/stashes",“query_string”:null,“body”:"{“dc”:“Sensu”,“path”:“silence/host”,“content”:{“reason”:“testing”,“source”:“uchiwa”,“timestamp”:1.494376285e+09,“username”:“admin”},“expire”:900}\n\n"},“status”:201,“content_length”:34,“time”:0.0}

Nothing appeared in the sensu-server log that matched up with this. Should I expect some output there?

We’re using the current, or very close to the current, version of Sensu for the master (0.29.0-7), and some of the clients are new, some are 0.20.3. But this worked flawlessly before we upgraded to the current one, and tbh that is what I’d expect of this. We are using some of our own plugins, but none of them are doing anything fancy, just if something meets this condition, fire off an alert.

We do use a non-standard handler for some nodes (Opsgenie), and the default for the rest, but I don’t see why that would interfere with silencing/stashing a node.

Thank you,

  • alan

On Wednesday, May 10, 2017 at 12:51:27 AM UTC+10, Kyle Anderson wrote:

Depending on what version of sensu you have and the exact setup, some handlers are not smart enough to respect silences like this.

But if you are getting literally no output, then yea it sounds like there is something wrong with the api.

Can you show a curl -v on the stashes endpoint? (the -s is for silent and is hurting your ability to see error codes)

And is it literally empty and not an empty array? ()

Don’t forget there is that 900 second expire time. Should be plenty of time to see something from the curl.

Another thing what would be helpful is looking at the log lines from sensu-server.log around the time that the handler is called.

On Mon, May 8, 2017 at 11:11 PM, alanp...@gmail.com wrote:

Hello!

I’m having an issue with sensu stashes - I put something into a stash, via Uchiwa, and it says it’s stashed, but alerts are still firing.

I’ve checked the sensu-api.log, and I can see that the call is being made:

{“timestamp”:“2017-05-09T15:48:47.034799+1000”,“level”:“info”,“message”:“api response”,“request”:{“remote_address”:“127.0.0.1”,“user_agent”:“Go-http-client/1.1”,“method”:“POST”,“uri”:"/stashes",“query_string”:null,“body”:"{“dc”:“Sensu”,“path”:“silence/node01”,“content”:{“reason”:“testing, testing”,“source”:“uchiwa”,“timestamp”:1.494308963e+09,“username”:“admin”},“expire”:900}\n\n"},“status”:201,“content_length”:34,“time”:0.0}

It looks like it should be creating the stash properly, however I get no output whatsoever from “curl -s http://localhost:4567/stashes | jq .”.

This is a standalone sensu master, with sensu, redis, rabbitmq, and uchiwa all on the same master node, if that makes a difference.

If anybody could give me an idea of what’s going on here, I’d greatly appreciate it.

Thank you,

  • alan