Subscription check flags bleeding over into other subscriptions

John · October 31, 2014, 7:51pm

I have run into an issue where I have a check such as one that checks for numbers of processors on the node that would have a general syntax such as the following for a subscription.
cat conf.d/check-ivy.json
…
“check-corecount”: {
“handlers”: [“default”, “mailer”],
“command”: “/etc/sensu/plugins/common/check-corecount.sh -w 18 -c 10”,
“interval”: 60,
“subscribers”: [ “ivy” ]
},
…

Where as we would also have one for some sandy bridge nodes
…
“check-corecount”: {
“handlers”: [“default”, “mailer”],
“command”: “/etc/sensu/plugins/common/check-corecount.sh -w 15 -c 8”,
“interval”: 60,
“subscribers”: [ “sandy” ]
},
…

However any node that is subscribed to the sandy subscription also gets the ivy subscription and of course the ivy flags for the check-corecount.sh script.

This hasn’t been an issue in the past few months that sensu has been running and was wondering if this is expected behaviour from sensu or perhaps a symptom of something else.

Thanks,
JB

Kyle_Anderson · November 1, 2014, 3:42am

It surprises me that the server would allow you to define two checks
with the same name and it would work.
Sensu loads up all the checks into one big hash.

The only other thing that reminds me of this is:

github.com/sensu/sensu

I had to restart the client to refresh the handlers for a check

opened 11:06PM - 22 Oct 14 UTC

closed 11:17PM - 09 Apr 16 UTC

FlorinAndrei

``` $ cat /etc/sensu/conf.d/check_foobar.json { "checks": { "foo_cp…u": { "handlers": ["default", "pagerduty"], "pager_team": "pd_foobar_metrics", "command": "check-stats.rb -h localhost:8080 -t 'foobar.*.*.*.*.os.cpu.user' -w 30 -c 50 -p -5mins", "interval": 60, "subscribers": [ "foometrics" ] } } } ``` sensu-0.14.0 on Ubuntu 14.04, installed from omnibus apt package. Server, client, api, redis, rabbitmq, and graphite, are all on the same machine (I also have a bunch of sensu clients in other places connecting back here). All sensu config (client, server, api) is in /etc/sensu/conf.d. check-stats.rb referenced in the JSON above checks metrics in graphite (-t is the graphite path). For that check I changed the second handler from "pagerduty" to "slack", and I restarted sensu-server. Then I triggered a high CPU condition on another machine and waited. In due time, the check triggered, but the alert was sent to pagerduty. I tried a few more times, but got same result. Only after I restarted sensu-client on this machine did the alert go to slack. Isn't this definition server-related? Shouldn't its behavior change when I restart the server? Why did it change only when restarting the client?

···

On Fri, Oct 31, 2014 at 12:51 PM, John B <wilshire461@gmail.com> wrote:

I have run into an issue where I have a check such as one that checks for
numbers of processors on the node that would have a general syntax such as
the following for a subscription.
cat conf.d/check-ivy.json
...
    "check-corecount": {
      "handlers": ["default", "mailer"],
      "command": "/etc/sensu/plugins/common/check-corecount.sh -w 18 -c 10",
      "interval": 60,
      "subscribers": [ "ivy" ]
      },
...

Where as we would also have one for some sandy bridge nodes
...
    "check-corecount": {
      "handlers": ["default", "mailer"],
      "command": "/etc/sensu/plugins/common/check-corecount.sh -w 15 -c 8",
      "interval": 60,
      "subscribers": [ "sandy" ]
      },
...

However any node that is subscribed to the sandy subscription also gets the
ivy subscription and of course the ivy flags for the check-corecount.sh
script.

This hasn't been an issue in the past few months that sensu has been running
and was wondering if this is expected behaviour from sensu or perhaps a
symptom of something else.

Thanks,
JB

John · November 2, 2014, 9:53pm

I was thinking it in the way that why would you bother to have two exact same checks that do the exact same thing defined twice.

I read into a few other issues connected with #806 and it looks as though just defining some ruby variables that then feed into the check should work? Seems very silly to have multiple copies of the same check that one has for each and every subscription.

···

On Friday, October 31, 2014 10:42:13 PM UTC-5, Kyle Anderson wrote:

It surprises me that the server would allow you to define two checks

with the same name and it would work.

Sensu loads up all the checks into one big hash.

The only other thing that reminds me of this is:

https://github.com/sensu/sensu/issues/806#issuecomment-60816339

Kyle_Anderson · November 3, 2014, 1:11am

Sensu-server is doing a big merge on all the json to load in the config.

github.com

sensu/sensu-settings/blob/95fcb0aa217a10a5deb007cdf05eec8273e5aee0/lib/sensu/settings/loader.rb#L91


      
          
          # Load settings from a JSON file.
          #
          # @param [String] file path.
          def load_file(file)
            if File.file?(file) && File.readable?(file)
              begin
                warning("loading config file", :file => file)
                contents = IO.read(file)
                config = MultiJson.load(contents, :symbolize_keys => true)
                merged = deep_merge(@settings, config)
                unless @loaded_files.empty?
                  changes = deep_diff(@settings, merged)
                  warning("config file applied changes", {
                    :file => file,
                    :changes => changes
                  })
                end
                @settings = merged
                @indifferent_access = false
                @loaded_files << file

Then, for each check it goes through the hash and schedules them:

github.com

sensu/sensu/blob/cb7780f268ee3b74d06156e0481ed1e4c7d1e7db/lib/sensu/server.rb#L512


      
                    :error => info[:error].to_s
                  })
                end
              end
            end
          end
          
          def schedule_checks(checks)
            check_count = 0
            stagger = testing? ? 0 : 2
            checks.each do |check|
              check_count += 1
              scheduling_delay = stagger * check_count % 30
              @timers[:master] << EM::Timer.new(scheduling_delay) do
                interval = testing? ? 0.5 : check[:interval]
                @timers[:master] << EM::PeriodicTimer.new(interval) do
                  unless check_request_subdued?(check)
                    publish_check_request(check)
                  else
                    @logger.info('check request was subdued', {
                      :check => check

When it schedules "check-corecount".... what will the contents of the
hash be? I'm pretty sure it will be the merge of both, the last file
read will win.

It doesn't seem silly to me to have multiple copies of this check, you
need check-corecount-sandy and check-corecount-ivy.
It's not about having different copies for each subscription, it is
about having different copies for each *config*.

Personally, I like the standalone module, check-corecount could be
deployed client-side to your nodes, and they would have different
definitions depending on the client. (as opposed to the
publish/subscribe module)

···

On Sun, Nov 2, 2014 at 1:53 PM, John B <wilshire461@gmail.com> wrote:

I was thinking it in the way that why would you bother to have two exact
same checks that do the exact same thing defined twice.

I read into a few other issues connected with #806 and it looks as though
just defining some ruby variables that then feed into the check should work?
Seems very silly to have multiple copies of the same check that one has for
each and every subscription.

On Friday, October 31, 2014 10:42:13 PM UTC-5, Kyle Anderson wrote:

It surprises me that the server would allow you to define two checks
with the same name and it would work.
Sensu loads up all the checks into one big hash.

The only other thing that reminds me of this is:
I had to restart the client to refresh the handlers for a check · Issue #806 · sensu/sensu · GitHub

John · November 3, 2014, 4:35am

https://github.com/sensu/sensu-settings/blob/95fcb0aa217a10a5deb007cdf05eec8273e5aee0/lib/sensu/settings/loader.rb#L91

Then, for each check it goes through the hash and schedules them:

https://github.com/sensu/sensu/blob/cb7780f268ee3b74d06156e0481ed1e4c7d1e7db/lib/sensu/server.rb#L512

When it schedules “check-corecount”… what will the contents of the

hash be? I’m pretty sure it will be the merge of both, the last file

read will win.

Sensu-server is doing a big merge on all the json to load in the config.

Thanks for the source references.

It doesn’t seem silly to me to have multiple copies of this check, you

need check-corecount-sandy and check-corecount-ivy.

It’s not about having different copies for each subscription, it is

about having different copies for each config.

Personally, I like the standalone module, check-corecount could be

deployed client-side to your nodes, and they would have different

definitions depending on the client. (as opposed to the

publish/subscribe module)

I forgot to even think about the standalone check path. Thank you for all the help!

Topic		Replies	Views
Check shows up under "Checks", but not in subscription Sensu Classic (EOL)	4	758	November 22, 2018
[Issue] Sensu Checks: Multiple check results to every command execution Sensu Classic (EOL)	3	905	November 22, 2018
sensu not really sudoing for checks? Sensu Classic (EOL)	4	432	November 22, 2018
Problem with subscription based checks not getting updated when their definition change Sensu Classic (EOL)	1	732	November 22, 2018
A standalone check with subscribers, will it work ? Sensu Classic (EOL)	1	453	December 27, 2015

Subscription check flags bleeding over into other subscriptions

Related Topics