Getting check is not subdued error for any plugin checks that I try to make it work

Sensu Integrations Sensu Plugins
1

I am setting up new sensu-go cluster using GKE.

I have my basic cluster up and running. I have tried pagerduty and email handler to see it works and they are working. I wanted to try one of the checks to see if it runs automatically based on some interval and work with handler. I tried so far cpu and disk plugins.

Ref. github dot com slash sensu-plugins slash sensu-plugins-disk-checks and same way sensu-plugins-cpu-checks

But, for both, I get below errors and I am not able to fix it. I have googled it and none of the search results have helped me so far.

{“component”:“schedulerd”,“level”:“debug”,“msg”:“check is not subdued”,“name”:“check-disk”,“namespace”:“default”,“scheduler_type”:“interval”,“time”:“2020-09-23T01:26:16Z”}
{“check”:“check-disk”,“component”:“schedulerd”,“level”:“debug”,“msg”:“sending check request”,“time”:“2020-09-23T01:26:16Z”,“topic”:“sensu:check:default:linux”}
{“check”:“check-disk”,“component”:“schedulerd”,“level”:“debug”,“msg”:“sending check request”,“time”:“2020-09-23T01:26:16Z”,“topic”:“sensu:check:default:system”}

{“component”:“schedulerd”,“level”:“debug”,“msg”:“check is not subdued”,“name”:“check-cpu”,“namespace”:“default”,“scheduler_type”:“interval”,“time”:“2020-09-23T01:35:17Z”}
{“check”:“check-cpu”,“component”:“schedulerd”,“level”:“debug”,“msg”:“sending check request”,“time”:“2020-09-23T01:35:17Z”,“topic”:“sensu:check:default:system”}
{“check”:“check-cpu”,“component”:“schedulerd”,“level”:“debug”,“msg”:“sending check request”,“time”:“2020-09-23T01:35:17Z”,“topic”:“sensu:check:default:linux”}

Below is my cluster info: (i have sensu-backend and agent running on same container and i have 3 containers)

sensuctl entity list

    ID          Class    OS         Subscriptions                  Last Seen

───────────────── ─────── ─────── ──────────────────────── ───────────────────────────────
sensu-backend-0 agent linux entity:sensu-backend-0 2020-09-23 01:35:21 +0000 UTC
sensu-backend-1 agent linux entity:sensu-backend-1 2020-09-23 01:35:25 +0000 UTC
sensu-backend-2 agent linux entity:sensu-backend-2 2020-09-23 01:35:37 +0000 UTC

s asset list | awk ‘{print $1}’ | sort -u

──────────────────────────────────────
cpu-checks-plugins
disk-checks-plugins
email-handler
ncr-devops-platform/nagiosfoundation
pagerduty-handler
sensu-ruby-runtime

s handler list

Name      Type   Timeout           Filters            Mutator                                                                                                  Execute                                                                                                               Environment Variables                      Assets

─────────── ────── ───────── ────────────────────────── ───────── ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────── ───────────────────
email pipe 10 is_incident,not_silenced RUN: sensu-email-handler -f<> -t <> -s <> -u xxx -p xxx email-handler
pagerduty pipe 10 is_incident RUN: sensu-pagerduty-handler PAGERDUTY_TOKEN=xxx pagerduty-handler

rpm -qa | grep -i sensu

sensu-go-cli-6.0.0-3003.x86_64
sensu-go-backend-6.0.0-3003.x86_64
sensu-go-agent-6.0.0-3003.x86_64

My base container image is centos7.

uname -a

Linux sensu-backend-0 4.14.138+ #1 SMP Tue Sep 3 02:58:08 PDT 2019 x86_64 x86_64 x86_64 GNU/Linux

Please let me know if any more info is needed.
Thank you.
Prashant

2

I don’t believe that is the cause of your issues, that is simply a debug message. The remainder of your log messages state that it is attempting to schedule and then run those checks.

Can you please post the output from sensuctl check info check-cpu --format yaml and sensuctl check info check-disk --format yaml?

3

Thanks Todd for replying. Please find below the output of the CLIs you requested:

s check info check-cpu --format yaml

type: CheckConfig
api_version: core/v2
metadata:
created_by: admin
labels:
Sensu | Page not found sensuctl
name: check-cpu
namespace: default
spec:
check_hooks: null
command: check-cpu.rb -w 1 -c 2
env_vars: null
handlers:

  • email
    high_flap_threshold: 0
    interval: 10
    low_flap_threshold: 0
    output_metric_format: “”
    output_metric_handlers: null
    proxy_entity_name: “”
    publish: true
    round_robin: false
    runtime_assets:
  • sensu-plugins/sensu-plugins-cpu-checks
  • sensu/sensu-ruby-runtime
    secrets: null
    stdin: false
    subdue: null
    subscriptions:
  • system
  • linux
    timeout: 0
    ttl: 0

s check info check-disk --format yaml

type: CheckConfig
api_version: core/v2
metadata:
created_by: admin
labels:
Sensu | Page not found sensuctl
name: check-disk
namespace: default
spec:
check_hooks: null
command: metrics-disk-usage.rb -w 1 -c 2
env_vars: null
handlers:

  • email
    high_flap_threshold: 0
    interval: 10
    low_flap_threshold: 0
    output_metric_format: graphite_plaintext
    output_metric_handlers:
  • influx-db
    proxy_entity_name: “”
    publish: true
    round_robin: false
    runtime_assets:
  • sensu-plugins/sensu-plugins-disk-checks
  • sensu/sensu-ruby-runtime
    secrets: null
    stdin: false
    subdue: null
    subscriptions:
  • linux
  • system
    timeout: 0
    ttl: 0
4 
also, please note below output for event list which doesn't show that check-cpu or check-disk has run else it would show OK event for them. And in my previous output, seems like format has changed while pasting the output in the reply. Let me know if you need it again.

s event list

  Entity              Check                                           Output                                     Status   Silenced             Timestamp                             UUID

───────────────── ──────────────────── ─────────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── ──────────────────────────────────────
sensu-backend-0 check-email-status can it send email? 1 false 2020-09-21 19:28:32 +0000 UTC 0e4b1abe-6638-4722-809c-9173e417cb5f
sensu-backend-0 check-mysql-status could not connect to mysql 0 false 2020-09-18 23:51:34 +0000 UTC 6764a610-1eb6-46cc-b325-64e5169ac27f
sensu-backend-0 file_exists /tmp/my-file.txt doesnt exist 1 false 2020-09-21 04:54:41 +0000 UTC 7f94431e-3c3a-439a-9881-6785521f772f
sensu-backend-0 keepalive Keepalive last sent from sensu-backend-0 at 2020-09-23 17:24:01 +0000 UTC 0 false 2020-09-23 17:24:02 +0000 UTC cb438cbc-a727-4ee8-94ce-e76e7e48719b
sensu-backend-1 check-pd-status could not connect to mysql 1 false 2020-09-21 04:42:40 +0000 UTC b4ec7ad8-5bba-479e-8578-aa794c1be060
sensu-backend-1 file_exists /tmp/my-file.txt doesnt exist 1 false 2020-09-21 04:28:17 +0000 UTC a1e3163c-1bf2-4791-b265-a75836b97bd8
sensu-backend-1 keepalive Keepalive last sent from sensu-backend-1 at 2020-09-23 17:23:45 +0000 UTC 0 false 2020-09-23 17:23:45 +0000 UTC 94ad935e-e6f8-4cf5-a0f5-e1923cfc4ada
sensu-backend-2 check-email-status can it send email? 1 false 2020-09-21 05:35:27 +0000 UTC ed42697f-7013-408e-8ef1-a5402edae6bb
sensu-backend-2 check-mysql-status could not connect to mysql 1 false 2020-09-18 23:19:43 +0000 UTC 5ffd98b5-25e0-4a7c-b665-f04be9511575
sensu-backend-2 keepalive Keepalive last sent from sensu-backend-2 at 2020-09-23 17:23:57 +0000 UTC 0 false 2020-09-23 17:23:57 +0000 UTC 707cb236-e9f8-41fb-820f-92d27e658ccc

5

Those two checks are configured for the subscriptions “system” and “linux”. However none of your entities are subscribed to those. They are only subscribed to their “entity:” subscriptions.

6

Ah!, but how do I change that ? Is it /etc/sensu/backend.yml or /etc/sensu/agent.yml? Below are their content:

cat backend.yml

log-level: “debug”
cache-dir: “/var/cache/sensu/sensu-backend”
state-dir: “/var/lib/sensu”
api-url: “http://localhost:8080”
etcd-name: sensu-backend-0
etcd-initial-advertise-peer-urls: http colon slash slash sensu-backend-0.sensu.sensu.svc.cluster.local:2380
etcd-initial-cluster-token: “sensu”
etcd-advertise-client-urls: http colon slash slash sensu-backend-0.sensu.sensu.svc.cluster.local:2379
etcd-listen-peer-urls: “http colon slash slash 0.0.0.0:2380”
etcd-listen-client-urls: “http colon slash slash 0.0.0.0:2379”
etcd-initial-cluster-state: “new”
etcd-initial-cluster: “sensu-backend-0=http colon slash slash sensu-backend-0.sensu.sensu.svc.cluster.local:2380,sensu-backend-1=http colon slash slash sensu-backend-1.sensu.sensu.svc.cluster.local:2380,sensu-backend-2=http colon slash slash sensu-backend-2.sensu.sensu.svc.cluster.local:2380”

cat agent.yml

backend-url: ws colon slash slash sensu-backend-0.sensu.sensu.svc.cluster.local:8081
api-host: “0.0.0.0”
api-port: 3031
events-burst-limit: 20
events-rate-limit: 100
socket-host: “0.0.0.0”
socket-port: 3030
subscriptions: “system linux”

I changed the last line of subscriptios to “system”, “linux” as well and restarted agent but the output of sensuctl entity list doesn’t change.

s entity list

    ID          Class    OS         Subscriptions                  Last Seen

───────────────── ─────── ─────── ──────────────────────── ───────────────────────────────
sensu-backend-0 agent linux entity:sensu-backend-0 2020-09-23 21:09:52 +0000 UTC
sensu-backend-1 agent linux entity:sensu-backend-1 2020-09-23 21:09:45 +0000 UTC
sensu-backend-2 agent linux entity:sensu-backend-2 2020-09-23 21:09:57 +0000 UTC

Can you please help me what do I do to fix this?

Thank you so much.

7

Hi Todd, I noticed that /etc/rc.d/init.d/sensu-agent file, line args=“start” didn’t had -c option so I changed that line to “start -c /etc/sensu/agent.yml” (similar to what I found in /etc/rc.d/init.d/sensu-backend). Restarted the sensu-agent and I saw that /var/log/sensu/sensu-agent.log was showing that it validated the file and successfully started. my output to cli : sensuctl entity list still didng’t change. So, I also tried changing the line to args=“start -c /etc/sensu/agent.yml --subscriptions system,linux” and restarted agent, but still my entity list output stays same. So, I am not sure what else I can try. Any help will be really appreciated.

8

Starting with Sensu 6.0, the settings in agent.yml are only used for initial configuration of the entity. Once an entity is registered with the backend the only way to update its subscriptions is with sensuctl (or via the API).

To change the subscriptions for an entity run either sensuctl edit entity <entity name> which will bring up an interactive editor ($EDITOR) and allow you to make changes. Or use sensuctl entity update <entity name> and be prompted to interactively update the entity (you should be prompted for entity class and subscriptions).

9

Ah!, didn’t know that. I can see the subscriptions now and check does run automatically and i can see an email. However, now, in the email the err msg it shows is:

sh: check-cpu.rb: command not found
sh: metrics-disk-usage.rb: command not found

here is my check info:

s check info check-cpu

=== check-cpu
Name: check-cpu
Interval: 10
Command: check-cpu.rb -w 1 -c 2
Cron:
Timeout: 0
TTL: 0
Subscriptions: system, linux
Handlers: email
Runtime Assets: sensu-plugins/sensu-plugins-cpu-checks, sensu/sensu-ruby-runtime
Hooks:
Publish?: true
Stdin?: false
Proxy Entity Name:
Namespace: default
Metric Format:
Metric Handlers:

s check info check-disk

=== check-disk
Name: check-disk
Interval: 10
Command: metrics-disk-usage.rb -w 1 -c 2
Cron:
Timeout: 0
TTL: 0
Subscriptions: linux, system
Handlers: email
Runtime Assets: sensu-plugins/sensu-plugins-disk-checks, sensu/sensu-ruby-runtime
Hooks:
Publish?: true
Stdin?: false
Proxy Entity Name:
Namespace: default
Metric Format: graphite_plaintext
Metric Handlers: influx-db

here is my asset info:

s asset list | grep centos7 | cut -d “/” -f1,4-

cpu-checks-plugins /…/sensu-plugins-cpu-checks_4.1.0_centos7_linux_amd64.tar.gz 8a01862
disk-checks-plugins /…/sensu-plugins-disk-checks_5.1.3_centos7_linux_amd64.tar.gz 7ad47f1
sensu-ruby-runtime /…/sensu-ruby-runtime_0.0.11_ruby-2.4.4_centos7_linux_amd64.tar.gz 444ff06

I have stripped of url to avoid url limit error for posting this msg.

I tried finding those commands, but they are not there:
[root@sensu-backend-0 /]# find / -name “check-cpu.rb” -print
[root@sensu-backend-0 /]# find / -name “metrics-disk-usage.rb” -print
[root@sensu-backend-0 /]#
Do I have to use sensu core pkg and sensu-install to install those commands? my understanding was that I should not need to install them as i have assets installed and that should take care of it during runtime of the check. Please help me in this also.

Thank you so much.

10

No, you don’t. The asset system should download and install for you.

The problem appears now to be that your assets listed in runtime_assets do not match the asset names you have defined.

Either update the checks to use the assets as you have them named. Or re-add the assets using the names defined in the checks.

11

Yes, you were right, when I created assets, I used -r option to rename it. So, basically I have to use that new name. that worked.

Thank you so much for all your time and help on these issues.
Prashant