Handler not triggered automatically, but can triggered manually

Sensu Go
1

our system is running sensu server service with a user called “sensu”,
$ ruby --version
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
(we are using sensu go v5.20)

 $ systemctl edit sensu-backend --full
[Service]
Type=simple
User=sensu
Group=sensu

hence all the sensu related files are all belongs to this sensu user, for example:
drwx------. 2 sensu sensu 196 Aug 10 16:43 conf.d
-rw-r–r-- 1 sensu sensu 6094 Aug 10 16:43 sensu_go_mail_template.erb
-rw-r–r-- 1 sensu sensu 681 Aug 10 16:43 sensu_go_slack_template.json

and now I can see the checks is running and provide event periodically, seeing from sensuctl event list:

api-checker      test-01-capabilities_check     CheckJson OK: valid JSON returned   0   false  2020-08-10 16:45:12 +0000 UTC

handlers are also setup, seeing from sensuctl handler list:

  mailer    pipe     0      RUN:  /opt/sensu-plugins-ruby/embedded/bin/handler-mailer.rb -j test-mail -c html -s '[Sensu]' --map-go-event-into-ruby    sensu-plugins-http,sensu-ruby-runtime

Now, if I trigger the event manually, i can received the email alert, by runnning these commands:
$ sensuctl event info api-checker test-01-capabilities_check --format json #saved the result into event file
$ cat event | /opt/sensu-plugins-ruby/embedded/bin/handler-mailer.rb -j test-mail -c html -s ‘[Sensu]’ --map-go-event-into-ruby

However… if I doing the last step to let it run the checks and alert by itself, I can see there is an event, I can see there is an handler existing, but no alert get triggered. I’m wondering if I missed some configurations or anything else?

2

Hi @melw

You could provide us your check configuration, using something like sensuctl check info test-01-capabilities_check --format yaml ?

Also, you should definitely take a look at your backend logs, and possibly increase the verbosity of those logs to debug; you should see some entries around the execution of this handler if everything goes well.

Thanks

3

Hey,
a couple of quick things to check.

  1. if you are running the command manually on a different host than where the handler runs… make sure the backend host has the necessary smtp server configuration. You may need to adjust the backend host configuration to ensure the smtp server is accessible

  2. Run manually as the sensu user under sudo if possible? It could be that your regular user running manually has additional privileges or settings that the sensu user doesn’t have.

4

Hi @palourde ,
which logs shall I enable, the event log( something like this: sensu-backend start --event-log-file /var/log/sensu/events.log )

it’s a simple check:
type: CheckConfig
api_version: core/v2
metadata:
created_by: sensu
labels:
name: test-01-capabilities_check
namespace: test
spec:
check_hooks: null
command: an_invalid_check_command
env_vars: null
handlers:
- handler-tests
high_flap_threshold: 0
interval: 10
low_flap_threshold: 0
output_metric_format: “”
output_metric_handlers: null
proxy_entity_name: api-checker
publish: true
round_robin: true
runtime_assets:
- sensu-plugins-http
- sensu-ruby-runtime
secrets: null
stdin: false
subdue: null
subscriptions:
- runner
timeout: 0
ttl: 0

    the handler set is like this, so far there are four handlers and none of them is working (the debug handler is cat some outputs from the troubleshooting page in the sensugo document, command: cat > /var/log/sensu/debug-event.json)
        type: Handler
        api_version: core/v2
        metadata:
          created_by: sensu
          labels:
            sensu.io/managed_by: sensuctl
          name: handler-tests
          namespace: test
        spec:
          env_vars: null
          filters: null
          handlers:
          - mailer
          - pagerduty
          - slack-integration-tests
          - debug-handler
          runtime_assets: null
          secrets: null
          timeout: 0
          type: set

    type: Handler
    api_version: core/v2
    metadata:
      created_by: sensu
      labels:
        sensu.io/managed_by: sensuctl
      name: mailer
      namespace: test
    spec:
      command: /opt/sensu-plugins-ruby/embedded/bin/handler-mailer.rb -j test-mail -c
        html -s '[Sensu]' --map-go-event-into-ruby
      env_vars: null
      filters: null
      handlers: null
      runtime_assets:
      - sensu-plugins-http
      - sensu-ruby-runtime
      secrets: null
      timeout: 0
      type: pipe
5

Hi @jspaleta

it is running on the same host, and smtp server has been setup since the email can be received if triggered the handler manually

and it is not able to run the handler script in root user, since there is no gem installed under the root user…

I tried to add a debug handler like this:

type: Handler
api_version: core/v2
metadata:
  name: debug
spec:
  type: pipe
  command: cat > /var/log/sensu/debug-event.json
  timeout: 2

and it is writing an event info to this file, so it should be somewhere stuck in executing the handler.
Is it possible to be related with etcd’s setting?

6

hi @palourde ,
I set up a backend log, and it shows shutting down scheduler, what does this mean? (the check is in published status when restart the server)

2020-08-12T20:39:48.034838+00:00 poc-sensu-go-srv-1f4w sensu-backend[4526]: {"component":"schedulerd","level":"warning","msg":"shutting down scheduler","name":"test-01-capabilities_check","namespace":"test","scheduler_type":"round-robin interval","time":"2020-08-12T20:39:48Z"}

7

Hi @melw,

You can ignore this log entry, it’s a known issue but it wouldn’t provoke the problem you are seeing: https://github.com/sensu/sensu-go/issues/3862.

You want to adjust the log-level config attribute of sensu-backend to debug, restart the backend(s) and look at the logs; you may filter by the handler name to make it easier to see what’s happening. At some point you should see some information about the handler being executed with its output.

8

it finally solved since the wrong user permission. I set up the debug log and finding some handlers are working but some are not ; )

Thank you both