How to find out if handler is executed?

Sensu Integrations Sensu Plugins
1

I have defined a handler for sending SMS via twilio:

=== sensu-go-twilio-handler-nofilter
Name:                  sensu-go-twilio-handler-nofilter
Type:                  pipe
Timeout:               0
Filters:               
Mutator:               
Execute:               RUN:  /opt/sensu-plugins-ruby/embedded/bin/sensu-go-twilio-handler -f +sourcenumber -r +targnumber -s acountsid -t accounttoken
Environment Variables: 
Runtime Assets:

I have omitted filters on purpose for testing.
I have tested the run command by copy & paste-ing it from this info output and cat’ting an example event to it. That works fine, I receive the SMS.

Then I have defined a check using this handler on the generated events:

=== check-cpu-test
Name:              check-cpu-test
Interval:          60
Command:           /opt/sensu-plugins-ruby/embedded/bin/check-cpu.rb -w 75 -c 90
Cron:              
Timeout:           0
TTL:               0
Subscriptions:     test
Handlers:          sensu-go-twilio-handler-nofilter
Runtime Assets:    
Hooks:             
Publish?:          true
Stdin?:            false
Proxy Entity Name: 
Namespace:         default
Metric Format:     
Metric Handlers:

The check is executed fine (I can see the event in the webUI). So the handler should be called, and without any filters to apply should send the SMS in any case (right?) - but that does not happen.

So the questions that arise are:

  • Did I understand the basic concepts about event handlers and filtering correctly?
  • How can I make sensu backend log execution (or not-execution) of handlers?
  • Can this be related to the problem of http_proxy env variables I described in the Sensu Go forum?

Thanks,

Dirk

2

Hallo Dirk,

You do understand the concepts of filtering & event handling correctly. I’d be keen to see what the logs are spitting out. Sensu uses journald as its logging facility, so you can find information by running journalctl -u sensu-backend. You could pipe that output through grep to narrow things down a bit more. Note that if you have Sensu clustered, you’re going to have to look through the journald output from each backend to find what may be happening.

3

Hello Aaron,

that is a good hint. I configured the test on the sensu agent of one of the cluster nodes to make testing easier, so I only looked into the logs of this one node.

And, so far, I only installed the handler binaries only on that one node - but it seems that another node tried to execute the handler.

It seems the agent on the one cluster node is “preferring” to send the the event to another cluster node. That is interesting.

Thanks, now I can go on testing.

Dirk

4

That really was the problem.
I have to be more aware of the communication flow - thanks a lot.

5

@dirkhschulz so I suspect I know the issue, but I’m curious about your configuration. I’ve been able to send sms’s using the following command on a box with an agent running:

curl -X POST \
-H 'Content-Type: application/json' \
-d '{
  "check": {
    "metadata": {
      "name": "check-mysql-status"
    },
    "status": 2,
    "output": "could not connect to mysql",
    "handlers": ["sensu-go-twilio-handler"]
  }
}' \
http://127.0.0.1:3031/events

This sends a message just fine. But what I’ve noticed is that if you specify the accountsid & authtoken as environment variables, they’re not honored, which is something on my part that I will need to further test and possibly fixed since that handler’s one that I wrote. Can you try testing without specifying the SID and authtoken as env vars if that’s the case?

6

Hallo @dirkhschulz, so I’m presuming the issue will still be around envvars. One of my teammates made me aware of an issue with the library being used for command line flags that prohibits envvars from being used if a flag is marked as required. As such, I’ve removed the requirement on the flags for the accountsid and the authtoken so that they’ll be honored as envvars and have just pushed out a release. Let me know if the new release still has an issue.

7

Hi Aaron,
sorry, I did not make that clear enough.
My problem came from installing the plugin just on one of the cluster nodes (I intended to test only on that). But the agent sent the event to a different cluster node and there it could not be processed because the event handler was not installed.
(And I did not find out because I did not expect that and thus did not look into the logs).

After I installed the handler plugin on all 3 nodes I was able to have sensu-backend send SMS through it.
But, incidentally, I had not configured ID and Token using the env vars but directly with the parameters.

Thanks a lot for your fast help!

Dirk