Add custom output to check-http.rb and pass it to Slack

Hey - I have been able to get a simple https check running and sending alerts to a Slack channel. I just installed Senso Go packages:

sensu-go-agent.x86_64 5.18.0-9470 @sensu_stable
sensu-go-backend.x86_64 5.18.0-9470 @sensu_stable
sensu-go-cli.x86_64 5.18.0-9470 @sensu_stable

I am using the sensu-plugins-http asset and check-http.rb in the check command.

I was looking to add some extra output to what is being sent now, what is the best way to achieve that?

I’ve tried using a hook, and I can see the output of the hook command via sensuctl looking at event info for the check and entity it runs on.

But that output is not making it to what is being sent to Slack. I am just looking at this point to have it display the URL that was tested and failed, but would like to also know in general how to do this so that I could add different output to different checks and see it in the alerts that get sent to Slack.

Thanks!

1 Like

The output in slack looks like this BTW when it fails:

CheckHttp CRITICAL: Request error: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read server session ticket A

Which isn’t capturing the URL I am testing in the check.

Thanks for posting your question here. As it appears right now, the Slack handler only makes use of the check.output, check.name, and entity.name fields from an event’s check. See code here. There already exists an issue to add templating support to the Slack handler.

Without the handler changing to support templating or alternative fields, you’ll need to look at getting the required information into the check’s output. And, unfortunately, I don’t see that as an option with check-http.rb or with the check_http included in the monitoring-plugins asset. You could possibly create an issue here for the check-http.rb plugin, requesting an option to include the URL in the check output. Or you could fork that repo, make the changes yourself and create your own Bonsai asset from it.

Another option may be a mutator that you could write to extract the URL from elsewhere in the event JSON and then munge the check.ouput to include the URL.

Could you provide your check definition?

It was created with this:

sensuctl check create check-https-target-server
–command ‘check-http.rb -k -u https://target-server.example.com/
–interval 30
–subscriptions backend
–runtime-assets sensu-plugins/sensu-plugins-http,sensu-ruby-runtime

I also added slack as a handler, and a hook that just outputs some test text.

Of course you could always create the check as a proxy entity, that way the site name could be the entity name and should then show up on the Slack alert.

https://docs.sensu.io/sensu-go/latest/guides/monitor-external-resources/#use-a-proxy-entity-to-monitor-a-website

Not sure if I have to escape the slashes somehow or if they’re just not allowed, but when I put in a URL as the entity - for example like this:

proxy_entity_name: https://sensu-client2.localdomain

sensuctl gives me this error:

Error: error putting resource #0 with name “check-sensu-client2-https” and namespace “default” (/api/core/v2/namespaces/default/checks/check-sensu-client2-https): invalid argument(s) received

It will accept/create it if I remove the slashes.

I’m guessing it’s just not allowed as if I try to use sensuctl directly with no .yml file it says:

Error: proxy entity name cannot contain spaces or special characters

Also for some reason when I define a proxy entity name, it seems to confuse the check’s ability to find libraries it needs, yet I am using the same runtime asset definitions I’ve used in other checks which work. Very confusing.

Check failed to run: undefined method request_uri' for #<URI::Generic -k>, ["/var/cache/sensu/sensu-agent/ed9c3c8f1fbc37f9c856498ee77d93e2cd045993786a3b59ee76e240aa799f98dd1313672adb6ffd81a131103d401351d4434b4ea95fb40cd5cd40c8afa1ac29/lib/ruby/2.4.0/bundler/gems/sensu-plugins-http-33783747d3af/bin/check-http.rb:266:in run’", “/var/cache/sensu/sensu-agent/ed9c3c8f1fbc37f9c856498ee77d93e2cd045993786a3b59ee76e240aa799f98dd1313672adb6ffd81a131103d401351d4434b4ea95fb40cd5cd40c8afa1ac29/lib/ruby/2.4.0/gems/sensu-plugin-4.0.0/lib/sensu-plugin/cli.rb:59:in `block in class:CLI’”]

Actually I guess the ruby errors are happening all the time now. Any new checks I make using check-http.rb now fail, while checks I made yesterday with the exact same spec are working?

Runtime Assets: sensu-plugins/sensu-plugins-http, sensu-ruby-runtime

Same command:

Command: check-http.rb -k -u URL

???

Yes, I guess I should have been more specific in regards to naming a proxy entity. It should be treated like any other standard hostname.

To eliminate any issues with the assets, try this:

  1. Stop the agent
  2. Remove the sensu-agent cache (rm -rf /var/cache/sensu/sensu-agent)
  3. Start the agent

Any need assets should be re-downloaded.

The library issues seem to have cleared on their own. I’m not sure why assets would be stored in cache like that though as many checks will be long-lived and it seems like a pretty disruptive thing to have to fix the monitoring system and do reinstalls any time the cache goes bad.

I would think storing the assets somewhere more permanent and less volatile would be a better idea, and would definitely be easier on the eyes than the long directory names used now. Being able to replicate the ruby environment the checks use would be helpful as well to be able to test them on the command line outside of Sensu.

Having to clear the cache that way is not a normal activity. I only suggested it to make sure there was nothing wrong with your downloaded assets.

Assets are meant to help you avoid having to use configuration management to deploy the necessary bits for your various checks and handlers. The power of assets really shines when dealing with ephemeral instances and/or containers. Since checks are subscription based, only the bits for them are downloaded when appropriate. However, if you prefer on your longer lived entities to have CM deploy your checks, using assets is not required.

As for testing assets on the command line outside of Sensu, you should check out this post.

Maybe I was just unlucky then but I’ve only run this for a few days and already seen the problem. I used the same assets in other checks, so to me it seems like the cache is volatile.

Container images can also be created with only the monitoring scripts/libraries they need, that would avoid needing to run config management on them and also avoid any cache issues.

I am looking for something that is friendly for both VMs/config management and for containers in the future. I’ve seen Sensu Core work really well with Chef and VMs in the past, it’s much easier to manage new hosts than say using Nagios as the server doesn’t need any changes - you just apply a recipe to new hosts and you’re done.

I like the idea of using subscriptions as it simplifies client setup even further, but not if it means checks can break because of cache issues. I wouldn’t want to risk that on a long or short lived container.