I am using the sensu-plugins-http asset and check-http.rb in the check command.
I was looking to add some extra output to what is being sent now, what is the best way to achieve that?
Iāve tried using a hook, and I can see the output of the hook command via sensuctl looking at event info for the check and entity it runs on.
But that output is not making it to what is being sent to Slack. I am just looking at this point to have it display the URL that was tested and failed, but would like to also know in general how to do this so that I could add different output to different checks and see it in the alerts that get sent to Slack.
Thanks for posting your question here. As it appears right now, the Slack handler only makes use of the check.output, check.name, and entity.name fields from an eventās check. See code here. There already exists an issue to add templating support to the Slack handler.
Without the handler changing to support templating or alternative fields, youāll need to look at getting the required information into the checkās output. And, unfortunately, I donāt see that as an option with check-http.rb or with the check_http included in the monitoring-plugins asset. You could possibly create an issue here for the check-http.rb plugin, requesting an option to include the URL in the check output. Or you could fork that repo, make the changes yourself and create your own Bonsai asset from it.
Another option may be a mutator that you could write to extract the URL from elsewhere in the event JSON and then munge the check.ouput to include the URL.
Of course you could always create the check as a proxy entity, that way the site name could be the entity name and should then show up on the Slack alert.
Error: error putting resource #0 with name ācheck-sensu-client2-httpsā and namespace ādefaultā (/api/core/v2/namespaces/default/checks/check-sensu-client2-https): invalid argument(s) received
It will accept/create it if I remove the slashes.
Iām guessing itās just not allowed as if I try to use sensuctl directly with no .yml file it says:
Error: proxy entity name cannot contain spaces or special characters
Also for some reason when I define a proxy entity name, it seems to confuse the checkās ability to find libraries it needs, yet I am using the same runtime asset definitions Iāve used in other checks which work. Very confusing.
Check failed to run: undefined method request_uri' for #<URI::Generic -k>, ["/var/cache/sensu/sensu-agent/ed9c3c8f1fbc37f9c856498ee77d93e2cd045993786a3b59ee76e240aa799f98dd1313672adb6ffd81a131103d401351d4434b4ea95fb40cd5cd40c8afa1ac29/lib/ruby/2.4.0/bundler/gems/sensu-plugins-http-33783747d3af/bin/check-http.rb:266:in runā", ā/var/cache/sensu/sensu-agent/ed9c3c8f1fbc37f9c856498ee77d93e2cd045993786a3b59ee76e240aa799f98dd1313672adb6ffd81a131103d401351d4434b4ea95fb40cd5cd40c8afa1ac29/lib/ruby/2.4.0/gems/sensu-plugin-4.0.0/lib/sensu-plugin/cli.rb:59:in `block in class:CLIāā]
Actually I guess the ruby errors are happening all the time now. Any new checks I make using check-http.rb now fail, while checks I made yesterday with the exact same spec are working?
The library issues seem to have cleared on their own. Iām not sure why assets would be stored in cache like that though as many checks will be long-lived and it seems like a pretty disruptive thing to have to fix the monitoring system and do reinstalls any time the cache goes bad.
I would think storing the assets somewhere more permanent and less volatile would be a better idea, and would definitely be easier on the eyes than the long directory names used now. Being able to replicate the ruby environment the checks use would be helpful as well to be able to test them on the command line outside of Sensu.
Having to clear the cache that way is not a normal activity. I only suggested it to make sure there was nothing wrong with your downloaded assets.
Assets are meant to help you avoid having to use configuration management to deploy the necessary bits for your various checks and handlers. The power of assets really shines when dealing with ephemeral instances and/or containers. Since checks are subscription based, only the bits for them are downloaded when appropriate. However, if you prefer on your longer lived entities to have CM deploy your checks, using assets is not required.
As for testing assets on the command line outside of Sensu, you should check out this post.
Maybe I was just unlucky then but Iāve only run this for a few days and already seen the problem. I used the same assets in other checks, so to me it seems like the cache is volatile.
Container images can also be created with only the monitoring scripts/libraries they need, that would avoid needing to run config management on them and also avoid any cache issues.
I am looking for something that is friendly for both VMs/config management and for containers in the future. Iāve seen Sensu Core work really well with Chef and VMs in the past, itās much easier to manage new hosts than say using Nagios as the server doesnāt need any changes - you just apply a recipe to new hosts and youāre done.
I like the idea of using subscriptions as it simplifies client setup even further, but not if it means checks can break because of cache issues. I wouldnāt want to risk that on a long or short lived container.