Hello,
How can I embed the results of a traceroute into an alert if a ping check fails?
There are many more scenarios described here:
http://ryanfrantz.com/posts/alert-design/
But for now I’d like to just embed traceroute results.
thanks,
Luke
Yea, I read that too. Super idea.
I was thinking about adjusting my handlers to respond to a "context"
key, if it exists.
I'll leave it up to my coworkers to fill in that key if there is
something they can add to provide context. (like mtr -r for a ping
check, or top -b for a cpu check like the article describes.)
The problem is, handlers run on the sensu-server, not the client. This
would work for things that require external resources anyway (ganglia
graphs), but I would prefer it to all be client-side if possible.
Of course you could embed your output in the check itself, but that
means you would have to add wrappers each of your checks.
Like the ping wrapper would run the check_ping command, and upon
failure, run an mtr -r, and output the return code from the failed
check and append the mtr output.
This would work fine. I guess things like SMS and IRC handlers could
just print the first line of the check output, where as things like
Email or Pagerduty handlers could output the full thing.
One downside to this is that the context command would run every time
the check runs. Eh.
Anthor approach to this is to simply have the check provide a link to
the graph you wish to embed, and then leave it up to the handler to
put the output and the context link together. This could work,
assuming either the sensu server or your email/pagerduty can see the
link. (might not work if your graphite/ganglia are internal, or behind
authentication?)
This approach also doesn't allow you to get context from commands that
need to run on the server itself.
So, to summarize, the best I can think of is:
* Internal things: wrap the check and provide the context in the check
output. Like:
#!/bin/bash
/usr/lib/nagios/plugins/check_ping -H google.com -w 80,10% -c 150,15%
RET=$?
if [[ $RET -ne 0 ]]; then
mtr -r google.com
fi
exit $RET
* External things: have handlers key off of links to embedded graphs
to provide "rich alerts"
(other examples of this:
Opsgenie: Alert & On-Call Management Support | Atlassian)
An example of this would be using the "playbook" key as an example:
https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/mailer.rb#L66
Can anyone else think of other better ways to do this? Maybe a...
check extension?
···
On Sun, May 25, 2014 at 5:19 PM, Luke Tymowski <ltymowski@gmail.com> wrote:
Hello,
How can I embed the results of a traceroute into an alert if a ping check
fails?There are many more scenarios described here:
http://ryanfrantz.com/posts/alert-design/But for now I'd like to just embed traceroute results.
thanks,
Luke
So, to summarize, the best I can think of is:
* Internal things: wrap the check and provide the context in the check
output. Like:
Thanks Kyle. The wrapper idea will work for some checks.
For others it'll be too painful (one client are ITIL-by-the-book so monthly
change windows only).
An example of this would be using the "playbook" key as an example:
https://github.com/sensu/sensu-community-plugins/blob/master/handlers/notification/mailer.rb#L66
I tried this before you responded.
But this morning, looking over what I did last night, I didn't do it
properly.
Can anyone else think of other better ways to do this? Maybe a...
check extension?
I found this last night also:
https://github.com/sensu/sensu-community-plugins/blob/master/handlers/remediation/sensu.rb
Luke