Also, I would work on automation on priority to make my life easier.
(please reply-all to sensu-users so others can learn and potentially
respond)
Yes, that looks good now. Your email handlers and check definitions
line up. That should have resulted in an actual email. (if mailx
worked)
Longer term you should work out why disk-fail.rb is failing on line
64. Also eventually you will want to use the mailer.rb community
handler over mailx to give you more flexibility.
Also configuration management (chef/puppet,etc) long term will allow
you to mange sensu configuration as it gets more complex without
hand-making json files, fewer typos, auto restarts, etc. I highly
recommend using a tool like this.
On Sun, May 3, 2015 at 1:07 PM, Srinivas Palle <srini5884@gmail.com> >> wrote:
> Thanks Kyle,
>
> Finally I can see no error showing. Please confirm if below log shows
> everything fine or not?
>
>
> {"timestamp":"2015-05-03T20:05:35.392668+0000","level":"info","message":"processing
>
> event","event":{"id":"03f7ea89-8ca4-4ff8-9cc5-e8684ba8d245","client":{"name":"server","address":"localhost","subscriptions":["ALL"],"version":"0.17.2","timestamp":1430683520},"check":{"command":"/etc/sensu/plugins/disk-fail.rb","interval":60,"handlers":["default","email"],"subscribers":["ALL"],"name":"disk-fail_check","issued":1430683535,"executed":1430683535,"duration":0.013,"output":"/etc/sensu/plugins/disk-fail.rb:64:
> class definition in method
>
> body\n","status":1,"history":["1","1","1","1","1","1","1","1","1","1","1","1","1","1","1","1","1","1","1","1","1"],"total_state_change":0},"occurrences":1756,"action":"create"}}
>
>
> {"timestamp":"2015-05-03T20:05:35.394632+0000","level":"info","message":"handler
>
> output","handler":{"type":"pipe","command":"true","name":"default"},"output":}
>
>
> {"timestamp":"2015-05-03T20:05:35.411406+0000","level":"info","message":"handler
> output","handler":{"type":"pipe","command":"mailx -s 'sensu alert'
> tnt-devops@olacabs.com","name":"email"},"output":}
>
>
> On Mon, May 4, 2015 at 1:17 AM, Kyle Anderson <kyle@xkyle.com> wrote:
>>
>> (please reply-all to sensu-users so others can learn and potentially
>> respond)
>>
>> Can you confirm you are restarting the sensu server when changing
>> check definitions and restating the server when changing handler
>> definitions? (long term a I suggest using a configuration management
>> tool to do this for you)
>>
>> Your check definition says "mail" as the handler, but your handler
>> json says the name is "email".
>>
>> The "unknown handler" message still indicates that the "email" handler
>> json is not being read or parsed or something. Inspect
>> sensu-server.log to see why that might be the case after restarting
>> sensu-server.
>>
>>
>> The next error you will have to tackle is
>> "/etc/sensu/plugins/disk-fail.rb:64: class definition in method
>> body\n" from the check, but I would worry about that after you have
>> fixed the handler situation.
>>
>> On Sun, May 3, 2015 at 12:35 PM, Srinivas Palle <srini5884@gmail.com> >> >> wrote:
>> > Here is my conf.d file for disk fail check:
>> >
>> > {
>> >
>> > "checks": {
>> >
>> > "disk-fail_check": {
>> >
>> > "command": "/etc/sensu/plugins/disk-fail.rb",
>> >
>> > "interval": 60,
>> >
>> > "handlers": ["default", "mail"],
>> >
>> > "subscribers": [ "ALL" ]
>> >
>> > }
>> >
>> > }
>> >
>> > }
>> >
>> >
>> > On Mon, May 4, 2015 at 1:03 AM, Srinivas Palle <srini5884@gmail.com> >> >> > wrote:
>> >>
>> >> I still see the same error even after making below corrections, now
>> >> am
>> >> trying to use only one handler.
>> >>
>> >> I hope this will be helpful...
>> >>
>> >> I would like to thank you for helping me to resolve this issue.
>> >>
>> >> On Mon, May 4, 2015 at 12:48 AM, Kyle Anderson <kyle@xkyle.com> >> >> >> wrote:
>> >>>
>> >>> It is a little confusing to have both an "email" handler defined
>> >>> that
>> >>> just uses the mail command and a "mailer.json" for using the
>> >>> community
>> >>> mailer handler.
>> >>>
>> >>> I recommend just using one, but I understand you are
>> >>> troubleshooting
>> >>> at this point.
>> >>>
>> >>> Please confirm, these configuration files are on the sensu-server,
>> >>> they are valid json and readable by the sensu user, and the sensu
>> >>> server has been restarted to load them?
>> >>>
>> >>> The "unknown handler","handler_name":"email" message indicates that
>> >>> a
>> >>> handler named email is not available, hinting that the
>> >>> handler_email.json file is hasn't been loaded.
>> >>>
>> >>> Additionally it looks like might have a typo in the email address
>> >>> invocation :
>> >>> "command": "mailx -s 'sensu alert' tnt-devops@@olacabs.com""
>> >>> It looks like there are two @ signs and two double quotes? Might
>> >>> just
>> >>> be email formatting, but that looks incorrect.
>> >>>
>> >>> You can always verify by inspecting the sensu-server logs to
>> >>> confirm
>> >>> your file is being loaded as intended without errors.
>> >>>
>> >>> On Sun, May 3, 2015 at 12:12 PM, Srinivas Palle >> >> >>> <srini5884@gmail.com> >> >> >>> wrote:
>> >>> > I do have event_file.json file in conf.d
>> >>> >
>> >>> >
>> >>> > "handlers": {
>> >>> >
>> >>> > "file": {
>> >>> >
>> >>> > "type": "pipe",
>> >>> >
>> >>> > "command": "/etc/sensu/plugins/event-file.rb",
>> >>> >
>> >>> > "timeout": 10,
>> >>> >
>> >>> > }
>> >>> >
>> >>> > }
>> >>> >
>> >>> > }
>> >>> >
>> >>> >
>> >>> > On Mon, May 4, 2015 at 12:40 AM, Srinivas Palle >> >> >>> > <srini5884@gmail.com> >> >> >>> > wrote:
>> >>> >>
>> >>> >> Thanks Kyle,
>> >>> >>
>> >>> >> Unfortunately am still not getting any alerts to my mail box, am
>> >>> >> using
>> >>> >> mailer.rb in /etc/sensu/handlers and in conf.d am using
>> >>> >> handler_email.json
>> >>> >> and mailer.json. Will that be fine if I use both the .json
>> >>> >> fileS?
>> >>> >> Please
>> >>> >> find configuration of handler and advise me..
>> >>> >>
>> >>> >> I am unable to resolve this using the above docs provided.
>> >>> >>
>> >>> >> Logs:
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> timestamp":"2015-05-03T19:04:35.392196+0000","level":"error","message":"unknown
>> >>> >> handler","handler_name":"email"}
>> >>> >>
>> >>> >>
>> >>> >> handler_email.json:
>> >>> >>
>> >>> >> {
>> >>> >>
>> >>> >> "handlers": {
>> >>> >>
>> >>> >> "email": {
>> >>> >>
>> >>> >> "type": "pipe",
>> >>> >>
>> >>> >> "command": "mailx -s 'sensu alert'
>> >>> >> tnt-devops@@olacabs.com""
>> >>> >>
>> >>> >> }
>> >>> >>
>> >>> >> }
>> >>> >>
>> >>> >> }
>> >>> >>
>> >>> >>
>> >>> >> Mailer.json:
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> {
>> >>> >>
>> >>> >> "mailer": {
>> >>> >>
>> >>> >> "admin_gui": "http://localhost:80/",
>> >>> >>
>> >>> >> "mail_from": "srinivas.palle@olacabs.com",
>> >>> >>
>> >>> >> "mail_to": "tnt-devops@olacabs.com",
>> >>> >>
>> >>> >> "smtp_address": "smtp.gmail.com",
>> >>> >>
>> >>> >> "smtp_port": "25",
>> >>> >>
>> >>> >> "smtp_domain": "gmail.com"
>> >>> >>
>> >>> >> }
>> >>> >>
>> >>> >> }
>> >>> >>
>> >>> >>
>> >>> >> On Mon, May 4, 2015 at 12:32 AM, Kyle Anderson <kyle@xkyle.com> >> >> >>> >> wrote:
>> >>> >>>
>> >>> >>> You could use a graphite extension to store event history.
>> >>> >>> (I'm still waiting for some sort of cool sensudb thing to be
>> >>> >>> invented
>> >>> >>> for historical analysis?)
>> >>> >>>
>> >>> >>> Most of the time when I need to debug a hanlder I use the
>> >>> >>> sensu-server
>> >>> >>> (sometimes in combination with the sensu-client) logs to try to
>> >>> >>> figure
>> >>> >>> out why a handler or event didn't do what I thought it should
>> >>> >>> do.
>> >>> >>>
>> >>> >>> I guess in that sense the history is stored, just in log form
>> >>> >>> (not
>> >>> >>> redis and not api)
>> >>> >>>
>> >>> >>> On Sun, May 3, 2015 at 11:54 AM, Srinivas Palle >> >> >>> >>> <srini5884@gmail.com> >> >> >>> >>> wrote:
>> >>> >>> > Thanks Kyle,
>> >>> >>> >
>> >>> >>> > In this case I may need to use graphite to store event
>> >>> >>> > history
>> >>> >>> > for
>> >>> >>> > certain
>> >>> >>> > period of time for debugging whenever required.
>> >>> >>> >
>> >>> >>> > Correct me if am wrong
>> >>> >>> >
>> >>> >>> > On Mon, May 4, 2015 at 12:12 AM, Kyle Anderson >> >> >>> >>> > <kyle@xkyle.com> >> >> >>> >>> > wrote:
>> >>> >>> >>
>> >>> >>> >> In the API and in the dashboard (uchiwa) the last 20 status
>> >>> >>> >> codes
>> >>> >>> >> are
>> >>> >>> >> available in the "history" section of an event
>> >>> >>> >>
>> >>> >>> >> http://static.oschina.net/uploads/img/201405/19063539_ovDI.jpg
>> >>> >>> >> http://sensuapp.org/docs/latest/events
>> >>> >>> >>
>> >>> >>> >> Note that because sensu is not a "green light dashboard",
>> >>> >>> >> this
>> >>> >>> >> event
>> >>> >>> >> history is only user-visible when the check is failing, not
>> >>> >>> >> on
>> >>> >>> >> checks
>> >>> >>> >> that are green.
>> >>> >>> >> In that sense they are not really "stored" (in redis) unless
>> >>> >>> >> the
>> >>> >>> >> check
>> >>> >>> >> is failing. Otherwise they are reported by the client to the
>> >>> >>> >> handler
>> >>> >>> >> as they come. (as event data)
>> >>> >>> >>
>> >>> >>> >> On Sun, May 3, 2015 at 11:12 AM, Srinivas Palle >> >> >>> >>> >> <srini5884@gmail.com> >> >> >>> >>> >> wrote:
>> >>> >>> >> > Thanks Kyle,
>> >>> >>> >> >
>> >>> >>> >> > May I know where can we get history of events in sensu
>> >>> >>> >> > monitoring
>> >>> >>> >> > set
>> >>> >>> >> > up?
>> >>> >>> >> > Can we get history with in server?
>> >>> >>> >> >
>> >>> >>> >> > Just trying to where do all the results gets stored before
>> >>> >>> >> > handling
>> >>> >>> >> > to
>> >>> >>> >> > handler.
>> >>> >>> >> >
>> >>> >>> >> > On Sun, May 3, 2015 at 11:16 PM, Kyle Anderson >> >> >>> >>> >> > <kyle@xkyle.com> >> >> >>> >>> >> > wrote:
>> >>> >>> >> >>
>> >>> >>> >> >> Yes, you do need to install/deploy the handler script
>> >>> >>> >> >> (mailer.rb,
>> >>> >>> >> >> etc.
>> >>> >>> >> >> Usually in /etc/sensu/handlers/).
>> >>> >>> >> >> Additionally, you must add configuration for that handler
>> >>> >>> >> >> (/etc/sensu/conf.d/..) to configure that handler to
>> >>> >>> >> >> perform
>> >>> >>> >> >> actions
>> >>> >>> >> >> on
>> >>> >>> >> >> events (to get alerts)
>> >>> >>> >> >> Read more about how to install and configure handlers at
>> >>> >>> >> >> the
>> >>> >>> >> >> docs
>> >>> >>> >> >> page:
>> >>> >>> >> >>
>> >>> >>> >> >>
>> >>> >>> >> >> http://sensuapp.org/docs/0.17/getting-started-with-handlers
>> >>> >>> >> >> http://sensuapp.org/docs/0.17/handlers
>> >>> >>> >> >>
>> >>> >>> >> >> On Sun, May 3, 2015 at 5:19 AM, Srinivas Palle >> >> >>> >>> >> >> <srini5884@gmail.com> >> >> >>> >>> >> >> wrote:
>> >>> >>> >> >> > Hi,
>> >>> >>> >> >> >
>> >>> >>> >> >> > I do have same issue where log states unknown handler
>> >>> >>> >> >> > email,
>> >>> >>> >> >> > may
>> >>> >>> >> >> > I
>> >>> >>> >> >> > know
>> >>> >>> >> >> > the
>> >>> >>> >> >> > need of having file.rb? I do not have this file in
>> >>> >>> >> >> > /etc/sensu/handlers/,
>> >>> >>> >> >> > do
>> >>> >>> >> >> > I need to install anything or add in /etc/sensu/conf.d/
>> >>> >>> >> >> > to
>> >>> >>> >> >> > get
>> >>> >>> >> >> > started
>> >>> >>> >> >> > in
>> >>> >>> >> >> > generating alerts from server?
>> >>> >>> >> >> >
>> >>> >>> >> >> > Please advise me...
>> >>> >>> >> >> >
>> >>> >>> >> >> > On Monday, March 2, 2015 at 12:10:40 AM UTC+5:30, Kyle >> >> >>> >>> >> >> > Anderson >> >> >>> >>> >> >> > wrote:
>> >>> >>> >> >> >>
>> >>> >>> >> >> >> It looks like there is no handler configured with the
>> >>> >>> >> >> >> name
>> >>> >>> >> >> >> of
>> >>> >>> >> >> >> "default"
>> >>> >>> >> >> >> http://sensuapp.org/docs/0.16/handlers
>> >>> >>> >> >> >>
>> >>> >>> >> >> >> Good handlers for metrics are things like graphite,
>> >>> >>> >> >> >> etc?
>> >>> >>> >> >> >>
>> >>> >>> >> >> >> On Fri, Feb 27, 2015 at 3:28 PM, >> >> >>> >>> >> >> >> <max2su...@gmail.com> >> >> >>> >>> >> >> >> wrote:
>> >>> >>> >> >> >> > Hi,
>> >>> >>> >> >> >> > We are facing a problem of not getting
>> >>> >>> >> >> >> > any
>> >>> >>> >> >> >> > check
>> >>> >>> >> >> >> > "metrics"
>> >>> >>> >> >> >> > response from the client while following the
>> >>> >>> >> >> >> > tutorial
>> >>> >>> >> >> >> > at
>> >>> >>> >> >> >> > link
>> >>> >>> >> >> >> > Adding
>> >>> >>> >> >> >> > a
>> >>> >>> >> >> >> > Metric.
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > Following are the outputs :-
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > Server > tail -f /var/log/sensu/sensu-server.log
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > {"timestamp":"2015-02-27T18:23:59.010129-0500","level":"error","message":"unknown
>> >>> >>> >> >> >> > handler","handler_name":"default"}
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > {"timestamp":"2015-02-27T18:24:06.362472-0500","level":"info","message":"determining
>> >>> >>> >> >> >> > stale clients"}
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > {"timestamp":"2015-02-27T18:24:06.384124-0500","level":"error","message":"unknown
>> >>> >>> >> >> >> > handler","handler_name":"default"}
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > {"timestamp":"2015-02-27T18:24:06.386882-0500","level":"error","message":"unknown
>> >>> >>> >> >> >> > handler","handler_name":"default"}
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > {"timestamp":"2015-02-27T18:24:06.389126-0500","level":"error","message":"unknown
>> >>> >>> >> >> >> > handler","handler_name":"default"}
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > {"timestamp":"2015-02-27T18:24:08.426386-0500","level":"info","message":"publishing
>> >>> >>> >> >> >> > check
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > request","payload":{"name":"load_metrics","issued":1425079448,"command":"load-metrics.rb"},"subscribers":["production"]}
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > Client > tail -f /var/log/sensu/sensu-client.log
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > {"timestamp":"2015-02-27T18:25:37.674068-0500","level":"info","message":"publishing
>> >>> >>> >> >> >> > check
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > result","payload":{"client":"sensu-client.25.max2","check":{"name":"load_metrics","issued":1425079538,"command":"load-metrics.rb","executed":1425079537,"duration":0.172,"output":"Dev.load_avg.one
>> >>> >>> >> >> >> > 0.13 1425079537\nDev.load_avg.five 0.12
>> >>> >>> >> >> >> > 1425079537\nDev.load_avg.fifteen
>> >>> >>> >> >> >> > 0.22 1425079537\n\n","status":0}}}
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > What are the configs that we are missing ?
>> >>> >>> >> >> >> >
>> >>> >>> >> >> >> > Thanks:-
>> >>> >>> >> >> >> >
>> >>> >>> >> >
>> >>> >>> >> >
>> >>> >>> >
>> >>> >>> >
>> >>> >>
>> >>> >>
>> >>> >
>> >>
>> >>
>> >
>
>
.