# Log monitoring

**URL:** <https://discourse.sensu.io/t/log-monitoring/274>\
**Category:** Sensu Classic (EOL)\
**Created:** [October 6, 2014, 8:33am UTC](https://discourse.sensu.io/t/log-monitoring/274 "2014-10-06T08:33:31Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alik\_Kurdyukov](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@Alik\_Kurdyukov](https://discourse.sensu.io/u/Alik_Kurdyukov)\
**Post date:** [October 6, 2014, 8:33am UTC](https://discourse.sensu.io/t/log-monitoring/274/1 "2014-10-06T08:33:31Z")

</div>

Hello!

I’m trying to implement sensu-based log checking for my infrastructure.

Requirements are:

1. System should checks in log4j/NLog format. Event should be fired when log line level ERROR or FATAL appeared in the log.

2. System should not clear event, only admin can delete event saying ‘ok, I’ve seen it’

3. System should not fire error on the same log line twice.

4. Logs should not be modified by monitoring.

Ideal implementation looks like this:

1. Sensu checks logs from latest mark to the end of log.

2. If some error or exception found - warning or error condition raised, mark is not moved. If no error found or previous event deleted by user - mark is moved to current end of log.

3. Administrator looks at the error description, performs some action and deletes event from sensu. No log modification is performed.

Unfortunately this method cannot be implemented with current Sensu because event handler cannot detect situation when event was removed by user.

Am I missing something and I can implement requirements some other way?

Regards,

Alik.

---

<div class="post-metadata">

**Author:** ![Kyle\_Anderson](https://avatars.discourse-cdn.com/v4/letter/k/e5b9ba/32.png) [@Kyle\_Anderson](https://discourse.sensu.io/u/Kyle_Anderson)\
**Post date:** [October 6, 2014, 3:03pm UTC](https://discourse.sensu.io/t/log-monitoring/274/2 "2014-10-06T15:03:14Z")

</div>

If you really want to literally fire an event whenever a certain line  
is found, and not automatically clear the event,  
then I would make the check utilize the local socket to send an  
arbitrary event, and only send CRIT, and never OK.

If you want to make sure that there are separate events and never fire  
for the same log line twice, hash the line and put the hash in the  
check name.

1. Sensu check that reads the log and marks an offset  
([http://linux.die.net/man/8/logtail\](http://linux.die.net/man/8/logtail%5C)) (check\_name = check\_the\_log)  
2. If an error is found, send a \*new\* event to the localhost socket  
with part of the hash of the line in the check\_name. (check\_name =  
logline\_found\_$hash)  
3. Congrats, your dashboard will be full of events that will never  
clear, no logs modified.

Working with the localhost socket is and advanced topic, I have a PR  
to have it documented here:

> <https://github.com/sensu/sensu-docs/pull/132>
>
> Closes #87 
> 
> Adds a redirect shortcode.
> 
> Added it to Uchiwa's section so tha…t we will be redirected to \[https://uchiwa.io/\](https://uchiwa.io/)

> **···**
>
> On Mon, Oct 6, 2014 at 1:33 AM, Alik Kurdyukov \<akurdyukov@gmail.com\> wrote:
> 
> > Hello!
> > 
> > I'm trying to implement sensu-based log checking for my infrastructure.
> > 
> > Requirements are:  
> > 1. System should checks in log4j/NLog format. Event should be fired when log  
> > line level ERROR or FATAL appeared in the log.  
> > 2. System should not clear event, only admin can delete event saying 'ok,  
> > I've seen it'  
> > 3. System should not fire error on the same log line twice.  
> > 4. Logs should not be modified by monitoring.
> > 
> > Ideal implementation looks like this:  
> > 1. Sensu checks logs from latest mark to the end of log.  
> > 2. If some error or exception found - warning or error condition raised,  
> > mark is not moved. If no error found or previous event deleted by user -  
> > mark is moved to current end of log.  
> > 3. Administrator looks at the error description, performs some action and  
> > deletes event from sensu. No log modification is performed.
> > 
> > Unfortunately this method cannot be implemented with current Sensu because  
> > event handler cannot detect situation when event was removed by user.
> > 
> > Am I missing something and I can implement requirements some other way?
> > 
> > Regards,  
> > Alik.

---

<div class="post-metadata">

**Author:** ![Alexander\_York](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@Alexander\_York](https://discourse.sensu.io/u/Alexander_York)\
**Post date:** [October 6, 2014, 6:14pm UTC](https://discourse.sensu.io/t/log-monitoring/274/3 "2014-10-06T18:14:37Z")

</div>

I’ve been wondering about the same thing for windows … FYI …

logtail with offset for winderz.

[http://logtail-v3.sourceforge.net/](http://logtail-v3.sourceforge.net/)

> **···**
>
> On Mon, Oct 6, 2014 at 11:03 AM, Kyle Anderson [kyle@xkyle.com](mailto:kyle@xkyle.com) wrote:
> 
> > If you really want to literally fire an event whenever a certain line
> > 
> > is found, and not automatically clear the event,
> > 
> > then I would make the check utilize the local socket to send an
> > 
> > arbitrary event, and only send CRIT, and never OK.
> > 
> > If you want to make sure that there are separate events and never fire
> > 
> > for the same log line twice, hash the line and put the hash in the
> > 
> > check name.
> > 
> > 1. Sensu check that reads the log and marks an offset
> > 
> > ([http://linux.die.net/man/8/logtail](http://linux.die.net/man/8/logtail)) (check\_name = check\_the\_log)
> > 
> > 1. If an error is found, send a _new_ event to the localhost socket
> > 
> > with part of the hash of the line in the check\_name. (check\_name =
> > 
> > logline\_found\_$hash)
> > 
> > 1. Congrats, your dashboard will be full of events that will never
> > 
> > clear, no logs modified.
> > 
> > Working with the localhost socket is and advanced topic, I have a PR
> > 
> > to have it documented here:
> > 
> > [https://github.com/sensu/sensu-docs/pull/132](https://github.com/sensu/sensu-docs/pull/132)
> > 
> > On Mon, Oct 6, 2014 at 1:33 AM, Alik Kurdyukov [akurdyukov@gmail.com](mailto:akurdyukov@gmail.com) wrote:
> > 
> > > Hello!
> > 
> > > 
> > 
> > > I’m trying to implement sensu-based log checking for my infrastructure.
> > 
> > > 
> > 
> > > Requirements are:
> > 
> > > 1. System should checks in log4j/NLog format. Event should be fired when log
> > 
> > > line level ERROR or FATAL appeared in the log.
> > 
> > > 1. System should not clear event, only admin can delete event saying 'ok,
> > 
> > > I’ve seen it’
> > 
> > > 1. System should not fire error on the same log line twice.
> > 
> > > 1. Logs should not be modified by monitoring.
> > 
> > > 
> > 
> > > Ideal implementation looks like this:
> > 
> > > 1. Sensu checks logs from latest mark to the end of log.
> > 
> > > 1. If some error or exception found - warning or error condition raised,
> > 
> > > mark is not moved. If no error found or previous event deleted by user -
> > 
> > > mark is moved to current end of log.
> > 
> > > 1. Administrator looks at the error description, performs some action and
> > 
> > > deletes event from sensu. No log modification is performed.
> > 
> > > 
> > 
> > > Unfortunately this method cannot be implemented with current Sensu because
> > 
> > > event handler cannot detect situation when event was removed by user.
> > 
> > > 
> > 
> > > Am I missing something and I can implement requirements some other way?
> > 
> > > 
> > 
> > > Regards,
> > 
> > > Alik.

---

<div class="post-metadata">

**Author:** ![Mathieu\_Martin](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Mathieu\_Martin](https://discourse.sensu.io/u/Mathieu_Martin)\
**Post date:** [October 7, 2014, 10:20am UTC](https://discourse.sensu.io/t/log-monitoring/274/4 "2014-10-07T10:20:50Z")

</div>

Hey Alik,

Sensu is good at periodic, stateless checks. Looking inside a log is not stateless (you have to remember the last position).

Even if you create a check that keeps this state and always starts at the last read position, you’ll run into other problems: what if the file was rotated since the last run? Do you want to go look at the rotated file to make sure you didn’t miss anything? What if the rotated file is gzipped? What if rotation happens while you’re looking at the log? Do you need to reopen the new log now to keep looking, or can this wait to the next run of the check?

Your life will be easier if you consider logs as streams, not as files.

For checks based on log events, you’re looking for a log processing pipeline like LogStash, I would think. Log shippers are inherently made to support log rotation and read everything as it comes in. Then LogStash can take an action every time it sees a specific pattern in a log (perform API call, send email, forward log event somewhere else, you pick). Bonus: now you have a log processing pipeline!

Cheers,

Mat

> **···**
>
> On Mon, Oct 6, 2014 at 11:03 AM, Kyle Anderson [kyle@xkyle.com](mailto:kyle@xkyle.com) wrote:
> 
> > If you really want to literally fire an event whenever a certain line
> > 
> > is found, and not automatically clear the event,
> > 
> > then I would make the check utilize the local socket to send an
> > 
> > arbitrary event, and only send CRIT, and never OK.
> > 
> > If you want to make sure that there are separate events and never fire
> > 
> > for the same log line twice, hash the line and put the hash in the
> > 
> > check name.
> > 
> > 1. Sensu check that reads the log and marks an offset
> > 
> > ([http://linux.die.net/man/8/logtail](http://linux.die.net/man/8/logtail)) (check\_name = check\_the\_log)
> > 
> > 1. If an error is found, send a _new_ event to the localhost socket
> > 
> > with part of the hash of the line in the check\_name. (check\_name =
> > 
> > logline\_found\_$hash)
> > 
> > 1. Congrats, your dashboard will be full of events that will never
> > 
> > clear, no logs modified.
> > 
> > Working with the localhost socket is and advanced topic, I have a PR
> > 
> > to have it documented here:
> > 
> > [https://github.com/sensu/sensu-docs/pull/132](https://github.com/sensu/sensu-docs/pull/132)
> > 
> > On Mon, Oct 6, 2014 at 1:33 AM, Alik Kurdyukov [akurdyukov@gmail.com](mailto:akurdyukov@gmail.com) wrote:
> > 
> > > Hello!
> > 
> > > 
> > 
> > > I’m trying to implement sensu-based log checking for my infrastructure.
> > 
> > > 
> > 
> > > Requirements are:
> > 
> > > 1. System should checks in log4j/NLog format. Event should be fired when log
> > 
> > > line level ERROR or FATAL appeared in the log.
> > 
> > > 1. System should not clear event, only admin can delete event saying 'ok,
> > 
> > > I’ve seen it’
> > 
> > > 1. System should not fire error on the same log line twice.
> > 
> > > 1. Logs should not be modified by monitoring.
> > 
> > > 
> > 
> > > Ideal implementation looks like this:
> > 
> > > 1. Sensu checks logs from latest mark to the end of log.
> > 
> > > 1. If some error or exception found - warning or error condition raised,
> > 
> > > mark is not moved. If no error found or previous event deleted by user -
> > 
> > > mark is moved to current end of log.
> > 
> > > 1. Administrator looks at the error description, performs some action and
> > 
> > > deletes event from sensu. No log modification is performed.
> > 
> > > 
> > 
> > > Unfortunately this method cannot be implemented with current Sensu because
> > 
> > > event handler cannot detect situation when event was removed by user.
> > 
> > > 
> > 
> > > Am I missing something and I can implement requirements some other way?
> > 
> > > 
> > 
> > > Regards,
> > 
> > > Alik.

---

<div class="post-metadata">

**Author:** ![Alik\_Kurdyukov](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@Alik\_Kurdyukov](https://discourse.sensu.io/u/Alik_Kurdyukov)\
**Post date:** [October 8, 2014, 7:27am UTC](https://discourse.sensu.io/t/log-monitoring/274/5 "2014-10-08T07:27:49Z")

</div>

Guys,

Thank you for such insightful advices, I managed to set everyone up using Graylog2 + my custom output plugin ([GitHub - akurdyukov/sensu-client-output: Gralog2 output plugin for posting log lines as sense events](https://github.com/akurdyukov/sensu-client-output)) and it’s working. The only problem left is configuration bug in ‘create new event for every new line’, hope guys from Graylog2 will help me fix it.

Final config works like this:

1. NLog (application logging framework) feeds log lines into GrayLog2 using Gelf4NLog

2. Graylog2 separates warn, errors and fatals into separate stream “Errors"

3. Stream “Errors” is fed to local Sensu client by plugin

4. Admin sees errors in Sensu web interface. Admin can resolve an error and be warned again only when new error log line comes.

> **···**
>
> On Mon, Oct 6, 2014 at 11:03 AM, Kyle Anderson [kyle@xkyle.com](mailto:kyle@xkyle.com) wrote:
> 
> > If you really want to literally fire an event whenever a certain line
> > 
> > is found, and not automatically clear the event,
> > 
> > then I would make the check utilize the local socket to send an
> > 
> > arbitrary event, and only send CRIT, and never OK.
> > 
> > If you want to make sure that there are separate events and never fire
> > 
> > for the same log line twice, hash the line and put the hash in the
> > 
> > check name.
> > 
> > 1. Sensu check that reads the log and marks an offset
> > 
> > ([http://linux.die.net/man/8/logtail](http://linux.die.net/man/8/logtail)) (check\_name = check\_the\_log)
> > 
> > 1. If an error is found, send a _new_ event to the localhost socket
> > 
> > with part of the hash of the line in the check\_name. (check\_name =
> > 
> > logline\_found\_$hash)
> > 
> > 1. Congrats, your dashboard will be full of events that will never
> > 
> > clear, no logs modified.
> > 
> > Working with the localhost socket is and advanced topic, I have a PR
> > 
> > to have it documented here:
> > 
> > [https://github.com/sensu/sensu-docs/pull/132](https://github.com/sensu/sensu-docs/pull/132)
> > 
> > On Mon, Oct 6, 2014 at 1:33 AM, Alik Kurdyukov [akurdyukov@gmail.com](mailto:akurdyukov@gmail.com) wrote:
> > 
> > > Hello!
> > 
> > > 
> > 
> > > I’m trying to implement sensu-based log checking for my infrastructure.
> > 
> > > 
> > 
> > > Requirements are:
> > 
> > > 1. System should checks in log4j/NLog format. Event should be fired when log
> > 
> > > line level ERROR or FATAL appeared in the log.
> > 
> > > 1. System should not clear event, only admin can delete event saying 'ok,
> > 
> > > I’ve seen it’
> > 
> > > 1. System should not fire error on the same log line twice.
> > 
> > > 1. Logs should not be modified by monitoring.
> > 
> > > 
> > 
> > > Ideal implementation looks like this:
> > 
> > > 1. Sensu checks logs from latest mark to the end of log.
> > 
> > > 1. If some error or exception found - warning or error condition raised,
> > 
> > > mark is not moved. If no error found or previous event deleted by user -
> > 
> > > mark is moved to current end of log.
> > 
> > > 1. Administrator looks at the error description, performs some action and
> > 
> > > deletes event from sensu. No log modification is performed.
> > 
> > > 
> > 
> > > Unfortunately this method cannot be implemented with current Sensu because
> > 
> > > event handler cannot detect situation when event was removed by user.
> > 
> > > 
> > 
> > > Am I missing something and I can implement requirements some other way?
> > 
> > > 
> > 
> > > Regards,
> > 
> > > Alik.

---

<div class="post-metadata">

**Author:** ![M.sarwar\_Monaf](https://avatars.discourse-cdn.com/v4/letter/m/e79b87/32.png) [@M.sarwar\_Monaf](https://discourse.sensu.io/u/M.sarwar_Monaf)\
**Post date:** [October 22, 2014, 2:21am UTC](https://discourse.sensu.io/t/log-monitoring/274/6 "2014-10-22T02:21:28Z")

</div>

I think the machines in C and D are likely to have different monitoring  
configurations, and the number of machines is a few. Also bandwidth is  
precious here.

> **···**
>
> * * *
> 
> [Outdoor Bluetooth Speaker](https://www.chinavasion.com/china/wholesale/Home_Audio_Video/Bluetooth-Speakers/) | [Bluetooth Speaker](https://www.chinavasion.com/china/wholesale/Home_Audio_Video/Bluetooth-Speakers/)
> 
> On Monday, October 6, 2014 1:33:31 PM UTC+5, Alik Kurdyukov wrote:
> 
> > Hello!
> 
> > I’m trying to implement sensu-based log checking for my infrastructure.
> 
> > 
> 
> > Requirements are:
> 
> > 1. System should checks in log4j/NLog format. Event should be fired when log line level ERROR or FATAL appeared in the log.
> 
> > 1. System should not clear event, only admin can delete event saying ‘ok, I’ve seen it’
> 
> > 1. System should not fire error on the same log line twice.
> 
> > 1. Logs should not be modified by monitoring.
> 
> > 
> 
> > Ideal implementation looks like this:
> 
> > 1. Sensu checks logs from latest mark to the end of log.
> 
> > 1. If some error or exception found - warning or error condition raised, mark is not moved. If no error found or previous event deleted by user - mark is moved to current end of log.
> 
> > 1. Administrator looks at the error description, performs some action and deletes event from sensu. No log modification is performed.
> 
> > 
> 
> > Unfortunately this method cannot be implemented with current Sensu because event handler cannot detect situation when event was removed by user.
> 
> > 
> 
> > Am I missing something and I can implement requirements some other way?
> 
> > 
> 
> > Regards,
> 
> > Alik.