Blue Light
Space shortcuts
Child pages
  • Logcheck administration
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 17 Next »

Versions

The information on this page was developed and tested on Debian 6.0 (Squeeze) with logcheck 1.3.13.

References

  1. Main documentation:
        Online: http://logcheck.org/docs/
        As installed: directories /usr/share/doc/logcheck and /usr/share/doc/logcheck-database.
            The .gz files may conveniently be read using zcat and less.  For example:
            zcat /usr/share/doc/logcheck-database/README.logcheck-database.gz | less
  2. logcheck man page (HTML format): http://linux.die.net/man/8/logcheck

 

How does logcheck work?

This description applies to a default installation on Debian.  The default configuration may be different for other distros.  The configuration may have been changed since installation.

Logcheck is run hourly by cron and selects log messages to mail to root.  It reads each new line (message) from /var/log/syslog and /var/log/auth.log.  For each message, logcheck:

  1. Sees if the message matches a regular expression in one of the files in /etc/logcheck/cracking.d.  If it does, logcheck selects the message for the ATTACK ALERT section of the email and continues with the next message. 
  2. Sees if the message matches a regular expression in one of the files in /etc/logcheck/violations.d.  If it does, logcheck sees if the message matches a regular expression in one of the files in /etc/logcheck/violations.ignore.d.  If it does, logcheck ignores the message and continues with the next message.  If it does not, logcheck selects the message for the SECURITY EVENTS section of the email and continues with the next message.
  3. Sees if the message matches a regular expression in one of the files in /etc/logcheck/ignore.d.server.  If it does, logcheck discards the message and continues with the next message.  If it does not, logcheck selects the message for the SYSTEM EVENTS section of the email.

Note: in the first two steps, a match selects the message.  In the last step a match discards the message.

If logcheck doesn't select any messages for the email, the email is not sent.

Terminology and naming

The logcheck documentation uses "filter", "pattern" and  "rule" interchangeably, applying them to directories, files and individual regular expressions.

On this WIKI page, only "filter" is used.  It is used only to mean a single filter (a line in a file).  A file of filters is called a "filter file" on this page.

Logcheck is developed by Debian.  The filter files developed by Debian for generic messages are called "logcheck".  The rest of the filter files are named after the Debian packages that include the software that may generate the messages they have filters for.

Unwanted messages under ATTACK ALERT in the emails

If the messages should be filtered out:

  1. Enable /etc/logcheck/cracking.ignore.d by setting SUPPORT_CRACKING_IGNORE to 1 in logcheck.conf
  2. Develop one or more filters for the messages
  3. Put the filter(s) in a filter file in /etc/logcheck/cracking.ignore.d

Note: the messages are discarded and do not appear anywhere in the email.

Unwanted messages under SECURITY EVENTS in the emails

If the messages should be filtered out:

  1. Develop one or more filters for the messages
  2. Put the filter(s) in a filter file in /etc/logcheck/violations.ignore.d

Note: the messages are discarded and do not appear anywhere in the email.

Unwanted messages under SYSTEM EVENTS in the emails

This section applies in the most common case when unwanted messages appear in the SYSTEM EVENTS section of the logcheck emails.

The first task is to decide whether the unwanted messages should be filtered out:

  • If a message shows there's an issue to be resolved, filtering the message out would hide essential information.
  • If a message is generated rarely, it is not worth the work of filtering it out.  Server boot messages are a good example.
  • Often it is not clear from the message itself whether it shows there's an issue to be be resolved or not.  Research work is required.

If the messages should be filtered out, the next task is to develop one or more regular expressions (filters) that match the messages.

For a single filter, when it works and assuming logcheck has the default configuration, the next step is to put it in a file in /etc/logcheck/ignore.d.server – either in an existing file or a new one.

Custom filters

"Custom filters" is used here to mean ones not installed with the logcheck package.  They might be locally developed or they might come from third parties or later versions of logcheck.

Third party filters

Third party filters might be supplied to go with a particular software (such as snoopy, filters here) or developed to extend the logcheck standard filters (such as Ties de Kock's set of local-* filter files here).

Later logcheck filters

Sometimes, especially when existing filters fail because the later version of a package produces differently formatted messages, a later version of logcheck has suitable filter files.  The latest can be found in the logcheck git repository.

Choosing filter file names for custom filters

Custom filters can most easily be kept in local-<something> filter files.  The alternatives are harder to administer:

  • Changing the files installed by the logcheck package means those files will not be upgraded when logcheck is upgraded.
  • A single file for all local filters does not allow installing a local filter file at the same time as a package is installed or upgraded (the format of the log messages may change with the upgrade).
  • Having multiple local filter files is helpful when a filter has accidentally been added which matches too many messages; individual files can be disabled until the file causing the problem is identified (a powerful technique when the first step is to to disable half the files, and so on in a "binary search").
  • If the filters are for messages generated by software not installed by a package (such as locally developed software or software installed from source), it is convenient to have a local-<something> file to install with the software in the same way that it might have files for /etc/cron.daily or /etc/logrotate.d.  The local- prefix ensures such a file will not be clobbered by a package that happens to have the same name.

If the local filter extends the set of filters in an as-installed <package name> filter file, the relationship can be made clear by putting it in local-<same package name> filter file.

Developing filters

Find a similar example

For example, suppose logcheck was mailing something like

May 23 10:08:33 LS1 nmbd[1199]: *****

The part in green is usually the name of a daemon or command.

Find files with filters for similar messages by

cd /etc/logcheck/ignore.d.server && grep --files-with-matches nmbd *

Choose one of the filters in the file(s) listed as the basis for developing the new one.  If no files are listed, take any filter – most (all?) messages match the message format above up to the last ":" except for the nmbd string.

Structure of a typical filter

A filter is a regular expression, specifically an egrep regular expression, for example looking like this:

^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ (openvpn|ovpn-[._[:alnum:]-]+)\[[[:digit:]]+\]:( ([-_.@[:alnum:]]+/)?[.[:digit:]]{7,15}:[[:digit:]]{2,5})? Replay-window backtrack occurred \[[[:digit:]]+\]$

  1. To ensure a complete match, filters begin with ^ and end with $ .
  2. The grey part matches from start of the line through the time stamp and server name.  It is common to most (all?) filters.
  3. The green part identifies the daemon or command.
  4. The blue part matches the process' PID shown in square brackets and followed by a :
  5. The rest is specific to the particular message.

Writing a filter

When writing a filter, you can almost certainly use parts 1 to 4 above from a filter for the same package and make up your own part 5.  The filter must be a POSIX extended regular expression matching the message you want to filter out.  The idea is to match only the messages you want to filter. 

If your regex is too general it may filter messages that show there's an issue to be resolved.  For example ...

    ^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ (openvpn|ovpn-[._[:alnum:]-]+)\[[[:digit:]]+\]:.*$

... would filter out all openvpn log messages.

If your regex is too specific you may need several filters to address closely related messages.  For example ... [TODO: add example]

If you are not familiar with extended regular expressions, it may be worth studying some existing logcheck filters alongside samples of the messages they filter to see how they work.

Extended regular expression references:

Testing and debugging

Regexes are tested by seeing if they match the message in the log file.  The first step is to find which log file the message came from.

The rest of this article assumes the message came from /var/log/syslog.

A convenient technique is to have the filter regexp in a file and to discard the output from egrep:

    egrep --quiet --file /tmp/my_filter /var/log/syslog && echo found || echo not found

Having the regexp in a file allows it to be easily changed and avoids the shell modifying it on the command line before it is passed to egrep.

If the regex does not work these can help:

  1. Progressively right-truncate the regex until it does match.  Then the last part discarded contains the error.  Having the regex in an editor with undo/redo helps.
  2. Drop egrep's --quiet option and add the --only-matching (-o) option.  This shows what the regex is actually matching.

Further information (includes tips on writing and testing filters):

zcat /usr/share/doc/logcheck-database/README.logcheck-database.gz | less

Notes on README.logcheck-database:

  • If using sort to order filter files as suggested, sort's --version-sort (-V) option may be required.
  • Multiple local-<package name> files have the advantage of being installable and removable with the associated package.

 

Installing filters in a filter file

The directory for the filter file is as listed in "Unwanted messages under ATTACK ALERT in the emails", "Unwanted messages under SECURITY EVENTS in the emails" or "Unwanted messages under SYSTEM EVENTS in the emails" above.

The name of the filter file could be chosen with reference to Choosing filter file names for custom filters above.

Filter files can be sorted to help identify duplicate and similar filters.  If doing so, using sort's --version-sort (-V) option may avoid surprises.

Gotchas

If a successfully tested filter does not work in production:

  • Are there any trailing spaces on the message in the log?  logcheck automatically removes these before looking for filter matches.  This behaviour can be mimicked during testing by:
        sed -e 's/[[:space:]]*$//' /var/log/<log name> | ...

If a valid filter in a filter file in /etc/logcheck/ignore.d.server filter does not work:

  • Does the message match a filter in a filter file in /etc/logcheck/cracking.d or /etc/logcheck/violations.d?  Note: if this is the case, the unfiltered message would not be in the SYSTEM EVENTS section of the emails.
  • No labels