Versions
The information on this page was developed and tested on Debian 6.0 (Squeeze) with logcheck 1.3.13.
References
- Main documentation:
Online: http://logcheck.org/docs/
As installed: directories /usr/share/doc/logcheck and /usr/share/doc/logcheck-database.
The .gz files may conveniently be read using zcat and less. For example:zcat /usr/share/doc/logcheck-database/README.logcheck-database.gz | less - logcheck man page (HTML format): http://linux.die.net/man/8/logcheck
How does logcheck work?
This description applies to a default installation on Debian. The default configuration may be different for other distros. The configuration may have been changed since installation.
Logcheck is run hourly by cron and selects log messages to mail to root. It reads each new line (message) from /var/log/syslog and /var/log/auth.log. For each message, logcheck:
- Sees if the message matches a regular expression in one of the files in /etc/logcheck/cracking.d. If it does, logcheck selects the message for the "Attack alert" section of the email and continues with the next message.
- Sees if the message matches a regular expression in one of the files in /etc/logcheck/violations.d. If it does, logcheck sees if the message matches a regular expression in one of the files in /etc/logcheck/violations.ignore.d. If it does, logcheck ignores the message and continues with the next message. If it does not, logcheck selects the message for the "Security events" section of the email and continues with the next message.
- Sees if the message matches a regular expression in one of the files in /etc/logcheck/ignore.d.server. If it does, logcheck discards the message and continues with the next message. If it does not, logcheck selects the message for the "System events" section of the email.
Note: in the first two steps, a match selects the message. In the last step a match discards the message.
If logcheck doesn't select any messages for the email, the email is not sent.
Terminology and naming
The logcheck documentation uses "filter", "pattern" and "rule" interchangeably, applying them to directories, files and individual regular expressions.
On this WIKI page, only "filter" is used. It is used only to mean a single filter (a line in a file). A file of filters is called a "filter file".
Logcheck is developed by Debian. The filter files developed by Debian for generic messages are called "logcheck". The rest of the filter files are named after the Debian packages that include the software that may generate the messages they have filters for.
How can messages be filtered out of the "System events"?
This section applies in the most common case when unwanted messages appear in the "System events" section of the logcheck email.
If a message shows there's an issue to be resolved, filtering the message out would hide essential information.
If a message is generated rarely, it is not worth the work of filtering it out. Server boot messages are a good example.
Often it is not clear from the message itself whether it shows there's an issue to be be resolved or not. Research work is required.
How can a message be filtered in or out?
Once a A filter is a regular expression matching the message to be filtered in or out.
Where to ich files to put locally developed filters in?
Locally developed filters can most easily be kept in /etc/logcheck/ignore.d.server/local-<package name> files.
The alternatives are harder to administer:
- Changing the files installed by the logcheck package means those files will not be upgraded when logcheck is upgraded.
- A single file for all local filters does not allow installing a local filter file at the same time as a package is installed or upgraded (the format of the log messages may change with the upgrade).
- Having multiple local filter files is helpful when a filter has accidentally been added which matches too many messages; individual files can be disabled until the file causing the problem is identified (a powerful technique when the first step is to to disable half the files, and so on in a "binary search").
Are more filters available?
Each release of logcheck has new filters. The logcheck git has all the latest filters @@@ development version's git
Ties de Kock's personal set of local-* files: https://github.com/ties/logcheck-extrarules/tree/master/ignore.d.server
Debian's latest
How to develop a filter?
Find a similar example
For example, suppose logcheck was mailing something like
May 23 10:08:33 LS1 nmbd[1199]: *****
The part in green is usually the name of a daemon or command.
Find files with filters for similar messages by
cd /etc/logcheck/ignore.d.server && grep --files-with-matches nmbd *
Choose one of the filters in the file(s) listed as the basis for developing the new one. If no files are listed, take any filter – most (all?) messages match the message format above up to the last ":" except for the nmbd string.
Structure of a typical filter
A filter is a regular expression, specifically an egrep regular expression, for example looking like this:
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ (openvpn|ovpn-[._[:alnum:]-]+)\[[[:digit:]]+\]:( ([-_.@[:alnum:]]+/)?[.[:digit:]]{7,15}:[[:digit:]]{2,5})? Replay-window backtrack occurred \[[[:digit:]]+\]$
- To ensure a complete match, filters begin with ^ and end with $ .
- The grey part matches from start of the line through the time stamp and server name. It is common to most (all?) filters.
- The green part identifies the daemon or command.
- The blue part matches the process' PID shown in square brackets and followed by a :
- The rest is specific to the particular message.
Writing a filter
When writing a filter, you can almost certainly use parts 1 to 4 above from a filter for the same package and make up your own part 5. The filter must be a POSIX extended regular expression matching the message you want to filter out. The idea is to match only the messages you want to filter.
If your regex is too general it may filter messages that show there's an issue to be resolved. For example ...
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ (openvpn|ovpn-[._[:alnum:]-]+)\[[[:digit:]]+\]:.*$
... would filter out all openvpn log messages.
If your regex is too specific you may need several filters to address closely related messages. For example ... [TODO: add example]
If you are not familiar with extended regular expressions, it may be worth studying some existing logcheck filters alongside samples of the messages they filter to see how they work.
Extended regular expression references:
- Man pages: grep (includes egrep) and regex (man pages section 7, not 3). Online at http://linux.die.net/man/1/grep and http://linux.die.net/man/7/regex.
- Wikipedia: http://en.wikipedia.org/wiki/Regular_expression#POSIX_Extended_Regular_Expressions.
- The Linux Documentation Project: http://tldp.org/LDP/abs/html/x17046.html.
Testing and debugging
Regexes are tested by seeing if they match the message in the log file. The first step is to find which log file the message came from.
The rest of this article assumes the message came from /var/log/syslog.
A convenient technique is to have the filter regexp in a file and to discard the output from egrep:
egrep --quiet --file /tmp/my_filter /var/log/syslog && echo found || echo not found
Having the regexp in a file allows it to be easily changed and avoids the shell modifying it on the command line before it is passed to egrep.
If the regex does not work these can help:
- Progressively right-truncate the regex until it does match. Then the last part discarded contains the error. Having the regex in an editor with undo/redo helps.
- Drop egrep's --quiet option and add the --only-matching (-o) option. This shows what the regex is actually matching.
Further information (includes tips on writing and testing filters):
zcat /usr/share/doc/logcheck-database/README.logcheck-database.gz | less
Notes on README.logcheck-database:
- If using sort to order filter files as suggested, sort's --version-sort (-V) option may be required.
- Multiple local-<package name> files have the advantage of being installable and removable with the associated package.
Installing the filter
To install a filterAssuming you have just successfully tested a filter for daemon/command foo ...
If the local filters file for foo, /etc/logcheck/ignore.d.server/local-foo, already exists, optionally make a backup. logcheck ignores files with a . in the name so local-foo.bak would be OK.
Add the new filter to local-foo on a line of its own.
Optionally sort local-foo. This will help identify duplicate and similar filters.
Assuming your newly tested filter is in /tmp/my_filter, this could all be done with something like:
cmd=foo; my_new_filter=/tmp/my_filter; cd /etc/logcheck/ignore.d.server \ && mv --force local-$cmd local-$cmd.bak \ && cat local-$cmd.bak$my_new_filtersort --version-sort > local-$cmd
Gotchas
If a valid filter does not pass testing:
- Are there any trailing spaces on the message in the log? logcheck automatically removes these before looking for filter matches. This behaviour can be mimicked during testing by:
sed -e 's/[[:space:]]*$//' /var/log/<log name> | ...
If a tested filter does not work:
- Does the message match logcheck's "SECURITY ALERTS" (defined in /etc/logcheck/cracking.d/* files) or "SECURITY EVENTS" (defined in /etc/logcheck/violations.d/* files)? If so, the solution is described in README.logcheck-database.
XXX