The information on this page is based on working with logcheck on Squeeze, Wheezy and Jessie: 1.3.13, 1.3.15 and 1.3.17.
zcat /usr/share/doc/logcheck-database/README.logcheck-database.gz | less
This description applies to a default installation on Debian. The default configuration may be different for other distros. The configuration may have been changed since installation.
Logcheck is run via /etc/cron.d/logcheck. /etc/cron.d/logcheck runs logcheck at reboot and at 2 minutes past every hour.
Logcheck's scheduled job can be disabled, for example by:
mv /etc/cron.d/logcheck{,.disabled}
Logcheck reads each new line (message) from /var/log/syslog and /var/log/auth.log. For each message:
Note: in the first two steps, a match selects the message. In the last step a match discards the message.
If logcheck doesn't select any messages for the email, the email is not sent.
The subject of the mail is prefixed with "Reboot: " when logcheck is run at boot.
Mail is sent to logcheck.
The "from" address is "logcheck system account".
"ignore.d.workstation. "... is only appropriate for relatively sheltered, non-critical machines"
If the messages should be filtered out:
Note: the messages are discarded and do not appear anywhere in the email.
If the messages should be filtered out:
Note: the messages are discarded and do not appear anywhere in the email.
This section applies in the most common case when unwanted messages appear in the SYSTEM EVENTS section of the logcheck emails.
The first task is to decide whether the unwanted messages should be filtered out:
If the messages should be filtered out, the next task is to develop one or more regular expressions (filters) that match the messages.
For a single filter, when it works and assuming logcheck has the default configuration, the next step is to put it in a file in /etc/logcheck/ignore.d.server – either in an existing file or a new one.
Typically logcheck sends messages associated with booting the first time it runs after reboot.
This can be fixed by modifying /etc/cron.d/logcheck, inserting a sleep that delays logcheck until the boot sequence has finished:
@reboot logcheck if [ -x /usr/sbin/logcheck ]; then sleep 10; nice -n10 /usr/sbin/logcheck -R; fi
Explanation: during boot, logcheck runs when the cron daemon is started. This is normally part way through the boot sequence. Any messages already in the logs are sent in the mail with subject "Reboot: ...". This is done because it is not practicable to have logcheck filters for all the messages generated during boot. Any later messages are sent in the next mail, subject to the usual filtering.
This section applies when messages appear in the logs but do not appear in the emails.
Is the log with the message a log that logcheck is searching? logcheck searches the logs configured in /etc/logcheck/logcheck.logfiles (default /var/log/syslog and /var/log/auth.log) or whichever file is nominated by logcheck's -L option.
Is the message matched by a filter in an ATTACK ALERT or SECURITY EVENT ignore filter file?
If the log is being searched and the message is not matched by an ignore filter file, the message must be matched by a SYSTEM EVENTS filter. In a default installation these are in filter files in the /etc/logcheck/server.d directory, otherwise they are in the /etc/logcheck/<report level>.d directory where <report level> is the value of REPORTLEVEL in /etc/logcheck.conf.
Disable the cron job. For example:
mv /etc/cron.d/logcheck{,.disabled}
"Custom filters" is used here to mean ones not installed with the logcheck package. They might come from third parties or later versions of logcheck. They might be developed locally.
Sometimes software includes filter files, for example snoopy, filters here.
There are some published filter file libraries to extend the logcheck standard filters, for example:
git clone git://git.bluelightav.org/logcheck
When considering third party filters, be aware that they may be designed to suit particular local conditions so not as widely suitable as filters from logcheck and other packages. For example, the Blue Light filter files are designed to suit low reliability Internet connections; in other locations, messages which we filter out as routine would indicate an unusual event requiring attention.
Sometimes, especially when existing filters fail because a later version of a package produces differently formatted messages, a later version of logcheck has suitable filter files. The latest can be found in the logcheck git repository.
For example, suppose logcheck was mailing something like
May 23 10:08:33 LS1 nmbd[1199]: *****
The part in green is usually the name of a daemon or command. In some cases it is more. For example ovpn-client uses ovpn-client.<configuration file name>.
Find filter files for similar messages by, for example
cd /etc/logcheck/ignore.d.server && grep --files-with-matches ovpn-client *
Choose one of the filters in the file(s) listed as the basis for developing a new one. If no files are listed, choose any filter; almost all messages match the message format above up to the last ":" except for the part in green in the example.
A filter is an egrep regular expression, for example:
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ (openvpn|ovpn-[._[:alnum:]-]+)\[[[:digit:]]+\]:( ([-_.@[:alnum:]]+/)?[.[:digit:]]{7,15}:[[:digit:]]{2,5})? Replay-window backtrack occurred \[[[:digit:]]+\]$
Notes:
When developing a filter, you can usually use parts 1 to 4 above from a similar filter and develop part 5. It should be designed to match the messages you want to filter in or out – and only those messages.
If your regex is too general it may filter messages that show there's an issue to be resolved. For example ...
^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ (openvpn|ovpn-[._[:alnum:]-]+)\[[[:digit:]]+\]:.*$
... would filter out all openvpn log messages.
If your regex is too specific you may need several filters to address closely related messages. For example ... [TODO: add example]
If you are not familiar with egrep regular expressions, it may be worth studying some existing logcheck filters alongside samples of the messages they filter to see how they work.
(Mon|Tue|Wed|Thu|Fri|Sat|Sun)
(/dev/)?(sd[[:lower:]]+[[:digit:]]*
)|(dm-[[:digit:]]
+)
[-[:alnum:]]+(\.[-[:alnum:]]+)*
[:[:digit:]]{8}
[[:digit:]]+(.[[:digit:]
]+){3}
[.0-9]{7,15}
[[:digit:]]{1,5}
[[:xdigit:]]*:*[:[:xdigit:]
]*:+[[:xdigit:]
]+
(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)
[^ ]+
[[:upper:]]{3}
Extended regular expression references:
Filters are tested by seeing if they match sample message(s).
The first step is to find which log the sample message(s) are in. In a default logcheck installation this will be either /var/log/syslog or /var/log/auth.log. /var/log/syslog is assumed in the examples below.
If /var/log/syslog has been rotated since logcheck found the message(s), they will be in syslog.1 or syslog.*.gz. They can all be searched using:
zgrep '<your string>' /var/log/syslog*
Having the filter(s) in a file in an editor is useful. Then it's easy to change, redo and undo between tests. Assuming the filter(s) to be tested are in /tmp/my_filter and the messages are in /var/log/syslog, they can be tested by:
sed -e 's/[[:space:]]*$//'
/var/log/syslog
| egrep --file /tmp/my_filter
Note: the sed -e 's/[[:space:]]*$//'
emulates logcheck internals which strip trailing spaces from messages.
When the messages are in a compressed log:
zcat
/var/log/syslog.3.gz
| sed -e 's/[[:space:]]*$//' | egrep --file /tmp/my_filter
When the messages are in compressed and uncompressed logs:
zgrep --no-filename . <list of log files>
| sed -e 's/[[:space:]]*$//'
\
| egrep --file /tmp/my_filter
If the filter does not work these can help:
For more tips on writing and testing filters:
zcat /usr/share/doc/logcheck-database/README.logcheck-database.gz | less
Notes on README.logcheck-database:
The directory for the filter file is as listed in "Unwanted messages under ATTACK ALERT in the emails", "Unwanted messages under SECURITY EVENTS in the emails" or "Unwanted messages under SYSTEM EVENTS in the emails" above.
Filter file names must be made of upper and lower case letters, digits, underscores, and hyphens. This is because logcheck uses run-parts --list
to select the filter files.
Custom filters can most easily be kept in local-<something> filter files. The alternatives are harder to administer:
If the local filter extends the filters in an as-installed <package name> filter file, the relationship can be made clear by putting it in local-<same package name>.
If the package name is not obvious from the message, it can usually be found by finding which file corresponds to the message and which package the file comes from. For example, on Debian for message Jul 12 09:19:55 LS1 named[1329]: listening on IPv4 interface tun0, 10.8.0.1#53
the package is bind9: # type named
named is /usr/sbin/named
# dpkg -S /usr/sbin/named
bind9: /usr/sbin/named
In case a standard set of local-* filter files are installed on multiple computers and the set needs to be extended, a different naming convention is needed for the extra files. They may, for example, be for a specific role, configuration or defect. They could be called local-local-*.
If updating an existing file, the original can be backed up with a name which is not entirely of upper and lower case letters, digits, underscores, and hyphens; for example local-foo.bak or local-foo~. These files will not be used by logcheck.
Filter files may have comment lines (beginning with #) and empty lines (containing only none or more spaces and tabs). These are ignored by logcheck.
Filter files can be sorted to help identify duplicate and similar filters. If doing so, using sort's --version-sort (-V) option may avoid surprises.
The message was "Sep 16 17:33:31 rose gnome-keyring-prompt: could not grab keyboard: already grabbed"
Found the package:root@rose:/etc/logcheck/ignore.d.server# dpkg -L gnome-keyring | grep gnome-keyring-prompt
/usr/share/applications/gnome-keyring-prompt.desktop
/usr/lib/gnome-keyring/gnome-keyring-prompt-3
/usr/lib/gnome-keyring/gnome-keyring-prompt
Found no existing filters:root@rose:/etc/logcheck/ignore.d.server# grep gnome-keyring-prompt *
[no output]
Found the log with the message:root@rose:/etc/logcheck/ignore.d.server# grep 'could not grab keyboard' /var/log/syslog
root@rose:/etc/logcheck/ignore.d.server# grep 'could not grab keyboard' /var/log/syslog.1
root@rose:/etc/logcheck/ignore.d.server# grep 'could not grab keyboard' /var/log/auth.log
Sep 16 17:33:31 rose gnome-keyring-prompt: could not grab keyboard: already grabbed
Developed a filter by copying an existing filter and modifying it. Tested it:root@rose:/etc/logcheck/ignore.d.server# egrep --file /tmp/my_filter /var/log/auth.log
Sep 16 17:33:31 rose gnome-keyring-prompt: could not grab keyboard: already grabbed
Added filter to existing local filter file:root@rose:/etc/logcheck/ignore.d.server# cat /tmp/my_filter >> local-gnome-keyring
If a successfully tested filter does not work in production:
sed -e 's/[[:space:]]*$//' /var/log/<log name> | ...
If a valid filter in a filter file in /etc/logcheck/ignore.d.server filter does not work: