Logcheck devel - Dec 2006 - Bug#401259: logcheck: logcheck needs to override locale for grep

Package: logcheck
Version: 1.2.51
Severity: normal

Logcheck has an implicit assumption that the default locale should be
used by grep when processing log files.  However, that's not always
the case.  For example, I use the locale "en_US.UTF-8", and
consequently grep assumes that its inputs are encoded as UTF-8.  But
the log files appear to be encoded as ISO 8859-1, which means that
sometimes my rules don't match.

Specifically, I have a rule that reads 

^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: input: .* as
/class/input/input[0-9]+$

which is supposed to ignore messages from the kernel announcing input
devices.  But the following log line isn't handled right:

Dec  2 00:05:01 ravna kernel: input: Microsoft Microsoft Wireless Optical Mouse?
1.0A as /class/input/input3

The reason it doesn't match is that the "R in a circle" character
is
encoded in the log file as using the ISO 8859-1 code 0xae, but this
isn't a valid first byte of a UTF-8 code.  Consequently, the "."
pattern doesn't match it.  In fact, I don't think there's _any_ way
to
match this byte sequence in a UTF-8 locale.

Unfortunately I'm not sure what to do about this, because it's not
obvious how the log-file messages relate to the locale.  This message
comes from the kernel, which presumably doesn't know what the locale
is.  Furthermore, this particular text is coming directly from the
device, and just being passed along by the kernel -- I have no idea if
USB specifies the character coding that is used in these strings, or
if it's just an uninterpreted sequence of bytes that are encoded any
way the manufacturer pleases.

One thing that works in this case is to set "LC_ALL=C" prior to
calling grep.  But if the log files sometimes contain UTF-8 coding,
this will mess that up  Perhaps the kernel log lines need to be
handled differently?

I hope you have a better idea about how to handle this.

-- System Information:
Debian Release: 4.0
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.19-cph1
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)

Versions of packages logcheck depends on:
ii  adduser          3.100                   Add and remove users and groups
ii  cron             3.0pl1-99               management of regular background p
ii  debconf          1.5.9                   Debian configuration management sy
ii  exim4            4.63-10                 metapackage to ease exim MTA (v4) 
ii  exim4-daemon-lig 4.63-10                 lightweight exim MTA (v4) daemon
ii  grep             2.5.1.ds2-6             GNU grep, egrep and fgrep
ii  lockfile-progs   0.1.10                  Programs for locking and unlocking
ii  logtail          1.2.51                  Print log file lines that have not
ii  mailx            1:8.1.2-0.20050715cvs-1 A simple mail user agent
ii  sysklogd [system 1.4.1-20                System Logging Daemon

Versions of packages logcheck recommends:
ii  logcheck-database             1.2.51     database of system log rules for t

-- debconf information:
  logcheck/changes:
  logcheck/install-note:

On Sat, Dec 02, 2006 at 01:17:28AM -0500, Chris Hanson
wrote:
> The reason it doesn't match is that the "R in a circle"
character is
> encoded in the log file as using the ISO 8859-1 code 0xae, but this
> isn't a valid first byte of a UTF-8 code.  Consequently, the
"."
> pattern doesn't match it.  In fact, I don't think there's _any_
way to
> match this byte sequence in a UTF-8 locale.
I guess [eg]libc's regex functions are a bit strict about their input.
However, grep also comes with its own DFA-based functions, which are
more lax about encoding errors; they are normally skipped for multibyte
encodings, but can be forced with GREP_USE_DFA=1.
> Unfortunately I'm not sure what to do about this, because it's not
> obvious how the log-file messages relate to the locale.  This message
They don't, at least not reliably.  There's stuff in there, like ssh
usernames, that comes directly from nefarious people who don't give a
rat's ass about your particular selection of encoding.
> One thing that works in this case is to set "LC_ALL=C" prior to
> calling grep.  But if the log files sometimes contain UTF-8 coding,
> this will mess that up
I doubt this would be a problem.  Pretty much everything that is matched
explicitly in any rule (hostname, IP address, process ID) is in ASCII.
Any chunk of arbitrary data should be matched with something like .* or
[^[:space:]]+, which will work whether it was decoded or not.

Now, it's true that POSIX restricts the "C" locale to 7-bit
characters,
but both grep and elibc appear to deal with binary characters just fine.


One unfortunate side-effect is that any error messages from grep will
therefore be in English, but that's probably a lesser evil.
(LC_MESSAGES cannot be left as is, since mixing different encodings is
not supported.)


-- 
Never trust an operating system you don't have sources for. ;-)
		-- Unknown source

Note to whoever will fix this: take the occasion to revert 30221d3 while
you're at it.


-- 
A Linux machine!  Because a 486 is a terrible thing to waste!
		-- Joe Sloan, jjs at wintermute.ucr.edu