"My" FreeBSD system somehow rebooted itself last friday in the early
hours in the morning, and since then /var/log/messages is full with
messages like these:
May 30 10:38:28 fourquid root: ZFS: zpool I/O failure, zpool=tank error=86
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da3
offset=278593630720 size=7680
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da4
offset=278593630720 size=7680
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da5
offset=278593630720 size=7680
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da0
offset=278593631232 size=7680
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da1
offset=278593631232 size=7680
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da2
offset=278593631232 size=7680
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da3
offset=278593630720 size=7680
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da4
offset=278593630720 size=7680
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da5
offset=278593630720 size=7680
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da0
offset=278593631232 size=7680
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da1
offset=278593631232 size=7680
May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank path=/dev/da2
offset=278593631232 size=7680
i.e. errors op all disks that make up the pool. There appear to be more
than just the two offsets as mentioned in this block of messages but
so far it seems the number of them is limited.
Seemingly they stopped around the time I logged in.
I started a scrub, just in case. zpool status -v says this:
fourquid.1:~$ sudo zpool status -v
pool: tank
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: scrub in progress for 0h23m, 4.70% done, 7h49m to go
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 6.15K
raidz2 ONLINE 0 0 24.5K
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
da5 ONLINE 0 0 0
errors: Permanent errors have been detected in the following files:
tank/vol-fourquid-1:<0x0>
tank/vol-fourquid-1:<0xc8190da>
How do I identify which files it is listing here?
-Olaf.
--
Pipe rene = new PipePicture(); assert(Not rene.GetType().Equals(Pipe));
On Mon 30 May 2011 at 11:35:46 +0200, Olaf Seibert wrote:> How do I identify which files it is listing here?The nighly 'find' has found the files for me. It is actually a bunch of directories, that were likely in use when the crash occurred. They give an "interesting" error when you try to ls them: find: /tank/vol-fourquid-1/evadh/CLEF-IP11/PARSED_CORPUS/EP/000000/45/97: Illegal byte sequence The file system is compressed, that may be the reason it can identify "illegal byte sequence"s. It isn't even possible to rm -r the directories, or even to mv them... (fortunately the standard trick works: move the parent directory instead, create a new one in its old place, and move the old, good, contents back). but now I seem to be left with some directories (elsewhere) that I still can't remove...> -Olaf. > -- > Pipe rene = new PipePicture(); assert(Not rene.GetType().Equals(Pipe));
Hi, 2011/5/30 Olaf Seibert <O.Seibert@cs.ru.nl>> "My" FreeBSD system somehow rebooted itself last friday in the early > hours in the morning, and since then /var/log/messages is full with > messages like these: > > May 30 10:38:28 fourquid root: ZFS: zpool I/O failure, zpool=tank error=86 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da3 offset=278593630720 size=7680 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da4 offset=278593630720 size=7680 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da5 offset=278593630720 size=7680 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da0 offset=278593631232 size=7680 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da1 offset=278593631232 size=7680 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da2 offset=278593631232 size=7680 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da3 offset=278593630720 size=7680 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da4 offset=278593630720 size=7680 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da5 offset=278593630720 size=7680 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da0 offset=278593631232 size=7680 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da1 offset=278593631232 size=7680 > May 30 10:38:38 fourquid root: ZFS: checksum mismatch, zpool=tank > path=/dev/da2 offset=278593631232 size=7680 > >looks like memory errors to me, check your RAM with memtest. Mickael