thr3ads.net - freebsd stable - Heavy fs corruption with 9.0-RELEASE [Mar 2012]

If this information is useful, please help other people find it:
Share via:

Arnaud Lacombe

2012-Mar-05 22:12 UTC

Heavy fs corruption with 9.0-RELEASE

Hi,

I've been running a couple of system with 9.0-RELEASE since it is out.
All the system were installed through the standard installation
procedure. After unclean reboot, either crash or power-failure, I get
a huge amount of really bad filesystem corruption (read: "silent",
fs-wide, corruptions). This happens with either i386 or amd64 build.
Systems involved use compact flash as their system permanent storage
medium.

Typical symptoms are:

[during rc startup]
Starting sshd.
/usr/lib/libkrb5.so.10: invalid file format/etc/rc: WARNING: failed to
start sshd
/usr/libexec/sendmail/sendmail: Undefined symbol
"SSL_library_init"/usr/libexec/sendmail/sendmail: Undefined symbol
"SSL_library_init"Starting cron.

[after startup, dropped in single user, remount / read-only + ran
`fsck -y /' and went back multi-user]
Starting sshd.
Segmentation fault
Mar  5 18:07:38 test kernel: Failed to write core file for process
sshd (error 14)
/etc/rc: WARNING: failed to start sshd
Segmentation fault
Segmentation fault
Starting cron.
/usr/lib/libgnuregex.so.5: invalid file
format/usr/lib/libgnuregex.so.5: invalid file formatStarting
background file system checks in 60 seconds.

well, something looks broken, let's investage...

# file /usr/include/* | tail -20
/usr/include/ulog.h:            broken symbolic link to `liblzma.so.5'
/usr/include/unctrl.h:          ELF 64-bit LSB shared object, x86-64,
version 1 (FreeBSD), dynamically linked, stripped
/usr/include/unistd.h:          ELF 64-bit LSB shared object, x86-64,
version 1 (FreeBSD), dynamically linked, stripped
/usr/include/usb.h:             current ar archive
/usr/include/usbhid.h:          current ar archive
/usr/include/utempter.h:        current ar archive
/usr/include/utime.h:           broken symbolic link to `librt.so.1'
/usr/include/utmpx.h:           current ar archive
/usr/include/uuid.h:            current ar archive
/usr/include/varargs.h:         ELF 64-bit LSB shared object, x86-64,
version 1 (FreeBSD), dynamically linked, stripped
/usr/include/vgl.h:             ELF 64-bit LSB shared object, x86-64,
version 1 (FreeBSD), dynamically linked, stripped
/usr/include/vis.h:             broken symbolic link to `librtld_db.so.2'
/usr/include/vm:                directory
/usr/include/wchar.h:           current ar archive
/usr/include/wctype.h:          current ar archive
/usr/include/wordexp.h:         ELF 64-bit LSB shared object, x86-64,
version 1 (FreeBSD), dynamically linked, stripped
/usr/include/x86:               directory
/usr/include/ypclnt.h:          ELF 64-bit LSB shared object, x86-64,
version 1 (FreeBSD), dynamically linked, stripped
/usr/include/zconf.h:           current ar archive
/usr/include/zlib.h:            current ar archive

"since when /usr/include contains a majority of binary file ?"

# ssh
Undefined symbol "ssh_compat20" referenced from COPY relocation in
/usr/bin/ssh

# file /usr/lib/libssh.so.5
/usr/lib/libssh.so.5: symbolic link to `libopie.so.7'

"what ?"

# file /usr/lib/snmp*
/usr/lib/snmp_atm.so:        ELF 64-bit LSB shared object, x86-64,
version 1 (FreeBSD), dynamically linked, stripped
/usr/lib/snmp_atm.so.6:      symbolic link to `pam_deny.so.5'
/usr/lib/snmp_bridge.so:     ELF 64-bit LSB shared object, x86-64,
version 1 (FreeBSD), dynamically linked, stripped
/usr/lib/snmp_bridge.so.6:   symbolic link to `pam_echo.so.5'
/usr/lib/snmp_hostres.so:    ELF 64-bit LSB shared object, x86-64,
version 1 (FreeBSD), dynamically linked, stripped
/usr/lib/snmp_hostres.so.6:  symbolic link to `pam_exec.so.5'
/usr/lib/snmp_mibII.so:      ELF 64-bit LSB shared object, x86-64,
version 1 (FreeBSD), dynamically linked, stripped
/usr/lib/snmp_mibII.so.6:    symbolic link to `pam_ftpusers.so.5'
/usr/lib/snmp_netgraph.so:   ELF 64-bit LSB shared object, x86-64,
version 1 (FreeBSD), dynamically linked, stripped
/usr/lib/snmp_netgraph.so.6: symbolic link to `pam_login_access.so.5'
[...]

"why `snmp_netgraph.so.6' would be linked to
`pam_login_access.so.5' ?"

Unsurprisingly, fsck (still) detects a lot of inconsistency:

# fsck -f /
** /dev/ada0p2 (NO WRITE)

USE JOURNAL? no

** Skipping journal, falling through to full fsck
SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
124184 DUP I=31488
UNEXPECTED SOFT UPDATE INCONSISTENCY

124185 DUP I=31488
UNEXPECTED SOFT UPDATE INCONSISTENCY

124186 DUP I=31488
UNEXPECTED SOFT UPDATE INCONSISTENCY

124187 DUP I=31488
UNEXPECTED SOFT UPDATE INCONSISTENCY

124188 DUP I=31488
UNEXPECTED SOFT UPDATE INCONSISTENCY

124189 DUP I=31488
UNEXPECTED SOFT UPDATE INCONSISTENCY

[...]

EXCESSIVE DUP BLKS I=31494
CONTINUE? [yn]

I do not see this behavior when running 9.0-RELEASE on top of a
7.4-RELEASE userland (including FS). I've seen this behavior on
various CF, so a single bad card is unlikely to be the culprit.

Here are the currently mounted filesystem on the machine, as well as
mount options:

# mount
/dev/ada0p2 on / (ufs, local, journaled soft-updates)
devfs on /dev (devfs, local, multilabel)

Any hints appreciated.

Thanks,
 - Arnaud

Xin Li

2012-Mar-05 22:22 UTC

head link

Heavy fs corruption with 9.0-RELEASE

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 03/05/12 14:12, Arnaud Lacombe wrote:> Hi,
> 
> I've been running a couple of system with 9.0-RELEASE since it is
> out. All the system were installed through the standard
> installation procedure. After unclean reboot, either crash or
> power-failure, I get a huge amount of really bad filesystem
> corruption (read: "silent", fs-wide, corruptions). This happens
> with either i386 or amd64 build. Systems involved use compact flash
> as their system permanent storage medium.
[...]> I do not see this behavior when running 9.0-RELEASE on top of a 
> 7.4-RELEASE userland (including FS). I've seen this behavior on 
> various CF, so a single bad card is unlikely to be the culprit.
FWIW FS is part of kernel.  So technically you're running 9.0-RELEASE
UFS with 7.4-RELEASE userland.
> Here are the currently mounted filesystem on the machine, as well
> as mount options:
> 
> # mount /dev/ada0p2 on / (ufs, local, journaled soft-updates) devfs
> on /dev (devfs, local, multilabel)
> 
> Any hints appreciated.
My first step would be take out the journalling soft-updates to see if
things improves.  Could you test if this happens using brand new
9.0-RELEASE, but with journaled soft-updates disabled?

(Hint: tunefs -j disable /; tunefs -n enable /; reboot in single user
mode).

Note that since you say 7.4-RELEASE userland wouldn't have similar
issue, it might indicate a bug with the new journalled soft-updates's
fsck implementation.

Cheers,
- -- 
Xin LI <delphij@delphij.net>	https://www.delphij.net/
FreeBSD - The Power to Serve!		Live free or die
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (FreeBSD)

iQEcBAEBCAAGBQJPVTyCAAoJEG80Jeu8UPuzTsgIALvLdbzoNrmE/xUqDEJwGtC8
zcfGl7tEVLU6BbE3i2ww2tUcMJmOMZwrAg1gZmFyyDhVQwEEQQ2zSTa1Uy3oiNRA
ts1X31VicEXIMam1Mt5K2G3lMbFES6TvyDrNOit6C2FeUrixHZ2C1JZmA7yzGlsg
7ZGco+3clwSy2yfKTf0ExjIibkC0Cgz60BgjpQKowpKjUCD5AzB/2EqmZk1pFz6S
/NYOlF+YG/Y72V3GW1K5DTDFao5znzkulhc5Q/RUoD08o6hmdiaqiRHHmqjIP6Rt
Gb8Dfggnm8XUBs+AzgUmssCSCvHMHYt9SEel2sB5gSVKzd5rIAXwLxaeKBco1Ws=Hi9P
-----END PGP SIGNATURE-----

Julian H. Stacey

2012-Mar-06 21:44 UTC

head link

Heavy fs corruption with 9.0-RELEASE

Thierry Thomas wrote:> Le mar  6 mar 12 à 10:41:16 +0100, Julian H. Stacey <jhs@berklix.com>
>  écrivait :
> 
> >     A German magazine (CT) reported on placebo cache chips some
> >     years back, chips with pins but no silicon inside.
> 
> This one is original, at least:
>
<http://www.facebook.com/photo.php?fbid=296226443776598&set=a.133588036707107.25033.109123525820225&type=1&ref=nf>
> 
> (Sorry, it might require a FB account)
It does not require a facebook account.
I burst out laughing when I saw the pictures,
I reccomend others look,
Thanks Thierry for posting the URL :-)

Cheers,
Julian
-- 
Julian Stacey, BSD Unix Linux C Sys Eng Consultants Munich http://berklix.com
 Reply below not above, cumulative like a play script, & indent with
"> ".
 Format: Plain text. Not HTML, multipart/alternative, base64, quoted-printable.
	Mail from @yahoo dumped @berklix.  http://berklix.org/yahoo/

David Thiel

2012-Mar-08 19:34 UTC

head link

Heavy fs corruption with 9.0-RELEASE

On Mon, Mar 05, 2012 at 05:12:16PM -0500, Arnaud Lacombe
wrote:> I've been running a couple of system with 9.0-RELEASE since it is out.
> All the system were installed through the standard installation
> procedure. After unclean reboot, either crash or power-failure, I get
> a huge amount of really bad filesystem corruption (read:
"silent",
> fs-wide, corruptions). This happens with either i386 or amd64 build.
> Systems involved use compact flash as their system permanent storage
> medium.
I have had this same behavior on every SUJ system I've built, both on 
SSDs and otherwise, on i386, PPC and amd64. Remove SUJ and revert to 
plain softupdates, and I strongly suspect your problems will disappear.

freebsd stable - Mar 2012 - Heavy fs corruption with 9.0-RELEASE

Heavy fs corruption with 9.0-RELEASE

Heavy fs corruption with 9.0-RELEASE

Heavy fs corruption with 9.0-RELEASE

Heavy fs corruption with 9.0-RELEASE