Ext3 users - Apr 2003 - recovery notice on every reboot, errors writing to fs

excuse the newbie nature of the post, we recently inherited a linux server which
has ext3 on it and we have encountered problems before being afforded the
opportunity to properly educate ourselves...


We had a situation where the server froze and required a hard restart.  On
reboot we get the following notices which were captured in the dmesg log

EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 232k freed
Adding Swap: 979956k swap-space (priority -1)
EXT3 FS 2.4-0.9.17, 10 Jan 2002 on sd(8,1), internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS 2.4-0.9.17, 10 Jan 2002 on sd(8,5), internal journal
EXT3-fs: mounted filesystem with ordered data mode.

and then it continues on to boot into the system.

Not all libraries function after this though, there are problems chmod'ing
files, we get numerous "Bus error" "Segmentation fault"
etc...  some errors
indicating that libraries may not being linked correctly or something.  For
example, a simply "man fsck" produces  "Inconsistency detected by
ld.so:
dynamic-link.h: 62: elf_get_dynamic_info: Assertion `! "bad dynamic
tag"'
failed!"

None of these errors existed before the forced restart.

Additionally, while the external journal file was easily found on the other
server, we have been unable to find one on this server (then again find may not
be reliable).

My experience with this type of problem usually results in dropping to single
user mode, running fsck, and getting back into business.  Would appreciate some
helpful guidance on some steps to rectify this problem, or even if it is related
to ext3 fs.

thanks

Dave

Dave,
> 
> Not all libraries function after this though, there are problems
chmod'ing
> files, we get numerous "Bus error" "Segmentation fault"
etc...  some errors
> indicating that libraries may not being linked correctly or something.  For
> example, a simply "man fsck" produces  "Inconsistency
detected by ld.so:
> dynamic-link.h: 62: elf_get_dynamic_info: Assertion `! "bad dynamic
tag"'
> failed!"
> 
> None of these errors existed before the forced restart.
> 
	You will need to umount the affected file systems and then run fsck on
the affected file systems. If that is impossible, because the file
systems could be containing the fsck binaries, then you should look at
the bootable business card Linux distribution and use that to perform
the fsck

	I had a similar issue with one of my workstations, it was hardlocked
due to the "Tux In Space" Screensaver, for some reason, when only that
screensaver runs, it will occasionally hardlock the system.

	When I rebooted, there were a few irregularities that required
umounting fsck'ing the partitions and then BAM! I was back in business.


	Regards,
	Robert Adkins
	IT Manager/Buyer
	Impel Industries, Inc.

Hi,

On Mon, 2003-04-21 at 00:37, Dave [Hawk-Systems] wrote:
> We had a situation where the server froze and required a hard restart.  On
> reboot we get the following notices which were captured in the dmesg log
> 
> EXT3-fs: INFO: recovery required on readonly filesystem.
> EXT3-fs: write access will be enabled during recovery.
> kjournald starting.  Commit interval 5 seconds
> EXT3-fs: recovery complete.
...

That's all completely normal.
> Not all libraries function after this though, there are problems
chmod'ing
> files, we get numerous "Bus error" "Segmentation fault"
etc...  some errors
> indicating that libraries may not being linked correctly or something.  For
> example, a simply "man fsck" produces  "Inconsistency
detected by ld.so:
> dynamic-link.h: 62: elf_get_dynamic_info: Assertion `! "bad dynamic
tag"'
> failed!"
Those sorts of errors are usually associated with bad ram. 
www.memtest86.com is your first port of call.

The other possibility is disk corruption.  If you're using an rpm-based
system, "rpm -V" can verify package contents, so you can test the
integrity of your libraries that way.  It would also be useful to force
a full fsck run to see if any problems are discovered.
> Additionally, while the external journal file was easily found on the other
> server, we have been unable to find one on this server (then again find may
not
> be reliable).
That's OK, current e2fsprogs tools create hidden journal files that
don't show up in the filesystem tree.

Cheers,
 Stephen