CentOS - Jul 2008 - File system goes read-only once in a while

Hi,

My Centos 5.2 server (5.1 suffered the same problem as well) has a 
logical volume on a RAID 10 array (4 SATA harddisks on a Highpoint 
RR2310 controller). /etc/fstab has an entry for this array as below
/dev/raid_vg0/raid_lv0  /mnt/raid               ext3    defaults        0 0

Normally it works OK. But, file system of "this volume" once in a
while
goes "read only" mode. The RAID software reports no problem with the 
hard disks. After reboot, the system comes back in normal rw mode.

What could be the reason?. I would appreciate any help/hint.

Thank you,
Mufit

Mufit Eribol wrote on Thu, 31 Jul 2008 10:08:48 +0300:
> But, file system of "this volume" once in a while 
> goes "read only" mode.
there will be log entries about this. Do a *forced* fsck.

Kai

-- 
Kai Sch?tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

On Thu, 2008-07-31 at 10:08 +0300, Mufit Eribol wrote:
> Hi,
> 
> My Centos 5.2 server (5.1 suffered the same problem as well) has a 
> logical volume on a RAID 10 array (4 SATA harddisks on a Highpoint 
> RR2310 controller). /etc/fstab has an entry for this array as below
> /dev/raid_vg0/raid_lv0  /mnt/raid               ext3    defaults        0 0
> 
> Normally it works OK. But, file system of "this volume" once in a
while
> goes "read only" mode. The RAID software reports no problem with
the
> hard disks. After reboot, the system comes back in normal rw mode.
If it happens again, you may be able to avoid the reboot with

    mount -o remount,rw /mnt/raid

As to your "how to check production ...", easy. The trade-off (down
time, reboot, ...) makes it easy to decide to knock users down, umount
the FS, run the check, remount, tell users they can go again.
> <snip>
> Mufit
> <snip sig stuff>
HTH
-- 
Bill

I think I found the culprit albeit I still don't know how to fix.

1. During boot the screen prints the following errors
"no fstab.sys, mounting internal defaults
...
No devices found
Setting up Logical Volume Management: /var/lock: mkdir failed: No such 
file or directory"

I have a LV on RAID mounted as /mnt/raid. Then /mnt/raid/var is 
symlinked to /var. I found on the internet that some linux systems look 
for /var/lock or /var/run on / partition only. Obviously LVM can not 
create its file in /var/lock, perhaps /mnt/raid is not mounted yet 
during /var/lock mkdir operation.

2. Second important finding is that /forcefsck forces only software raid 
not the hardware one. It does the check for md0 (/temp), md1 (/boot), 
and md2 (/). It skips /dev/raid_vg0/raid_lv0 (/mnt/raid) altogether. I 
don't know how to force to check it durin reboot.

3. I changed to init level 1. Then tried to umount /mnt/raid. But all I 
received was "device is busy" prompt. "umount -l /mnt/raid"
was able
unmount /mnt/raid. Then tried to run "fsck /mnt/raid". This time I
received
"fsck.ext2: Is a directory while trying to open /mnt/raid
The superblock could not be read or does not describe a correct ext2 
filesystem. If the device is valid and it really contains an ext2 
filesystem (and not swap or ufs or something else), then the superblock 
is corrupt, and you might try running e2fsck with on alternate superblock:

e2fsck -b 8193 <device>"

I tried with one of the superblocks on /mnt/raid. This time I get
"fsck.ext3: Device or resource busy while trying to open 
/dev/raid_vg0/raid_lv0
Filesystem mounted or opened exclusively by another program?"

Sorry for the long post.

This is the point I arrived. I am stumped.

Thank your for all the support.

Mufit