CentOS - Dec 2006 - /var goes read-only

This is an issue that I have been having with one of our production servers
for a couple of months, just out of no where /var goes to read only.

There aren't any errors that show up in /var/log/messages nor dmesg, I
checked to make sure that the drives on error option is set to continue and
there isn't anything as far as I can tell wrong with the disks. Also there
is plenty of disk space.

Here is the output of tune2fs -l /dev/sda3:

Filesystem volume name:   /var1
Last mounted on:          <not available>
Filesystem UUID:          6bb81ddf-b0f8-4c47-ae9f-b39f504c114e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index
filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1281696
Block count:              2560359
Reserved block count:     128017
Free blocks:              1946888
Free inodes:              1279680
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      625
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16224
Inode blocks per group:   507
Filesystem created:       Wed Jun 29 10:43:21 2005
Last mount time:          Fri Dec  1 10:30:30 2006
Last write time:          Fri Dec  1 10:30:30 2006
Mount count:              21
Maximum mount count:      -1
Last checked:             Wed Jun 29 10:43:21 2005
Check interval:           0 (<none>)
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      6f24f054-98af-4bd8-9af2-d40254126ac7
Journal backup:           inode blocks


Here is what is in /etc/fstab:

# This file is edited by fstab-sync - see 'man fstab-sync' for details
LABEL=/1                /                       ext3    defaults        1 1
none                    /dev/pts                devpts  gid=5,mode=620  0 0
none                    /dev/shm                tmpfs   defaults        0 0
none                    /proc                   proc    defaults        0 0
LABEL=/qt1              /opt                     ext3    defaults        1 2
none                    /sys                    sysfs   defaults        0 0
LABEL=/var1             /var                    ext3    defaults        1 2
LABEL=SWAP-sda5         swap                    swap    defaults        0 0
/dev/hda                /media/cdrom            auto
pamconsole,exec,noauto,managed 0 0
/dev/fd0                /media/floppy           auto
pamconsole,exec,noauto,managed 0 0

Here is the output of df -h:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              22G  2.0G   19G  10% /
none                  4.0G     0  4.0G   0% /dev/shm
/dev/sdb1             269G   48G  208G  19% /opt
/dev/sda3             9.7G  2.3G  7.0G  25% /var

Kernel version: 2.6.9-42.0.3.ELsmp

Any help would be greatly appreciated!!

-- 
Thx
Joshua Gimer
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20061201/2f1eac25/attachment-0001.html>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, Dec 01, 2006 at 02:07:49PM -0700, Joshua Gimer
wrote:
> This is an issue that I have been having with one of our production servers
for
> a couple of months, just out of no where /var goes to read only. 
> 
> There aren't any errors that show up in /var/log/messages nor dmesg, I
checked
> to make sure that the drives on error option is set to continue and there
isn't
> anything as far as I can tell wrong with the disks. Also there is plenty of
> disk space.
My suggestion is the standard one: try forcing a check on the disk.

[]s

- -- 
Rodrigo Barbosa
"Quid quid Latine dictum sit, altum viditur"
"Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFcJ/apdyWzQ5b5ckRAnD6AJ0RMYfM/0+mvrpREL3kNRbOyd/LAACgvlnh
OK0nU7ChpAqgu808cEo5luo=sTJS
-----END PGP SIGNATURE-----

On Dec 1, 2006, at 16:07, Joshua Gimer wrote:
> This is an issue that I have been having with one of our production  
> servers for a couple of months, just out of no where /var goes to  
> read only.
Maybe the disk is going bad.  I had the same (or similar) thing  
happen to me with the root partition, which in my case also contains  
the /var partition.  It turned out to be a disk going bad.  Check the  
archives of this mailing list for a thread titled "EXT3-fs error  
(devive dm-0) in start_transaction: Journal has aborted" earlier this  
year.  Run smarctl on the drive with the /var partition if it  
supports it.

Alfred