CentOS - Mar 2008 - my centos switched alone its root fs to read-only

My test server was in very eavy load, running kolab's components:
postfix, cyrus-imap and openldap
A second test machine was sending, and reading emails as fast as it can.
The CPU was never idle !
I left the server at about 17H30
and at 22H I find it "uncommitted" (difficult to work when the
partition in read-only !)
However the mount command report it as RW !
The /boot is still RW for real indeed.

some more info

[root at c700 trunk]# touch /foo.bar
touch: cannot touch `/foo.bar': Read-only file system
[root at c700 trunk]# mount
/dev/md1 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/md0 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
max:/s0 on /s0 type nfs (rw,addr=192.168.*.*)
[root at c700 trunk]# cat /var/log/messages
Mar 30 04:02:13 c700 syslogd 1.4.1: restart.
Mar 30 05:47:29 c700 restorecond: Will not restore a file with more
than one hard link (/etc/resolv.conf) Invalid argument
Mar 31 07:41:37 c700 restorecond: Will not restore a file with more
than one hard link (/etc/resolv.conf) Invalid argument
[root at c700 trunk]# date
Mon Mar 31 22:14:02 CEST 2008
[root at c700 log]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md1              18316104   3921164  13449520  23% /
/dev/md0                101018     11076     84726  12% /boot
tmpfs                   257756         0    257756   0% /dev/shm
max:/s0              425343328 389031904  14705248  97% /s0

Any idea ?

The system is still up and running until I need to carry on my test further.

Regards.

-- 
Alain Spineux
aspineux gmail com
May the sources be with you

I went to the basement to check the "console" :

I got this

EXT3-fs error (device md1): htree_dirblock_to_tree: bad entry in directory #2483
009: rec_len % 4 != 0 - offset=0, inode=211961333, rec_len=5205, name_len=231
Abordinf journal on device /dev/md1
journal  commit IO error
ext3_abot called.
EXT3-fs error  (device md1): ext3_journal_start_sb: Detected aborted journal
Remounting file read-only

Grrrrrr

On Mon, Mar 31, 2008 at 10:39 PM, Alain Spineux <aspineux at gmail.com>
wrote:
> My test server was in very eavy load, running kolab's components:
>  postfix, cyrus-imap and openldap
>  A second test machine was sending, and reading emails as fast as it can.
>  The CPU was never idle !
>  I left the server at about 17H30
>  and at 22H I find it "uncommitted" (difficult to work when the
>  partition in read-only !)
>  However the mount command report it as RW !
>  The /boot is still RW for real indeed.
>
>  some more info
>
>  [root at c700 trunk]# touch /foo.bar
>  touch: cannot touch `/foo.bar': Read-only file system
>  [root at c700 trunk]# mount
>  /dev/md1 on / type ext3 (rw)
>  proc on /proc type proc (rw)
>  sysfs on /sys type sysfs (rw)
>  devpts on /dev/pts type devpts (rw,gid=5,mode=620)
>  /dev/md0 on /boot type ext3 (rw)
>  tmpfs on /dev/shm type tmpfs (rw)
>  none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
>  sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
>  max:/s0 on /s0 type nfs (rw,addr=192.168.*.*)
>  [root at c700 trunk]# cat /var/log/messages
>  Mar 30 04:02:13 c700 syslogd 1.4.1: restart.
>  Mar 30 05:47:29 c700 restorecond: Will not restore a file with more
>  than one hard link (/etc/resolv.conf) Invalid argument
>  Mar 31 07:41:37 c700 restorecond: Will not restore a file with more
>  than one hard link (/etc/resolv.conf) Invalid argument
>  [root at c700 trunk]# date
>  Mon Mar 31 22:14:02 CEST 2008
>  [root at c700 log]# df
>  Filesystem           1K-blocks      Used Available Use% Mounted on
>  /dev/md1              18316104   3921164  13449520  23% /
>  /dev/md0                101018     11076     84726  12% /boot
>  tmpfs                   257756         0    257756   0% /dev/shm
>  max:/s0              425343328 389031904  14705248  97% /s0
>
>  Any idea ?
>
>  The system is still up and running until I need to carry on my test
further.
>
>  Regards.
>
>  --
>  Alain Spineux
>  aspineux gmail com
>  May the sources be with you
>


-- 
Alain Spineux
aspineux gmail com
May the sources be with you