Hi
In our office environment few servers mostly database servers and yesterday it
happened
for one application server(first time) the partion is getting "read
only".
I was checking the archives, found may be similar kind of issues in the
2007-July archives.
But how it has been solved if someone describes me that will be really helpful.
In our case, just at the problem started found the line in log file as follows:
EXT3-fs error (device dm-12): edxt3_find_entry: reading directory #2015496
offset 2
Then one blank line
Then the line is
Aborting journal on device dm-12.
ext3_abort called
Ext3-fs error (device dm-12): ext3_journal_start_sb: Detected aborted
journal
Remounting filesysem read-only
Then the continuous line as follows:
EXT3-fs error (device dm-12) in start_transaction: Journal has aborted
The above message is continuous until we remount the filesystem and partion
becomes
'read-write'.
We could not figure it out what is the root cause of the system.
We are using individual EMC luns and are configured with LVM volume groups and
then mounted on logical
volumes.
Here i am giving the server description:
____________________________________________________________
[root at server ~]# lsmod |grep -i qla
qla2300 130304 0
qla2xxx_conf 305924 0
qla2xxx 307448 21 qla2300
scsi_mod 117709 5 sg,emcp,qla2xxx,cciss,sd_mod
____________________________________________________________
[root at server ~]# cat /etc/modprobe.conf
alias eth0 tg3
alias eth1 tg3
alias eth2 e1000
alias eth3 e1000
alias eth4 e1000
alias eth5 e1000
alias bond0 bonding
alias scsi_hostadapter cciss
options bond0 max_bonds=2 miimon=100 mode=1
alias scsi_hostadapter1 qla2xxx
alias scsi_hostadapter2 qla2xxx_conf
#alias scsi_hostadapter3 qla6312
options qla2xxx ql2xmaxqdepth=16 qlport_down_retry=64
ql2xloginretrycount=30 ql2xfailover=0 ql2xlbType=0
install qla2xxx /sbin/modprobe qla2xxx_conf; /sbin/modprobe
--ignore-install qla2xxx
remove qla2xxx /sbin/modprobe -r --first-time --ignore-remove qla2xxx &&
{
/sbin/modprobe -r --ignore-remove qla2xxx_conf; }
###BEGINPP
include /etc/modprobe.conf.pp
###ENDPP
###BEGINPP
include /etc/modprobe.conf.pp
###ENDPP
###BEGINPP
include /etc/modprobe.conf.pp
###ENDPP
________________________________________________
[root at server ~]# rpm -qa |grep -i EMC
EMCpower.LINUX-4.5.1-022
________________________________________________
[root at server ~]# rpm -qa|grep -i scli
scli-1.06.16-57
________________________________________________
[root at server ~]# rpm -qa|grep -i nav
naviagentcli-6.19.1.3.0-1
________________________________________________
product: QLA2312 Fibre Channel Adapter
________________________________________________
[root at server ~]# rpm -qa|grep -i lvm
lvm2-2.02.06-6.0.RHEL4
system-config-lvm-1.0.19-1.0
________________________________________________
If I missed any info, pl. let me know.
It would be really appreciated if I get some hints to solve the issues
Thanks in advance
-swapana
____________________________________________________________________________________
Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated
for today's economy) at Yahoo! Games.
http://get.games.yahoo.com/proddesc?gamekey=monopolyherenow
It looks to me like your system is setup sot that filesystems revert to read-only on errors. The system has detected an error (possibly a read error), and has now reverted the system to read-only. Check your system logs for errors related to the partition that you're having problems with (this includes the devices underlying the LVM). Then, (presuming that it's not a hardware problem) I'd suggest: 1) a backup of whatever data you can currently retreive from the syatem, the, 2) go to single-user and manually run fsck on the suspect filesystem. Check another relatively recent thread about problems with RAID cards on risers (1U and 2U systems). On 9/22/07, Swapana Ghosh <swapana_ghosh at yahoo.com> wrote:> Hi > > In our office environment few servers mostly database servers and yesterday it > happened > for one application server(first time) the partion is getting "read only". > > I was checking the archives, found may be similar kind of issues in the > 2007-July archives. > But how it has been solved if someone describes me that will be really helpful. > > In our case, just at the problem started found the line in log file as follows: > > EXT3-fs error (device dm-12): edxt3_find_entry: reading directory #2015496 > offset 2 >..... -- Stephen Samuel http://www.bcgreen.com 778-861-7641
what are you using for managing the multipathing to your SAN? /Balu On 9/23/07, Swapana Ghosh <swapana_ghosh at yahoo.com> wrote:> > Hi > > In our office environment few servers mostly database servers and > yesterday it > happened > for one application server(first time) the partion is getting "read only". > > I was checking the archives, found may be similar kind of issues in the > 2007-July archives. > But how it has been solved if someone describes me that will be really > helpful. > > In our case, just at the problem started found the line in log file as > follows: > > EXT3-fs error (device dm-12): edxt3_find_entry: reading directory > #2015496 > offset 2 > > Then one blank line > Then the line is > > Aborting journal on device dm-12. > ext3_abort called > > Ext3-fs error (device dm-12): ext3_journal_start_sb: Detected aborted > journal > Remounting filesysem read-only > > Then the continuous line as follows: > > > EXT3-fs error (device dm-12) in start_transaction: Journal has aborted > > > > The above message is continuous until we remount the filesystem and > partion > becomes > 'read-write'. > > We could not figure it out what is the root cause of the system. > > We are using individual EMC luns and are configured with LVM volume groups > and > then mounted on logical > volumes. > > Here i am giving the server description: > > ____________________________________________________________ > > [root at server ~]# lsmod |grep -i qla > qla2300 130304 0 > qla2xxx_conf 305924 0 > qla2xxx 307448 21 qla2300 > scsi_mod 117709 5 sg,emcp,qla2xxx,cciss,sd_mod > > ____________________________________________________________ > [root at server ~]# cat /etc/modprobe.conf > alias eth0 tg3 > alias eth1 tg3 > alias eth2 e1000 > alias eth3 e1000 > alias eth4 e1000 > alias eth5 e1000 > alias bond0 bonding > alias scsi_hostadapter cciss > options bond0 max_bonds=2 miimon=100 mode=1 > alias scsi_hostadapter1 qla2xxx > alias scsi_hostadapter2 qla2xxx_conf > #alias scsi_hostadapter3 qla6312 > options qla2xxx ql2xmaxqdepth=16 qlport_down_retry=64 > ql2xloginretrycount=30 ql2xfailover=0 ql2xlbType=0 > install qla2xxx /sbin/modprobe qla2xxx_conf; /sbin/modprobe > --ignore-install qla2xxx > remove qla2xxx /sbin/modprobe -r --first-time --ignore-remove qla2xxx && { > /sbin/modprobe -r --ignore-remove qla2xxx_conf; } > ###BEGINPP > include /etc/modprobe.conf.pp > ###ENDPP > ###BEGINPP > include /etc/modprobe.conf.pp > ###ENDPP > ###BEGINPP > include /etc/modprobe.conf.pp > ###ENDPP > > ________________________________________________ > [root at server ~]# rpm -qa |grep -i EMC > EMCpower.LINUX-4.5.1-022 > > ________________________________________________ > [root at server ~]# rpm -qa|grep -i scli > scli-1.06.16-57 > > ________________________________________________ > [root at server ~]# rpm -qa|grep -i nav > naviagentcli-6.19.1.3.0-1 > > ________________________________________________ > product: QLA2312 Fibre Channel Adapter > > ________________________________________________ > [root at server ~]# rpm -qa|grep -i lvm > lvm2-2.02.06-6.0.RHEL4 > system-config-lvm-1.0.19-1.0 > > ________________________________________________ > > If I missed any info, pl. let me know. > > It would be really appreciated if I get some hints to solve the issues > > Thanks in advance > -swapana > > > > > > ____________________________________________________________________________________ > Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated > for today's economy) at Yahoo! Games. > http://get.games.yahoo.com/proddesc?gamekey=monopolyherenow > > _______________________________________________ > Ext3-users mailing list > Ext3-users at redhat.com > https://www.redhat.com/mailman/listinfo/ext3-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/ext3-users/attachments/20070923/55c433db/attachment.htm>
Hi, It seems like what it happened to me. I did this to solve this issue: Mark the filesystem as it does not have a journal (take it to ext2) tune2fs -O ^has_journal /dev/cciss/c0d0p2 fsck it to delete the journal: e2fsck /dev/cciss/c0d0p2 Create the journal (take it back to ext3) tune2fs -j /dev/cciss/c0d0p2 and finaly, remount it. In my case it was with a local disk, but with your SAN disk should be the same. Jordi Swapana Ghosh wrote:> Hi > > In our office environment few servers mostly database servers and yesterday it > happened > for one application server(first time) the partion is getting "read only". > > I was checking the archives, found may be similar kind of issues in the > 2007-July archives. > But how it has been solved if someone describes me that will be really helpful. > > In our case, just at the problem started found the line in log file as follows: > > EXT3-fs error (device dm-12): edxt3_find_entry: reading directory #2015496 > offset 2 > > Then one blank line > Then the line is > > Aborting journal on device dm-12. > ext3_abort called > > Ext3-fs error (device dm-12): ext3_journal_start_sb: Detected aborted > journal > Remounting filesysem read-only > > Then the continuous line as follows: > > > EXT3-fs error (device dm-12) in start_transaction: Journal has aborted > > > > The above message is continuous until we remount the filesystem and partion > becomes > 'read-write'. > > We could not figure it out what is the root cause of the system. > > We are using individual EMC luns and are configured with LVM volume groups and > then mounted on logical > volumes. > > Here i am giving the server description: > > ____________________________________________________________ > > [root at server ~]# lsmod |grep -i qla > qla2300 130304 0 > qla2xxx_conf 305924 0 > qla2xxx 307448 21 qla2300 > scsi_mod 117709 5 sg,emcp,qla2xxx,cciss,sd_mod > > ____________________________________________________________ > [root at server ~]# cat /etc/modprobe.conf > alias eth0 tg3 > alias eth1 tg3 > alias eth2 e1000 > alias eth3 e1000 > alias eth4 e1000 > alias eth5 e1000 > alias bond0 bonding > alias scsi_hostadapter cciss > options bond0 max_bonds=2 miimon=100 mode=1 > alias scsi_hostadapter1 qla2xxx > alias scsi_hostadapter2 qla2xxx_conf > #alias scsi_hostadapter3 qla6312 > options qla2xxx ql2xmaxqdepth=16 qlport_down_retry=64 > ql2xloginretrycount=30 ql2xfailover=0 ql2xlbType=0 > install qla2xxx /sbin/modprobe qla2xxx_conf; /sbin/modprobe > --ignore-install qla2xxx > remove qla2xxx /sbin/modprobe -r --first-time --ignore-remove qla2xxx && { > /sbin/modprobe -r --ignore-remove qla2xxx_conf; } > ###BEGINPP > include /etc/modprobe.conf.pp > ###ENDPP > ###BEGINPP > include /etc/modprobe.conf.pp > ###ENDPP > ###BEGINPP > include /etc/modprobe.conf.pp > ###ENDPP > > ________________________________________________ > [root at server ~]# rpm -qa |grep -i EMC > EMCpower.LINUX-4.5.1-022 > > ________________________________________________ > [root at server ~]# rpm -qa|grep -i scli > scli-1.06.16-57 > > ________________________________________________ > [root at server ~]# rpm -qa|grep -i nav > naviagentcli-6.19.1.3.0-1 > > ________________________________________________ > product: QLA2312 Fibre Channel Adapter > > ________________________________________________ > [root at server ~]# rpm -qa|grep -i lvm > lvm2-2.02.06-6.0.RHEL4 > system-config-lvm-1.0.19-1.0 > > ________________________________________________ > > If I missed any info, pl. let me know. > > It would be really appreciated if I get some hints to solve the issues > > Thanks in advance > -swapana > > > > > ____________________________________________________________________________________ > Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games. > http://get.games.yahoo.com/proddesc?gamekey=monopolyherenow > > _______________________________________________ > Ext3-users mailing list > Ext3-users at redhat.com > https://www.redhat.com/mailman/listinfo/ext3-users > > >-- ...................................................................... __ / / Jordi Prats C E / S / C A Dept. de Sistemes /_/ Centre de Supercomputaci? de Catalunya Gran Capit?, 2-4 (Edifici Nexus) ? 08034 Barcelona T. 93 205 6464 ? F. 93 205 6979 ? jprats at cesca.es ......................................................................