srinivas.eeda at oracle.com
2011-Jul-28 10:09 UTC
[Ocfs2-users] ocfs2, redhat 2.6.18-238.x kernel and async i/o hang
On 07/28/2011 09:20 AM, Matthew Huff wrote:> I migrated a Oracle 11gR2 database from one HP RHEL5 box to another. The only difference I can tell is that the older box was running 2.16.18-194 kernel, and the newer box is running 2.6.18.238 kernel (I've tried 2.6.18.238.el5, 238.5.1.el5 and 239.19.1.el5). On the new box, any Oracle instance hangs on startup mounting the files if async io is turned on. Direct I/O works file. The files are all on ocfs2 partitions. > > Anyone seen this issue? The databases run fine if the spfile has:We came to know about this recently and are looking into it. Till it is resolved please use the older working kernel.> *.disk_asynch_io=FALSE > *.filesystemio_options='directio' > > But hang 100% of the time (even after a few hours of waiting) if the parameters are set to: > *.disk_asynch_io=TRUE > *.filesystemio_options='setall' > > > Other than having libaio rpm loaded, I don't think there are any configuration or knobs I need to turn. > > > Here are some details: > > > [oracle at hubble lib64]$ ldd $ORACLE_HOME/bin/oracle > linux-vdso.so.1 => (0x00007fff9b7fc000) > libodm11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libodm11.so (0x00002b487959a000) > libcell11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libcell11.so (0x00002b487969c000) > libskgxp11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libskgxp11.so (0x00002b4879800000) > librt.so.1 => /lib64/librt.so.1 (0x00000032fa600000) > libnnz11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libnnz11.so (0x00002b48799ea000) > libclsra11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libclsra11.so (0x00002b4879db7000) > libdbcfg11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libdbcfg11.so (0x00002b4879ecf000) > libhasgen11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libhasgen11.so (0x00002b4879ff2000) > libskgxn2.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libskgxn2.so (0x00002b487a7c6000) > libocr11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libocr11.so (0x00002b487a8c8000) > libocrb11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libocrb11.so (0x00002b487aa6e000) > libocrutl11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libocrutl11.so (0x00002b487ac0e000) > libaio.so.1 => /usr/lib64/libaio.so.1 (0x00002b487ad19000) > libdl.so.2 => /lib64/libdl.so.2 (0x00000032f9a00000) > libm.so.6 => /lib64/libm.so.6 (0x00000032f9600000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00000032f9e00000) > libnsl.so.1 => /lib64/libnsl.so.1 (0x00000032fc600000) > libc.so.6 => /lib64/libc.so.6 (0x00000032f9200000) > /lib64/ld-linux-x86-64.so.2 (0x00000032f8e00000) > > [oracle at hubble lib64]$ rpm -qa | grep aio > libaio-0.3.106-5 > libaio-devel-0.3.106-5 > libsane-hpaio-1.6.7-6.el5_6.1 > libaio-0.3.106-5 > libaio-devel-0.3.106-5 > > [oracle at hubble dbs]$ cd /usr/lib64 > [oracle at hubble lib64]$ ls -lF *aio* > -rw-r--r-- 1 root root 15870 Dec 21 2009 libaio.a > lrwxrwxrwx 1 root root 15 Jul 23 15:21 libaio.so -> libaio.so.1.0.1* > lrwxrwxrwx 1 root root 15 Jul 22 21:59 libaio.so.1 -> libaio.so.1.0.1* > -rwxr-xr-x 1 root root 3768 Dec 21 2009 libaio.so.1.0.0* > -rwxr-xr-x 1 root root 3768 Dec 21 2009 libaio.so.1.0.1* > -rw-r--r-- 1 root root 92866 Apr 8 09:14 librtkaio.a > > [root at hubble ~]# grep ocfs2 /etc/fstab > LABEL=oracle_logfiles /u01 ocfs2 _netdev,datavolume,nointr > LABEL=oracle_datafiles /u02 ocfs2 _netdev,datavolume,nointr > > [root at hubble ~]# grep ocfs2 /etc/mtab > ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0 > /dev/cciss/c0d1 /u01 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0 > /dev/cciss/c0d2 /u02 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0 > > [oracle at hubble lib64]$ rpm -qa | grep ocfs > ocfs2console-1.4.4-1.el5 > ocfs2-2.6.18-238.19.1.el5-1.4.7-1.el5 > ocfs2-tools-1.4.4-1.el5 > > [root at hubble ~]# uname -a > Linux hubble.ox.com 2.6.18-238.19.1.el5 #1 SMP Sun Jul 10 08:43:41 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux > > > ---- > Matthew Huff | 1 Manhattanville Rd > Director of Operations | Purchase, NY 10577 > OTA Management LLC | Phone: 914-460-4039 > aim: matthewbhuff | Fax: 914-460-4139 > > > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users at oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users
Matthew Huff
2011-Jul-28 16:20 UTC
[Ocfs2-users] ocfs2, redhat 2.6.18-238.x kernel and async i/o hang
I migrated a Oracle 11gR2 database from one HP RHEL5 box to another. The only
difference I can tell is that the older box was running 2.16.18-194 kernel, and
the newer box is running 2.6.18.238 kernel (I've tried 2.6.18.238.el5,
238.5.1.el5 and 239.19.1.el5). On the new box, any Oracle instance hangs on
startup mounting the files if async io is turned on. Direct I/O works file. The
files are all on ocfs2 partitions.
Anyone seen this issue? The databases run fine if the spfile has:
*.disk_asynch_io=FALSE
*.filesystemio_options='directio'
But hang 100% of the time (even after a few hours of waiting) if the parameters
are set to:
*.disk_asynch_io=TRUE
*.filesystemio_options='setall'
Other than having libaio rpm loaded, I don't think there are any
configuration or knobs I need to turn.
Here are some details:
[oracle at hubble lib64]$ ldd $ORACLE_HOME/bin/oracle
linux-vdso.so.1 => (0x00007fff9b7fc000)
libodm11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libodm11.so
(0x00002b487959a000)
libcell11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libcell11.so
(0x00002b487969c000)
libskgxp11.so =>
/opt/oracle/product/11.2.0/dbhome_1/lib/libskgxp11.so (0x00002b4879800000)
librt.so.1 => /lib64/librt.so.1 (0x00000032fa600000)
libnnz11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libnnz11.so
(0x00002b48799ea000)
libclsra11.so =>
/opt/oracle/product/11.2.0/dbhome_1/lib/libclsra11.so (0x00002b4879db7000)
libdbcfg11.so =>
/opt/oracle/product/11.2.0/dbhome_1/lib/libdbcfg11.so (0x00002b4879ecf000)
libhasgen11.so =>
/opt/oracle/product/11.2.0/dbhome_1/lib/libhasgen11.so (0x00002b4879ff2000)
libskgxn2.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libskgxn2.so
(0x00002b487a7c6000)
libocr11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libocr11.so
(0x00002b487a8c8000)
libocrb11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libocrb11.so
(0x00002b487aa6e000)
libocrutl11.so =>
/opt/oracle/product/11.2.0/dbhome_1/lib/libocrutl11.so (0x00002b487ac0e000)
libaio.so.1 => /usr/lib64/libaio.so.1 (0x00002b487ad19000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000032f9a00000)
libm.so.6 => /lib64/libm.so.6 (0x00000032f9600000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000032f9e00000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00000032fc600000)
libc.so.6 => /lib64/libc.so.6 (0x00000032f9200000)
/lib64/ld-linux-x86-64.so.2 (0x00000032f8e00000)
[oracle at hubble lib64]$ rpm -qa | grep aio
libaio-0.3.106-5
libaio-devel-0.3.106-5
libsane-hpaio-1.6.7-6.el5_6.1
libaio-0.3.106-5
libaio-devel-0.3.106-5
[oracle at hubble dbs]$ cd /usr/lib64
[oracle at hubble lib64]$ ls -lF *aio*
-rw-r--r-- 1 root root 15870 Dec 21 2009 libaio.a
lrwxrwxrwx 1 root root 15 Jul 23 15:21 libaio.so -> libaio.so.1.0.1*
lrwxrwxrwx 1 root root 15 Jul 22 21:59 libaio.so.1 -> libaio.so.1.0.1*
-rwxr-xr-x 1 root root 3768 Dec 21 2009 libaio.so.1.0.0*
-rwxr-xr-x 1 root root 3768 Dec 21 2009 libaio.so.1.0.1*
-rw-r--r-- 1 root root 92866 Apr 8 09:14 librtkaio.a
[root at hubble ~]# grep ocfs2 /etc/fstab
LABEL=oracle_logfiles /u01 ocfs2
_netdev,datavolume,nointr
LABEL=oracle_datafiles /u02 ocfs2
_netdev,datavolume,nointr
[root at hubble ~]# grep ocfs2 /etc/mtab
ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0
/dev/cciss/c0d1 /u01 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0
/dev/cciss/c0d2 /u02 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0
[oracle at hubble lib64]$ rpm -qa | grep ocfs
ocfs2console-1.4.4-1.el5
ocfs2-2.6.18-238.19.1.el5-1.4.7-1.el5
ocfs2-tools-1.4.4-1.el5
[root at hubble ~]# uname -a
Linux hubble.ox.com 2.6.18-238.19.1.el5 #1 SMP Sun Jul 10 08:43:41 EDT 2011
x86_64 x86_64 x86_64 GNU/Linux
----
Matthew Huff? | 1 Manhattanville Rd
Director of Operations???| Purchase, NY 10577
OTA Management LLC | Phone: 914-460-4039
aim: matthewbhuff? | Fax:?? 914-460-4139