thr3ads.net - Ocfs2 users - [Ocfs2-users] Encountered disk I/O error 19502 [Apr 2009]

If this information is useful, please help other people find it:
Share via:

Diane Petersen

2009-Apr-06 14:05 UTC

[Ocfs2-users] Encountered disk I/O error 19502

Hi,

We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5 with Linux kernel
2.6.18-92.1.17.el5 64-bit. Lately we've been seeing errors on both nodes
almost ever other day. The system administrator has checked the SAN array and
said there are no issues being reported.

Another part of the problem, it appears the instances alter the service_names
parameter not allowing new connections to the node with the reported error, but
also terminate sessions already connected using the RAC service. The errors all
start with - Encountered disk I/O error 19502 - and contain the following:


ARC2: Encountered disk I/O error 19502
 (ifxdb2)
Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 155649
Additional information: -1
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)
Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 155649
Additional information: -1
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)
ARC2: I/O error 19502 archiving log 10 to
'/u03/arch/2_1917_656008464.dbf'
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance ifxdb2 - Archival Error
ORA-16038: log 10 sequence# 1917 cannot be archived
ORA-19502: write error on file "", block number  (block size=)
ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-16038: log 10 sequence# 1917 cannot be archived
ORA-19502: write error on file "", block number  (block size=)
ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Sun Apr 05 15:05:16 2009
ALTER SYSTEM SET service_names='ifxdb.gointranet.com' SCOPE=MEMORY
SID='ifxdb2';
Immediate Kill Session#: 185, Serial#: 40263
Immediate Kill Session: sess: 0x1274fabc8  OS pid: 13270
Immediate Kill Session#: 187, Serial#: 41391
Immediate Kill Session: sess: 0x1274fd710  OS pid: 27697
Immediate Kill Session#: 191, Serial#: 40464
Immediate Kill Session: sess: 0x127502da0  OS pid: 30697
Immediate Kill Session#: 195, Serial#: 57362
Immediate Kill Session: sess: 0x127508430  OS pid: 27967
Immediate Kill Session#: 196, Serial#: 2028
Immediate Kill Session: sess: 0x124544048  OS pid: 22900
Immediate Kill Session#: 205, Serial#: 17412
Immediate Kill Session: sess: 0x127515c98  OS pid: 20110
Immediate Kill Session#: 206, Serial#: 14805
Immediate Kill Session: sess: 0x1245518b0  OS pid: 10464
Immediate Kill Session#: 207, Serial#: 52184
Immediate Kill Session: sess: 0x1275187e0  OS pid: 19787
Immediate Kill Session#: 208, Serial#: 62825
Immediate Kill Session: sess: 0x1245543f8  OS pid: 13578
Immediate Kill Session#: 213, Serial#: 36907
Immediate Kill Session: sess: 0x1275209b8  OS pid: 31397
Immediate Kill Session#: 214, Serial#: 49032
Immediate Kill Session: sess: 0x12455c5d0  OS pid: 2427
Immediate Kill Session#: 215, Serial#: 2711
Immediate Kill Session: sess: 0x127523500  OS pid: 15942
Immediate Kill Session#: 216, Serial#: 30060
Immediate Kill Session: sess: 0x12455f118  OS pid: 1217
Immediate Kill Session#: 219, Serial#: 35932
Immediate Kill Session: sess: 0x127528b90  OS pid: 27883
Immediate Kill Session#: 222, Serial#: 26007
Immediate Kill Session: sess: 0x1245672f0  OS pid: 1036
Immediate Kill Session#: 223, Serial#: 42462
Immediate Kill Session: sess: 0x12752e220  OS pid: 13726
Immediate Kill Session#: 224, Serial#: 33323
Immediate Kill Session: sess: 0x124569e38  OS pid: 29928
Immediate Kill Session#: 225, Serial#: 49752
Immediate Kill Session: sess: 0x127530d68  OS pid: 20147
Immediate Kill Session#: 227, Serial#: 34834
Immediate Kill Session: sess: 0x1275338b0  OS pid: 9365
Immediate Kill Session#: 230, Serial#: 19879
Immediate Kill Session: sess: 0x124572010  OS pid: 15791
Immediate Kill Session#: 231, Serial#: 16554
Immediate Kill Session: sess: 0x127538f40  OS pid: 15490
Immediate Kill Session#: 233, Serial#: 25251
Immediate Kill Session: sess: 0x12753ba88  OS pid: 6972
Immediate Kill Session#: 236, Serial#: 36970
Immediate Kill Session: sess: 0x12457a1e8  OS pid: 12354
Immediate Kill Session#: 244, Serial#: 37284
Immediate Kill Session: sess: 0x124584f08  OS pid: 19290
Immediate Kill Session#: 245, Serial#: 55792
Immediate Kill Session: sess: 0x12754be38  OS pid: 19288
Immediate Kill Session#: 246, Serial#: 25115
Immediate Kill Session: sess: 0x124587a50  OS pid: 3111
Immediate Kill Session#: 247, Serial#: 6416
Immediate Kill Session: sess: 0x12754e980  OS pid: 19471
Immediate Kill Session#: 251, Serial#: 19899
Immediate Kill Session: sess: 0x127554010  OS pid: 21486
Immediate Kill Session#: 252, Serial#: 34731
Immediate Kill Session: sess: 0x12458fc28  OS pid: 30540
Immediate Kill Session#: 253, Serial#: 32638
Immediate Kill Session: sess: 0x127556b58  OS pid: 5493
Immediate Kill Session#: 259, Serial#: 29155
Immediate Kill Session: sess: 0x12755ed30  OS pid: 29463
Immediate Kill Session#: 261, Serial#: 14481
Immediate Kill Session: sess: 0x127561878  OS pid: 31054
Immediate Kill Session#: 265, Serial#: 37618
Immediate Kill Session: sess: 0x127566f08  OS pid: 868
Immediate Kill Session#: 267, Serial#: 42580
Immediate Kill Session: sess: 0x127569a50  OS pid: 16839
Immediate Kill Session#: 268, Serial#: 50893
Immediate Kill Session: sess: 0x1245a5668  OS pid: 27778
Immediate Kill Session#: 274, Serial#: 34459
Immediate Kill Session: sess: 0x1245ad840  OS pid: 9808
Immediate Kill Session#: 278, Serial#: 59445
Immediate Kill Session: sess: 0x1245b2ed0  OS pid: 28434
Immediate Kill Session#: 281, Serial#: 50119
Immediate Kill Session: sess: 0x12757c948  OS pid: 12606
Immediate Kill Session#: 282, Serial#: 30208
Immediate Kill Session: sess: 0x1245b8560  OS pid: 17944
Immediate Kill Session#: 285, Serial#: 53580
Immediate Kill Session: sess: 0x127581fd8  OS pid: 16670
Immediate Kill Session#: 286, Serial#: 5929
Immediate Kill Session: sess: 0x1245bdbf0  OS pid: 20149
Immediate Kill Session#: 289, Serial#: 53725
Immediate Kill Session: sess: 0x127587668  OS pid: 14697
Immediate Kill Session#: 290, Serial#: 30378
Immediate Kill Session: sess: 0x1245c3280  OS pid: 19757
Immediate Kill Session#: 293, Serial#: 53710
Immediate Kill Session: sess: 0x12758ccf8  OS pid: 11096
Immediate Kill Session#: 296, Serial#: 34022
Immediate Kill Session: sess: 0x1245cb458  OS pid: 10881
Immediate Kill Session#: 299, Serial#: 53951
Immediate Kill Session: sess: 0x127594ed0  OS pid: 1453
Immediate Kill Session#: 304, Serial#: 15149
Immediate Kill Session: sess: 0x1245d6178  OS pid: 22008
Immediate Kill Session#: 308, Serial#: 34245
Immediate Kill Session: sess: 0x1245db808  OS pid: 19156
Immediate Kill Session#: 315, Serial#: 15240
Immediate Kill Session: sess: 0x1275aa910  OS pid: 32148
Immediate Kill Session#: 317, Serial#: 41792
Immediate Kill Session: sess: 0x1275ad458  OS pid: 15660
Immediate Kill Session#: 318, Serial#: 7839
Immediate Kill Session: sess: 0x1245e9070  OS pid: 24999
Immediate Kill Session#: 321, Serial#: 4422
Immediate Kill Session: sess: 0x1275b2ae8  OS pid: 16028
Immediate Kill Session#: 324, Serial#: 6833
Immediate Kill Session: sess: 0x1245f1248  OS pid: 21909
Immediate Kill Session#: 332, Serial#: 18018
Immediate Kill Session: sess: 0x1245fbf68  OS pid: 15819
Immediate Kill Session#: 333, Serial#: 37534
Immediate Kill Session: sess: 0x1275c2e98  OS pid: 16433
Immediate Kill Session#: 334, Serial#: 50463
Immediate Kill Session: sess: 0x1245feab0  OS pid: 5660
Immediate Kill Session#: 335, Serial#: 11994
Immediate Kill Session: sess: 0x1275c59e0  OS pid: 29575
Immediate Kill Session#: 336, Serial#: 26542
Immediate Kill Session: sess: 0x1246015f8  OS pid: 31868
Immediate Kill Session#: 345, Serial#: 46583
Immediate Kill Session: sess: 0x1275d3248  OS pid: 25399
Sun Apr 05 15:05:43 2009
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance ifxdb2 - Archival Error
ORA-16014: log 10 sequence# 1917 not archived, no available destinations
ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-16014: log 10 sequence# 1917 not archived, no available destinations
ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Sun Apr 05 15:10:52 2009
kcrrdmx: Successful archiving of previously failed ORL
Archiver process freed from errors. No longer stopped
Sun Apr 05 15:10:53 2009
ALTER SYSTEM SET service_names='ifxdb.gointranet.com','ifxserv'
SCOPE=MEMORY SID='ifxdb2';These incidences are all occurring during
archiving (redo logs and database files are using ASM, archiving and backups are
on OCFS2). Even though it usually only lasts a few minutes, it's very
noticeable to the customers because of all the sessions that are terminated.

What should we be looking at to resolve this problem?
Please let me know if you have any questions.

Thanks,
Diane Petersen
ServerCare, Inc.


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090406/5a872bff/attachment.html

Srinivas Eeda

2009-Apr-06 15:37 UTC

head link

[Ocfs2-users] Encountered disk I/O error 19502

do you notice any errors in messages files during the same time as the 
error reported in alert.log?.

please forward the output of  echo "stat 
/arch/2_1917_656008464.dbf"|debugfs.ocfs2 <device>

Can you please file a tar with Oracle and forward me the tar#.

thanks,
--Srini

Diane Petersen wrote:> Hi,
>
> We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5 with Linux 
> kernel 2.6.18-92.1.17.el5 64-bit. Lately we've been seeing errors on 
> both nodes almost ever other day. The system administrator has checked 
> the SAN array and said there are no issues being reported.
>
> Another part of the problem, it appears the instances alter the 
> service_names parameter not allowing new connections to the node with 
> the reported error, but also terminate sessions already connected 
> using the RAC service. The errors all start with - Encountered disk 
> I/O error 19502 - and contain the following:
>
> ARC2: Encountered disk I/O error 19502
>  (ifxdb2)
> Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
> ORA-19502: write error on file
>  "/u03/arch/2_1917_656008464.dbf", block number 155649 (block
size=512)
> ORA-27072: File I/O error
> Linux-x86_64 Error: 5: Input/output error
> Additional information: 4
> Additional information: 155649
> Additional information: -1
> ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf",
block number 155649 (block size=512)
> Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
> ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf",
block number 155649 (block size=512)
> ORA-27072: File I/O error
> Linux-x86_64 Error: 5: Input/output error
> Additional information: 4
> Additional information: 155649
> Additional information: -1
> ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf",
block number 155649 (block size=512)
> ARC2: I/O error 19502 archiving log 10 to
'/u03/arch/2_1917_656008464.dbf'
> ARCH: Archival stopped, error occurred. Will continue retrying
> ORACLE
>  Instance ifxdb2 - Archival Error
> ORA-16038: log 10 sequence# 1917 cannot be archived
> ORA-19502: write error on file "", block number  (block size=)
> ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
> Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
> ORA-16038: log 10 sequence# 1917 cannot be archived
> ORA-19502: write error on file "", block number  (block size=)
> ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
> Sun Apr 05 15:05:16 2009
> ALTER SYSTEM SET service_names='ifxdb.gointranet.com' SCOPE=MEMORY
SID='ifxdb2';
> Immediate Kill Session#: 185, Serial#: 40263
> Immediate Kill Session: sess: 0x1274fabc8  OS pid: 13270
> Immediate Kill Session#: 187, Serial#: 41391
> Immediate Kill Session: sess: 0x1274fd710  OS pid: 27697
> Immediate Kill Session#: 191, Serial#: 40464
> Immediate Kill Session: sess: 0x127502da0  OS pid:
>  30697
> Immediate Kill Session#: 195, Serial#: 57362
> Immediate Kill Session: sess: 0x127508430  OS pid: 27967
> Immediate Kill Session#: 196, Serial#: 2028
> Immediate Kill Session: sess: 0x124544048  OS pid: 22900
> Immediate Kill Session#: 205, Serial#: 17412
> Immediate Kill Session: sess: 0x127515c98  OS pid: 20110
> Immediate Kill Session#: 206, Serial#: 14805
> Immediate Kill Session: sess: 0x1245518b0  OS pid: 10464
> Immediate Kill Session#: 207, Serial#: 52184
> Immediate Kill Session: sess: 0x1275187e0  OS pid: 19787
> Immediate Kill Session#: 208, Serial#: 62825
> Immediate Kill Session: sess: 0x1245543f8  OS pid: 13578
> Immediate Kill Session#: 213, Serial#: 36907
> Immediate Kill Session: sess: 0x1275209b8  OS pid: 31397
> Immediate Kill Session#: 214, Serial#: 49032
> Immediate Kill Session: sess: 0x12455c5d0  OS pid: 2427
> Immediate Kill Session#: 215, Serial#: 2711
> Immediate Kill Session: sess: 0x127523500  OS
>  pid: 15942
> Immediate Kill Session#: 216, Serial#: 30060
> Immediate Kill Session: sess: 0x12455f118  OS pid: 1217
> Immediate Kill Session#: 219, Serial#: 35932
> Immediate Kill Session: sess: 0x127528b90  OS pid: 27883
> Immediate Kill Session#: 222, Serial#: 26007
> Immediate Kill Session: sess: 0x1245672f0  OS pid: 1036
> Immediate Kill Session#: 223, Serial#: 42462
> Immediate Kill Session: sess: 0x12752e220  OS pid: 13726
> Immediate Kill Session#: 224, Serial#: 33323
> Immediate Kill Session: sess: 0x124569e38  OS pid: 29928
> Immediate Kill Session#: 225, Serial#: 49752
> Immediate Kill Session: sess: 0x127530d68  OS pid: 20147
> Immediate Kill Session#: 227, Serial#: 34834
> Immediate Kill Session: sess: 0x1275338b0  OS pid: 9365
> Immediate Kill Session#: 230, Serial#: 19879
> Immediate Kill Session: sess: 0x124572010  OS pid: 15791
> Immediate Kill Session#: 231, Serial#: 16554
> Immediate Kill Session: sess: 0x127538f40 
>  OS pid: 15490
> Immediate Kill Session#: 233, Serial#: 25251
> Immediate Kill Session: sess: 0x12753ba88  OS pid: 6972
> Immediate Kill Session#: 236, Serial#: 36970
> Immediate Kill Session: sess: 0x12457a1e8  OS pid: 12354
> Immediate Kill Session#: 244, Serial#: 37284
> Immediate Kill Session: sess: 0x124584f08  OS pid: 19290
> Immediate Kill Session#: 245, Serial#: 55792
> Immediate Kill Session: sess: 0x12754be38  OS pid: 19288
> Immediate Kill Session#: 246, Serial#: 25115
> Immediate Kill Session: sess: 0x124587a50  OS pid: 3111
> Immediate Kill Session#: 247, Serial#: 6416
> Immediate Kill Session: sess: 0x12754e980  OS pid: 19471
> Immediate Kill Session#: 251, Serial#: 19899
> Immediate Kill Session: sess: 0x127554010  OS pid: 21486
> Immediate Kill Session#: 252, Serial#: 34731
> Immediate Kill Session: sess: 0x12458fc28  OS pid: 30540
> Immediate Kill Session#: 253, Serial#: 32638
> Immediate Kill Session: sess: 0x127556b58
>   OS pid: 5493
> Immediate Kill Session#: 259, Serial#: 29155
> Immediate Kill Session: sess: 0x12755ed30  OS pid: 29463
> Immediate Kill Session#: 261, Serial#: 14481
> Immediate Kill Session: sess: 0x127561878  OS pid: 31054
> Immediate Kill Session#: 265, Serial#: 37618
> Immediate Kill Session: sess: 0x127566f08  OS pid: 868
> Immediate Kill Session#: 267, Serial#: 42580
> Immediate Kill Session: sess: 0x127569a50  OS pid: 16839
> Immediate Kill Session#: 268, Serial#: 50893
> Immediate Kill Session: sess: 0x1245a5668  OS pid: 27778
> Immediate Kill Session#: 274, Serial#: 34459
> Immediate Kill Session: sess: 0x1245ad840  OS pid: 9808
> Immediate Kill Session#: 278, Serial#: 59445
> Immediate Kill Session: sess: 0x1245b2ed0  OS pid: 28434
> Immediate Kill Session#: 281, Serial#: 50119
> Immediate Kill Session: sess: 0x12757c948  OS pid: 12606
> Immediate Kill Session#: 282, Serial#: 30208
> Immediate Kill Session: sess: 0x1245b8560
>   OS pid: 17944
> Immediate Kill Session#: 285, Serial#: 53580
> Immediate Kill Session: sess: 0x127581fd8  OS pid: 16670
> Immediate Kill Session#: 286, Serial#: 5929
> Immediate Kill Session: sess: 0x1245bdbf0  OS pid: 20149
> Immediate Kill Session#: 289, Serial#: 53725
> Immediate Kill Session: sess: 0x127587668  OS pid: 14697
> Immediate Kill Session#: 290, Serial#: 30378
> Immediate Kill Session: sess: 0x1245c3280  OS pid: 19757
> Immediate Kill Session#: 293, Serial#: 53710
> Immediate Kill Session: sess: 0x12758ccf8  OS pid: 11096
> Immediate Kill Session#: 296, Serial#: 34022
> Immediate Kill Session: sess: 0x1245cb458  OS pid: 10881
> Immediate Kill Session#: 299, Serial#: 53951
> Immediate Kill Session: sess: 0x127594ed0  OS pid: 1453
> Immediate Kill Session#: 304, Serial#: 15149
> Immediate Kill Session: sess: 0x1245d6178  OS pid: 22008
> Immediate Kill Session#: 308, Serial#: 34245
> Immediate Kill Session: sess:
>  0x1245db808  OS pid: 19156
> Immediate Kill Session#: 315, Serial#: 15240
> Immediate Kill Session: sess: 0x1275aa910  OS pid: 32148
> Immediate Kill Session#: 317, Serial#: 41792
> Immediate Kill Session: sess: 0x1275ad458  OS pid: 15660
> Immediate Kill Session#: 318, Serial#: 7839
> Immediate Kill Session: sess: 0x1245e9070  OS pid: 24999
> Immediate Kill Session#: 321, Serial#: 4422
> Immediate Kill Session: sess: 0x1275b2ae8  OS pid: 16028
> Immediate Kill Session#: 324, Serial#: 6833
> Immediate Kill Session: sess: 0x1245f1248  OS pid: 21909
> Immediate Kill Session#: 332, Serial#: 18018
> Immediate Kill Session: sess: 0x1245fbf68  OS pid: 15819
> Immediate Kill Session#: 333, Serial#: 37534
> Immediate Kill Session: sess: 0x1275c2e98  OS pid: 16433
> Immediate Kill Session#: 334, Serial#: 50463
> Immediate Kill Session: sess: 0x1245feab0  OS pid: 5660
> Immediate Kill Session#: 335, Serial#: 11994
> Immediate Kill Session: sess:
>  0x1275c59e0  OS pid: 29575
> Immediate Kill Session#: 336, Serial#: 26542
> Immediate Kill Session: sess: 0x1246015f8  OS pid: 31868
> Immediate Kill Session#: 345, Serial#: 46583
> Immediate Kill Session: sess: 0x1275d3248  OS pid: 25399
> Sun Apr 05 15:05:43 2009
> ARCH: Archival stopped, error occurred. Will continue retrying
> ORACLE Instance ifxdb2 - Archival Error
> ORA-16014: log 10 sequence# 1917 not archived, no available destinations
> ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
> Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
> ORA-16014: log 10 sequence# 1917 not archived, no available destinations
> ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
> Sun Apr 05 15:10:52 2009
> kcrrdmx: Successful archiving of previously failed ORL
> Archiver process freed from errors. No longer stopped
> Sun Apr 05 15:10:53
>  2009
> ALTER SYSTEM SET
service_names='ifxdb.gointranet.com','ifxserv' SCOPE=MEMORY
SID='ifxdb2';
> These incidences are all occurring during archiving (redo logs and 
> database files are using ASM, archiving and backups are on OCFS2). 
> Even though it usually only lasts a few minutes, it's very noticeable 
> to the customers because of all the sessions that are terminated.
>
> What should we be looking at to resolve this problem?
> Please let me know if you have any questions.
>
> Thanks,
> Diane Petersen
> ServerCare, Inc.
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090406/c64879ea/attachment.html

Karim Alkhayer

2009-Apr-06 16:11 UTC

head link

[Ocfs2-users] Encountered disk I/O error 19502

Hello Diane,

 

I believe that implementing TAF could help a bit in this case, at least to
become transparent to the end users, unless of course, the following points
are blocking in your case:

 

1.       ALTER SESSION statements are lost:  

Statements such as "ALTER SESSION ..." are not automatically re-issued
to
the server following a failover. This can have a significant effect on
application behavior. For example: 

ALTER SESSION SET NLS_DATE_FORMAT='YYYY-MM-DD';

select sysdate from dual;

Result> 2009-01-31

<< Fail over the connection >>

select sysdate from dual;

Result> 31-JAN-09

2.       In-progress transactions must be rolled back 

3.       Continuing work on existing cursors may raise an error (eg:
ORA-25401 "cannot continue fetches") 

4.       Failed over selects may take time to re-position (when
FAILOVER_TYPE=SELECT) 

5.       Client awareness of a Failover

 

Can we have an overview of the database setup, nature of transactions, and
parameters?

 

It would also help to examine the troublesome node behavior and recovery
measures.

 

Best regards,

Karim Alkhayer

 

From: ocfs2-users-bounces at oss.oracle.com
[mailto:ocfs2-users-bounces at oss.oracle.com] On Behalf Of Diane Petersen
Sent: Monday, April 06, 2009 4:06 PM
To: ocfs2-users at oss.oracle.com
Subject: [Ocfs2-users] Encountered disk I/O error 19502

 

Hi,

We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5 with Linux
kernel 2.6.18-92.1.17.el5 64-bit. Lately we've been seeing errors on both
nodes almost ever other day. The system administrator has checked the SAN
array and said there are no issues being reported. 

Another part of the problem, it appears the instances alter the
service_names parameter not allowing new connections to the node with the
reported error, but also terminate sessions already connected using the RAC
service. The errors all start with - Encountered disk I/O error 19502 - and
contain the following:

ARC2: Encountered disk I/O error 19502






 (ifxdb2)






Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:






ORA-19502: write error on file
 "/u03/arch/2_1917_656008464.dbf", block number 155649 (block
size=512)






ORA-27072: File I/O error






Linux-x86_64 Error: 5: Input/output error






Additional information: 4






Additional information: 155649






Additional information: -1






ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)






Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:






ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)






ORA-27072: File I/O error






Linux-x86_64 Error: 5: Input/output error






Additional information: 4






Additional information: 155649






Additional information: -1






ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)






ARC2: I/O error 19502 archiving log 10 to
'/u03/arch/2_1917_656008464.dbf'






ARCH: Archival stopped, error occurred. Will continue retrying






ORACLE
 Instance ifxdb2 - Archival Error






ORA-16038: log 10 sequence# 1917 cannot be archived






ORA-19502: write error on file "", block number  (block size=)






ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'






Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:






ORA-16038: log 10 sequence# 1917 cannot be archived






ORA-19502: write error on file "", block number  (block size=)






ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'






Sun Apr 05 15:05:16 2009






ALTER SYSTEM SET service_names='ifxdb.gointranet.com' SCOPE=MEMORY
SID='ifxdb2';






Immediate Kill Session#: 185, Serial#: 40263






Immediate Kill Session: sess: 0x1274fabc8  OS pid: 13270






Immediate Kill Session#: 187, Serial#: 41391






Immediate Kill Session: sess: 0x1274fd710  OS pid: 27697






Immediate Kill Session#: 191, Serial#: 40464






Immediate Kill Session: sess: 0x127502da0  OS pid:
 30697






Immediate Kill Session#: 195, Serial#: 57362






Immediate Kill Session: sess: 0x127508430  OS pid: 27967






Immediate Kill Session#: 196, Serial#: 2028






Immediate Kill Session: sess: 0x124544048  OS pid: 22900






Immediate Kill Session#: 205, Serial#: 17412






Immediate Kill Session: sess: 0x127515c98  OS pid: 20110






Immediate Kill Session#: 206, Serial#: 14805






Immediate Kill Session: sess: 0x1245518b0  OS pid: 10464






Immediate Kill Session#: 207, Serial#: 52184






Immediate Kill Session: sess: 0x1275187e0  OS pid: 19787






Immediate Kill Session#: 208, Serial#: 62825






Immediate Kill Session: sess: 0x1245543f8  OS pid: 13578






Immediate Kill Session#: 213, Serial#: 36907






Immediate Kill Session: sess: 0x1275209b8  OS pid: 31397






Immediate Kill Session#: 214, Serial#: 49032






Immediate Kill Session: sess: 0x12455c5d0  OS pid: 2427






Immediate Kill Session#: 215, Serial#: 2711






Immediate Kill Session: sess: 0x127523500  OS
 pid: 15942






Immediate Kill Session#: 216, Serial#: 30060






Immediate Kill Session: sess: 0x12455f118  OS pid: 1217






Immediate Kill Session#: 219, Serial#: 35932






Immediate Kill Session: sess: 0x127528b90  OS pid: 27883






Immediate Kill Session#: 222, Serial#: 26007






Immediate Kill Session: sess: 0x1245672f0  OS pid: 1036






Immediate Kill Session#: 223, Serial#: 42462






Immediate Kill Session: sess: 0x12752e220  OS pid: 13726






Immediate Kill Session#: 224, Serial#: 33323






Immediate Kill Session: sess: 0x124569e38  OS pid: 29928






Immediate Kill Session#: 225, Serial#: 49752






Immediate Kill Session: sess: 0x127530d68  OS pid: 20147






Immediate Kill Session#: 227, Serial#: 34834






Immediate Kill Session: sess: 0x1275338b0  OS pid: 9365






Immediate Kill Session#: 230, Serial#: 19879






Immediate Kill Session: sess: 0x124572010  OS pid: 15791






Immediate Kill Session#: 231, Serial#: 16554






Immediate Kill Session: sess: 0x127538f40 
 OS pid: 15490






Immediate Kill Session#: 233, Serial#: 25251






Immediate Kill Session: sess: 0x12753ba88  OS pid: 6972






Immediate Kill Session#: 236, Serial#: 36970






Immediate Kill Session: sess: 0x12457a1e8  OS pid: 12354






Immediate Kill Session#: 244, Serial#: 37284






Immediate Kill Session: sess: 0x124584f08  OS pid: 19290






Immediate Kill Session#: 245, Serial#: 55792






Immediate Kill Session: sess: 0x12754be38  OS pid: 19288






Immediate Kill Session#: 246, Serial#: 25115






Immediate Kill Session: sess: 0x124587a50  OS pid: 3111






Immediate Kill Session#: 247, Serial#: 6416






Immediate Kill Session: sess: 0x12754e980  OS pid: 19471






Immediate Kill Session#: 251, Serial#: 19899






Immediate Kill Session: sess: 0x127554010  OS pid: 21486






Immediate Kill Session#: 252, Serial#: 34731






Immediate Kill Session: sess: 0x12458fc28  OS pid: 30540






Immediate Kill Session#: 253, Serial#: 32638






Immediate Kill Session: sess: 0x127556b58
  OS pid: 5493






Immediate Kill Session#: 259, Serial#: 29155






Immediate Kill Session: sess: 0x12755ed30  OS pid: 29463






Immediate Kill Session#: 261, Serial#: 14481






Immediate Kill Session: sess: 0x127561878  OS pid: 31054






Immediate Kill Session#: 265, Serial#: 37618






Immediate Kill Session: sess: 0x127566f08  OS pid: 868






Immediate Kill Session#: 267, Serial#: 42580






Immediate Kill Session: sess: 0x127569a50  OS pid: 16839






Immediate Kill Session#: 268, Serial#: 50893






Immediate Kill Session: sess: 0x1245a5668  OS pid: 27778






Immediate Kill Session#: 274, Serial#: 34459






Immediate Kill Session: sess: 0x1245ad840  OS pid: 9808






Immediate Kill Session#: 278, Serial#: 59445






Immediate Kill Session: sess: 0x1245b2ed0  OS pid: 28434






Immediate Kill Session#: 281, Serial#: 50119






Immediate Kill Session: sess: 0x12757c948  OS pid: 12606






Immediate Kill Session#: 282, Serial#: 30208






Immediate Kill Session: sess: 0x1245b8560
  OS pid: 17944






Immediate Kill Session#: 285, Serial#: 53580






Immediate Kill Session: sess: 0x127581fd8  OS pid: 16670






Immediate Kill Session#: 286, Serial#: 5929






Immediate Kill Session: sess: 0x1245bdbf0  OS pid: 20149






Immediate Kill Session#: 289, Serial#: 53725






Immediate Kill Session: sess: 0x127587668  OS pid: 14697






Immediate Kill Session#: 290, Serial#: 30378






Immediate Kill Session: sess: 0x1245c3280  OS pid: 19757






Immediate Kill Session#: 293, Serial#: 53710






Immediate Kill Session: sess: 0x12758ccf8  OS pid: 11096






Immediate Kill Session#: 296, Serial#: 34022






Immediate Kill Session: sess: 0x1245cb458  OS pid: 10881






Immediate Kill Session#: 299, Serial#: 53951






Immediate Kill Session: sess: 0x127594ed0  OS pid: 1453






Immediate Kill Session#: 304, Serial#: 15149






Immediate Kill Session: sess: 0x1245d6178  OS pid: 22008






Immediate Kill Session#: 308, Serial#: 34245






Immediate Kill Session: sess:
 0x1245db808  OS pid: 19156






Immediate Kill Session#: 315, Serial#: 15240






Immediate Kill Session: sess: 0x1275aa910  OS pid: 32148






Immediate Kill Session#: 317, Serial#: 41792






Immediate Kill Session: sess: 0x1275ad458  OS pid: 15660






Immediate Kill Session#: 318, Serial#: 7839






Immediate Kill Session: sess: 0x1245e9070  OS pid: 24999






Immediate Kill Session#: 321, Serial#: 4422






Immediate Kill Session: sess: 0x1275b2ae8  OS pid: 16028






Immediate Kill Session#: 324, Serial#: 6833






Immediate Kill Session: sess: 0x1245f1248  OS pid: 21909






Immediate Kill Session#: 332, Serial#: 18018






Immediate Kill Session: sess: 0x1245fbf68  OS pid: 15819






Immediate Kill Session#: 333, Serial#: 37534






Immediate Kill Session: sess: 0x1275c2e98  OS pid: 16433






Immediate Kill Session#: 334, Serial#: 50463






Immediate Kill Session: sess: 0x1245feab0  OS pid: 5660






Immediate Kill Session#: 335, Serial#: 11994






Immediate Kill Session: sess:
 0x1275c59e0  OS pid: 29575






Immediate Kill Session#: 336, Serial#: 26542






Immediate Kill Session: sess: 0x1246015f8  OS pid: 31868






Immediate Kill Session#: 345, Serial#: 46583






Immediate Kill Session: sess: 0x1275d3248  OS pid: 25399






Sun Apr 05 15:05:43 2009






ARCH: Archival stopped, error occurred. Will continue retrying






ORACLE Instance ifxdb2 - Archival Error






ORA-16014: log 10 sequence# 1917 not archived, no available destinations






ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'






Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:






ORA-16014: log 10 sequence# 1917 not archived, no available destinations






ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'






Sun Apr 05 15:10:52 2009






kcrrdmx: Successful archiving of previously failed ORL






Archiver process freed from errors. No longer stopped






Sun Apr 05 15:10:53
 2009






ALTER SYSTEM SET service_names='ifxdb.gointranet.com','ifxserv'
SCOPE=MEMORY
SID='ifxdb2';

These incidences are all occurring during archiving (redo logs and database
files are using ASM, archiving and backups are on OCFS2). Even though it
usually only lasts a few minutes, it's very noticeable to the customers
because of all the sessions that are terminated. 

What should we be looking at to resolve this problem?
Please let me know if you have any questions.

Thanks,
Diane Petersen
ServerCare, Inc.

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090406/c7d505a8/attachment-0001.html

Ocfs2 users - Apr 2009 - Encountered disk I/O error 19502

[Ocfs2-users] Encountered disk I/O error 19502

[Ocfs2-users] Encountered disk I/O error 19502

[Ocfs2-users] Encountered disk I/O error 19502