CentOS - May 2020 - XFS problem

Hello,

My server is running kernel 3.10.0-1062.12.1 in a CentOS Linux release 7.7.1908.
Since some weeks ago, server is restarting after XFS errors. Logs in /var/crash
reported this information:
[...]
[443804.295916] sd 0:0:0:0: [sda] FAILED Result: hostbyte=DID_NO_CONNECT
driverbyte=DRIVER_OK
[443804.295919] sd 0:0:0:0: [sda] CDB: Read(10) 28 00 04 53 e8 b0 00 00 28 00
[443804.295922] blk_update_request: I/O error, dev sda, sector 72607920
[443804.295969] sd 0:0:0:0: rejecting I/O to offline device
[443804.296245] sd 0:0:0:0: rejecting I/O to offline device
[443804.296288] systemd-journal[946]: segfault at 55f73081c820 ip
000055f73081c820 sp 00007fffb3c0f558 error 15
[443804.321330] sd 0:0:1:0: [sdb] tag#35 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_TIMEOUT
[443804.321335] sd 0:0:1:0: [sdb] tag#35 CDB: Write(16) 8a 00 00 00 00 02 67 e3
f5 f8 00 00 02 00 00 00
[443804.321338] blk_update_request: I/O error, dev sdb, sector 10332927480
[443804.321376] sd 0:0:0:0: rejecting I/O to offline device
[443804.321384] XFS (dm-2): metadata I/O error: block 0xf00001
("xfs_trans_read_buf_map") error 5 numblks 1
[443804.321390] XFS (dm-2): xfs_do_force_shutdown(0x1) called from line 239 of
file fs/xfs/libxfs/xfs_defer.c.  Return address = 0xffffffffc073a90b
[443804.321421] sd 0:0:0:0: rejecting I/O to offline device
[443804.321431] XFS (dm-2): metadata I/O error: block 0x1e04cee
("xlog_iodone") error 5 numblks 64
[443804.321433] XFS (dm-2): xfs_do_force_shutdown(0x2) called from line 1238 of
file fs/xfs/xfs_log.c.  Return address = 0xffffffffc077b210
[443804.321442] XFS (dm-2): Log I/O Error Detected.  Shutting down filesystem
[443804.321443] XFS (dm-2): Please umount the filesystem and rectify the
problem(s)
[443804.351688] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b
[...]

It seems problem are hard disk or XFS module, but physical RAID controller
reports all hard disks are OK, so I suppose problem is XFS module. Also, I have
read in this forum (https://bugs.centos.org/view.php?id=16960) something
similar.

Could someone say me is lastest kernel version for CentOS-7 (3.10.0-1127) solves
this problem?

Thanks.

On Mon, 11 May 2020 12:40:15 +0000
Gesti? Servidors <sysadmin.caos at uab.cat> wrote:
> Hello,
> 
> My server is running kernel 3.10.0-1062.12.1 in a CentOS Linux
> release 7.7.1908. Since some weeks ago, server is restarting after
> XFS errors. Logs in /var/crash reported this information: [...]
...
> [443804.295922] blk_update_request: I/O error, dev sda, sector
> 72607920 [443804.295969] sd 0:0:0:0: rejecting I/O to offline device
...
> 10332927480 [443804.321376] sd 0:0:0:0: rejecting I/O to offline
> device
> [443804.321384] XFS (dm-2): metadata I/O error: block 0xf00001
... 
> It seems problem are hard disk or XFS module, but physical RAID
> controller reports all hard disks are OK, so I suppose problem is XFS
> module. Also, I have read in this forum
> (https://bugs.centos.org/view.php?id=16960) something similar.
Just like your logs the thread you're referring to had SCSI errors
before the XFS errors. The thread ended with the OP concluding that it
was a problem on the hypervisor side (nothing to do with kernel/xfs on
the guest).

If you have a hardware RAID serving that block device I'd look at that
(and possibly its driver).
 
> Could someone say me is lastest kernel version for CentOS-7
> (3.10.0-1127) solves this problem?
Since neither are clear or confirmed it is not likely that anyone can
say...

/Peter