search for: blk_update_request

Back to replica 3 w/o arbiter. Two fio jobs running (direct=1 and direct=0), rebooting one node... and VM dmesg looks like: [ 483.862664] blk_update_request: I/O error, dev vda, sector 23125016 [ 483.898034] blk_update_request: I/O error, dev vda, sector 2161832 [ 483.901103] blk_update_request: I/O error, dev vda, sector 2161832 [ 483.904045] Aborting journal on device vda1-8. [ 483.906959] blk_update_request: I/O error, dev vda, sector 2099200 [...

FYI I set up replica 3 (no arbiter this time), did the same thing - rebooted one node during lots of file IO on VM and IO stopped. As I mentioned either here or in another thread, this behavior is caused by high default of network.ping-timeout. My main problem used to be that setting it to low values like 3s or even 2s did not prevent the FS to be mounted as read-only in the past (at least with

...tall went fine and ran for a couple days then got errors on the console. Here is an example: [168176.995064] sd 0:0:0:0: [sda] tag#14 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [168177.004050] sd 0:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 01 04 68 b0 00 00 08 00 [168177.011615] blk_update_request: I/O error, dev sda, sector 17066160 [168487.534510] sd 0:0:0:0: [sda] tag#17 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [168487.543576] sd 0:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 01 04 68 b0 00 00 08 00 [168487.551206] blk_update_request: I/O error, dev sda, sector 17066160 [1...

I would check disks status and accessibility of mount points where your gluster volumes reside. On Tue, May 29, 2018, 22:28 Jim Kusznir <jim at palousetech.com> wrote: > On one ovirt server, I'm now seeing these messages: > [56474.239725] blk_update_request: 63 callbacks suppressed > [56474.239732] blk_update_request: I/O error, dev dm-2, sector 0 > [56474.240602] blk_update_request: I/O error, dev dm-2, sector 3905945472 > [56474.241346] blk_update_request: I/O error, dev dm-2, sector 3905945584 > [56474.242236] blk_update_request: I/O er...

...(1 attempts) Repeated a few times, until eventually Apr 4 11:19:44 funk kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Apr 4 11:19:44 funk kernel: sd 7:0:0:1: [sdd] CDB: Apr 4 11:19:44 funk kernel: Write(10): 2a 00 01 df c7 e8 00 00 18 00 Apr 4 11:19:44 funk kernel: blk_update_request: I/O error, dev sdd, sector 31442920 Apr 4 11:19:44 funk kernel: [ 658.127596] sd 7:0:0:1: [sdd] Apr 4 11:19:44 funk kernel: [ 658.127688] Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK Apr 4 11:19:44 funk kernel: [ 658.127761] sd 7:0:0:1: [sdd] CDB: Apr 4 11:19:44 funk kern...

...the >> console. Here is an example: >> >> [168176.995064] sd 0:0:0:0: [sda] tag#14 FAILED Result: >> hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK >> [168177.004050] sd 0:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 01 04 68 b0 >> 00 00 08 00 >> [168177.011615] blk_update_request: I/O error, dev sda, sector 17066160 >> [168487.534510] sd 0:0:0:0: [sda] tag#17 FAILED Result: >> hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK >> [168487.543576] sd 0:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 01 04 68 b0 >> 00 00 08 00 >> [168487.551206] blk_update_requ...

...gt;> the console. Here is an example: >> >> [168176.995064] sd 0:0:0:0: [sda] tag#14 FAILED Result: >> hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK >> [168177.004050] sd 0:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 01 04 68 >> b0 00 00 08 00 >> [168177.011615] blk_update_request: I/O error, dev sda, sector 17066160 >> [168487.534510] sd 0:0:0:0: [sda] tag#17 FAILED Result: >> hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK >> [168487.543576] sd 0:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 01 04 68 >> b0 00 00 08 00 >> [168487.551206] blk_update_re...

...gt; >> >> Apr 4 11:19:44 funk kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED >> driverbyte=DRIVER_OK >> Apr 4 11:19:44 funk kernel: sd 7:0:0:1: [sdd] CDB: >> Apr 4 11:19:44 funk kernel: Write(10): 2a 00 01 df c7 e8 00 00 18 00 >> Apr 4 11:19:44 funk kernel: blk_update_request: I/O error, dev sdd, sector >> 31442920 >> Apr 4 11:19:44 funk kernel: [ 658.127596] sd 7:0:0:1: [sdd] >> Apr 4 11:19:44 funk kernel: [ 658.127688] Result: >> hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK >> Apr 4 11:19:44 funk kernel: [ 658.127761] sd 7:...

...imes, until eventually > > > Apr 4 11:19:44 funk kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED > driverbyte=DRIVER_OK > Apr 4 11:19:44 funk kernel: sd 7:0:0:1: [sdd] CDB: > Apr 4 11:19:44 funk kernel: Write(10): 2a 00 01 df c7 e8 00 00 18 00 > Apr 4 11:19:44 funk kernel: blk_update_request: I/O error, dev sdd, sector > 31442920 > Apr 4 11:19:44 funk kernel: [ 658.127596] sd 7:0:0:1: [sdd] > Apr 4 11:19:44 funk kernel: [ 658.127688] Result: > hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK > Apr 4 11:19:44 funk kernel: [ 658.127761] sd 7:0:0:1: [sdd] CDB: &g...

...nd ran for a couple days then got errors on the console. Here is an example: > > [168176.995064] sd 0:0:0:0: [sda] tag#14 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK > [168177.004050] sd 0:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 01 04 68 b0 00 00 08 00 > [168177.011615] blk_update_request: I/O error, dev sda, sector 17066160 > [168487.534510] sd 0:0:0:0: [sda] tag#17 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK > [168487.543576] sd 0:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 01 04 68 b0 00 00 08 00 > [168487.551206] blk_update_request: I/O error, dev sda, secto...

...tall went fine and ran for a couple days then got errors on the console. Here is an example: [168176.995064] sd 0:0:0:0: [sda] tag#14 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [168177.004050] sd 0:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 01 04 68 b0 00 00 08 00 [168177.011615] blk_update_request: I/O error, dev sda, sector 17066160 [168487.534510] sd 0:0:0:0: [sda] tag#17 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [168487.543576] sd 0:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 01 04 68 b0 00 00 08 00 [168487.551206] blk_update_request: I/O error, dev sda, sector 17066160 [1...

...uple days then got errors on the > console. Here is an example: > > [168176.995064] sd 0:0:0:0: [sda] tag#14 FAILED Result: > hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK > [168177.004050] sd 0:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 01 04 68 b0 > 00 00 08 00 > [168177.011615] blk_update_request: I/O error, dev sda, sector 17066160 > [168487.534510] sd 0:0:0:0: [sda] tag#17 FAILED Result: > hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK > [168487.543576] sd 0:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 01 04 68 b0 > 00 00 08 00 > [168487.551206] blk_update_request: I/O error, dev...

...e days then got errors on the > console. Here is an example: > > [168176.995064] sd 0:0:0:0: [sda] tag#14 FAILED Result: > hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK > [168177.004050] sd 0:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 01 04 68 b0 > 00 00 08 00 > [168177.011615] blk_update_request: I/O error, dev sda, sector 17066160 > [168487.534510] sd 0:0:0:0: [sda] tag#17 FAILED Result: > hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK > [168487.543576] sd 0:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 01 04 68 b0 > 00 00 08 00 > [168487.551206] blk_update_request: I/O error, de...

...;> Apr 4 11:19:44 funk kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED >>> driverbyte=DRIVER_OK >>> Apr 4 11:19:44 funk kernel: sd 7:0:0:1: [sdd] CDB: >>> Apr 4 11:19:44 funk kernel: Write(10): 2a 00 01 df c7 e8 00 00 18 00 >>> Apr 4 11:19:44 funk kernel: blk_update_request: I/O error, dev sdd, >>> sector >>> 31442920 >>> Apr 4 11:19:44 funk kernel: [ 658.127596] sd 7:0:0:1: [sdd] >>> Apr 4 11:19:44 funk kernel: [ 658.127688] Result: >>> hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK >>> Apr 4 11:19:44...

...ince some weeks ago, server is restarting after XFS errors. Logs in /var/crash reported this information: [...] [443804.295916] sd 0:0:0:0: [sda] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK [443804.295919] sd 0:0:0:0: [sda] CDB: Read(10) 28 00 04 53 e8 b0 00 00 28 00 [443804.295922] blk_update_request: I/O error, dev sda, sector 72607920 [443804.295969] sd 0:0:0:0: rejecting I/O to offline device [443804.296245] sd 0:0:0:0: rejecting I/O to offline device [443804.296288] systemd-journal[946]: segfault at 55f73081c820 ip 000055f73081c820 sp 00007fffb3c0f558 error 15 [443804.321330] sd 0:0:1:0: [s...

...the >> console. Here is an example: >> >> [168176.995064] sd 0:0:0:0: [sda] tag#14 FAILED Result: >> hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK >> [168177.004050] sd 0:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 01 04 68 b0 >> 00 00 08 00 >> [168177.011615] blk_update_request: I/O error, dev sda, sector 17066160 >> [168487.534510] sd 0:0:0:0: [sda] tag#17 FAILED Result: >> hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK >> [168487.543576] sd 0:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 01 04 68 b0 >> 00 00 08 00 >> [168487.551206] blk_update_requ...

...oints where your >>>> gluster volumes reside. >>>> >>>> On Tue, May 29, 2018, 22:28 Jim Kusznir <jim at palousetech.com> wrote: >>>> >>>>> On one ovirt server, I'm now seeing these messages: >>>>> [56474.239725] blk_update_request: 63 callbacks suppressed >>>>> [56474.239732] blk_update_request: I/O error, dev dm-2, sector 0 >>>>> [56474.240602] blk_update_request: I/O error, dev dm-2, sector >>>>> 3905945472 >>>>> [56474.241346] blk_update_request: I/O error, dev d...

...h2 # Before I start test09 I open a console to the guest and observe what is going on in it. Once I try to start test09 (and get a message about locked resource) on h2 hypervisor, I can see the following messages in the console and the vm goes to ro mode: on test09's console: [ 567.394148] blk_update_request: I/O error, dev vda, sector 13296056 [ 567.395883] blk_update_request: I/O error, dev vda, sector 13296056 [ 572.871905] blk_update_request: I/O error, dev vda, sector 8654040 [ 572.872627] Aborting journal on device vda1-8. [ 572.873978] blk_update_request: I/O error, dev vda, sector 8652800...

...imes, until eventually > > > Apr 4 11:19:44 funk kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED > driverbyte=DRIVER_OK > Apr 4 11:19:44 funk kernel: sd 7:0:0:1: [sdd] CDB: > Apr 4 11:19:44 funk kernel: Write(10): 2a 00 01 df c7 e8 00 00 18 00 > Apr 4 11:19:44 funk kernel: blk_update_request: I/O error, dev sdd, > sector 31442920 > Apr 4 11:19:44 funk kernel: [ 658.127596] sd 7:0:0:1: [sdd] > Apr 4 11:19:44 funk kernel: [ 658.127688] Result: > hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK > Apr 4 11:19:44 funk kernel: [ 658.127761] sd 7:0:0:1: [sdd] CDB: &g...

...s an example: > >> > >> [168176.995064] sd 0:0:0:0: [sda] tag#14 FAILED Result: > >> hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK > >> [168177.004050] sd 0:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 01 04 68 b0 > >> 00 00 08 00 > >> [168177.011615] blk_update_request: I/O error, dev sda, sector 17066160 > >> [168487.534510] sd 0:0:0:0: [sda] tag#17 FAILED Result: > >> hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK > >> [168487.543576] sd 0:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 01 04 68 b0 > >> 00 00 08 00 > >> [16848...