CentOS virt - Apr 2016 - Problems with scsi-target-utils when hosted on dom0 centos 7 xen box

By natively, I take it using
kernel /vmlinuz (vs kernel /xen)

Not yet, but working on setting up such an environment.

(At this time, I was using virt-install to reproduce the problem, and 
the original server we are testing on did not support kvm but the 2nd 
server does).

On 2016-04-12 03:26 AM, George Dunlap wrote:
> On Mon, Apr 11, 2016 at 9:14 PM, Nathan Coulson <nathan at
bravenet.com> wrote:
>> Hello
>>
>> We were attempting to use scsi-target-utils, hosted on a xen dom0 vm
using
>> localhost, and running into some problems.  I was not able to reproduce
this
>> on a centos 7.2 server using the default kernel.
> Have you tried booting the Virt SIG kernel natively and seeing if you
> can reproduce the problem at all?
>
> Thanks,
>   -George
>
>>
>> (From dmesg)
>> Apr  4 11:18:42 funk kernel: [  596.511204]  connection2:0: detected
conn
>> error (1022)
>> Apr  4 11:18:42 funk kernel: connection2:0: ping timeout of 5 secs
expired,
>> recv timeout 5, last rx 4295253788, last ping 4295258790, now
4295263808
>> Apr  4 11:18:42 funk kernel: connection2:0: detected conn error (1022)
>> Apr  4 11:18:42 funk iscsid: Kernel reported iSCSI connection 2:0 error
>> (1022 - Invalid or unknown error code) state (3)
>> Apr  4 11:18:44 funk iscsid: connection2:0 is operational after
recovery (1
>> attempts)
>>
>> Repeated a few times, until eventually
>>
>>
>> Apr  4 11:19:44 funk kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED
>> driverbyte=DRIVER_OK
>> Apr  4 11:19:44 funk kernel: sd 7:0:0:1: [sdd] CDB:
>> Apr  4 11:19:44 funk kernel: Write(10): 2a 00 01 df c7 e8 00 00 18 00
>> Apr  4 11:19:44 funk kernel: blk_update_request: I/O error, dev sdd,
sector
>> 31442920
>> Apr  4 11:19:44 funk kernel: [  658.127596] sd 7:0:0:1: [sdd]
>> Apr  4 11:19:44 funk kernel: [  658.127688] Result:
>> hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
>> Apr  4 11:19:44 funk kernel: [  658.127761] sd 7:0:0:1: [sdd] CDB:
>> Apr  4 11:19:44 funk kernel: [  658.127826] Write(10): 2a 00 01 df c7
e8 00
>> 00 18 00
>> Apr  4 11:19:44 funk kernel: [  658.127927] blk_update_request: I/O
error,
>> dev sdd, sector 31442920
>> Apr  4 11:19:44 funk kernel: [  658.128040] sd 7:0:0:1: [sdd]
>> Apr  4 11:19:44 funk kernel: sd 7:0:0:1: [sdd]
>> Apr  4 11:19:44 funk kernel: [  658.128105] Result:
>> hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
>> Apr  4 11:19:44 funk kernel: [  658.128177] sd 7:0:0:1: [sdd] CDB:
>> Apr  4 11:19:44 funk kernel: [  658.128241] Write(10): 2a 00 00 00 08
00 00
>> 00 18 00
>> Apr  4 11:19:44 funk kernel: [  658.128339] blk_update_request: I/O
error,
>> dev sdd, sector 2048
>> Apr  4 11:19:44 funk kernel: Result: hostbyte=DID_TRANSPORT_DISRUPTED
>> driverbyte=DRIVER_OK
>> Apr  4 11:19:44 funk kernel: sd 7:0:0:1: [sdd] CDB:
>> Apr  4 11:19:44 funk kernel: Write(10): 2a 00 00 00 08 00 00 00 18 00
>> Apr  4 11:19:44 funk kernel: blk_update_request: I/O error, dev sdd,
sector
>> 2048
>>
>>
>> (Test Setup)
>> scsi-target-utils installed via yum, default config
>> /etc/tgt/conf.d/xenguests.conf
>> <target iqn.2016-02.com.bravenet:test>
>>      backing-store //mnt/vmdisk/test # vm image
>> </target>
>>
>> systemctl tgtd restart
>>
>> iscsiadm -m discovery -t sendtargets -p localhost
>>
>> iscsiadm -m node -T iqn.2016-02.com.bravenet:test -l
>>
>>
>> add it to lvm (pvcreate, vgcreate), let's call it
/dev/vmdisk.vg/test.lv
>>
>> and then use libvirt to attempt to install an os on
/dev/vmdisk.vg/test.lv
>> (using anaconda)
>>
>>
>>
>>
>> Around the time it tries to create the disk label, is when the conn
errors
>> start, until eventually it gives up trying to create the disk label.
>>
>>
>>
>> We tested a similar setup on a centos 7.2 host we use kvm based
>> virtualmachine hosting on (default 3.10 kernel), and it worked fine. 
It may
>> be similar to what was reported on
>> https://bugzilla.redhat.com/show_bug.cgi?id=1245990, but I never saw a
>> resolution on what they discovered (other then a reference to comment18
>> which does not appear to exist).
>>
>> Testing over the network appears to also work as well (where another
machine
>> connects to scsi-target-utils on the funk server above.
>>
>>
>>
>>
>>
>> Longterm Purpose of the above setup, was to get direct access to a
>> filesystem image hosted on a gluster setup, using bs-type glfs on
>> scsi-target-utils.
>>
>> --
>> Nathan Coulson
>> www.bravenet.com
>> nathan at bravenet.com
>> _______________________________________________
>> CentOS-virt mailing list
>> CentOS-virt at centos.org
>> https://lists.centos.org/mailman/listinfo/centos-virt
> _______________________________________________
> CentOS-virt mailing list
> CentOS-virt at centos.org
> https://lists.centos.org/mailman/listinfo/centos-virt
>

-- 
Nathan Coulson
System Administrator for Bravenet
www.bravenet.com
nathan at bravenet.com

On 2016-04-12 09:43 AM, Nathan Coulson wrote:
> By natively, I take it using
> kernel /vmlinuz (vs kernel /xen)
>
> Not yet, but working on setting up such an environment.
>
> (At this time, I was using virt-install to reproduce the problem, and 
> the original server we are testing on did not support kvm but the 2nd 
> server does).
>
> On 2016-04-12 03:26 AM, George Dunlap wrote:
>> On Mon, Apr 11, 2016 at 9:14 PM, Nathan Coulson <nathan at
bravenet.com>
>> wrote:
>>> Hello
>>>
>>> We were attempting to use scsi-target-utils, hosted on a xen dom0
vm
>>> using
>>> localhost, and running into some problems.  I was not able to 
>>> reproduce this
>>> on a centos 7.2 server using the default kernel.
>> Have you tried booting the Virt SIG kernel natively and seeing if you
>> can reproduce the problem at all?
>>
>> Thanks,
>>   -George

(Apologies for the earlier top post)

Running the kernel natively, on 3.10 or 3.18 (kernel from virt sig)
* CentOS Linux (3.18.25-19.el7.x86_64) 7 (Core))
* CentOS Linux (3.10.0-327.13.1.el7.x86_64) 7 (Core)

It works as expected with no issues.



But when booting as dom0 using the xen hypervisor
* CentOS Linux, with Xen hypervisor

tested with dom0_mem=2048M,max:2048M (as well as 1024M)

the problems occur as I have describe.


>>>
>>> (From dmesg)
>>> Apr  4 11:18:42 funk kernel: [  596.511204]  connection2:0:
detected
>>> conn
>>> error (1022)
>>> Apr  4 11:18:42 funk kernel: connection2:0: ping timeout of 5 secs 
>>> expired,
>>> recv timeout 5, last rx 4295253788, last ping 4295258790, now 
>>> 4295263808
>>> Apr  4 11:18:42 funk kernel: connection2:0: detected conn error
(1022)
>>> Apr  4 11:18:42 funk iscsid: Kernel reported iSCSI connection 2:0
error
>>> (1022 - Invalid or unknown error code) state (3)
>>> Apr  4 11:18:44 funk iscsid: connection2:0 is operational after 
>>> recovery (1
>>> attempts)
>>>
>>> Repeated a few times, until eventually
>>>
>>>
>>> Apr  4 11:19:44 funk kernel: Result:
hostbyte=DID_TRANSPORT_DISRUPTED
>>> driverbyte=DRIVER_OK
>>> Apr  4 11:19:44 funk kernel: sd 7:0:0:1: [sdd] CDB:
>>> Apr  4 11:19:44 funk kernel: Write(10): 2a 00 01 df c7 e8 00 00 18
00
>>> Apr  4 11:19:44 funk kernel: blk_update_request: I/O error, dev
sdd,
>>> sector
>>> 31442920
>>> Apr  4 11:19:44 funk kernel: [  658.127596] sd 7:0:0:1: [sdd]
>>> Apr  4 11:19:44 funk kernel: [  658.127688] Result:
>>> hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
>>> Apr  4 11:19:44 funk kernel: [  658.127761] sd 7:0:0:1: [sdd] CDB:
>>> Apr  4 11:19:44 funk kernel: [  658.127826] Write(10): 2a 00 01 df 
>>> c7 e8 00
>>> 00 18 00
>>> Apr  4 11:19:44 funk kernel: [  658.127927] blk_update_request: I/O
>>> error,
>>> dev sdd, sector 31442920
>>> Apr  4 11:19:44 funk kernel: [  658.128040] sd 7:0:0:1: [sdd]
>>> Apr  4 11:19:44 funk kernel: sd 7:0:0:1: [sdd]
>>> Apr  4 11:19:44 funk kernel: [  658.128105] Result:
>>> hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
>>> Apr  4 11:19:44 funk kernel: [  658.128177] sd 7:0:0:1: [sdd] CDB:
>>> Apr  4 11:19:44 funk kernel: [  658.128241] Write(10): 2a 00 00 00 
>>> 08 00 00
>>> 00 18 00
>>> Apr  4 11:19:44 funk kernel: [  658.128339] blk_update_request: I/O
>>> error,
>>> dev sdd, sector 2048
>>> Apr  4 11:19:44 funk kernel: Result:
hostbyte=DID_TRANSPORT_DISRUPTED
>>> driverbyte=DRIVER_OK
>>> Apr  4 11:19:44 funk kernel: sd 7:0:0:1: [sdd] CDB:
>>> Apr  4 11:19:44 funk kernel: Write(10): 2a 00 00 00 08 00 00 00 18
00
>>> Apr  4 11:19:44 funk kernel: blk_update_request: I/O error, dev
sdd,
>>> sector
>>> 2048
>>>
>>>
>>> (Test Setup)
>>> scsi-target-utils installed via yum, default config
>>> /etc/tgt/conf.d/xenguests.conf
>>> <target iqn.2016-02.com.bravenet:test>
>>>      backing-store //mnt/vmdisk/test # vm image
>>> </target>
>>>
>>> systemctl tgtd restart
>>>
>>> iscsiadm -m discovery -t sendtargets -p localhost
>>>
>>> iscsiadm -m node -T iqn.2016-02.com.bravenet:test -l
>>>
>>>
>>> add it to lvm (pvcreate, vgcreate), let's call it 
>>> /dev/vmdisk.vg/test.lv
>>>
>>> and then use libvirt to attempt to install an os on 
>>> /dev/vmdisk.vg/test.lv
>>> (using anaconda)
>>>
>>>
>>>
>>>
>>> Around the time it tries to create the disk label, is when the conn
>>> errors
>>> start, until eventually it gives up trying to create the disk
label.
>>>
>>>
>>>
>>> We tested a similar setup on a centos 7.2 host we use kvm based
>>> virtualmachine hosting on (default 3.10 kernel), and it worked 
>>> fine.  It may
>>> be similar to what was reported on
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1245990, but I never
saw a
>>> resolution on what they discovered (other then a reference to
comment18
>>> which does not appear to exist).
>>>
>>> Testing over the network appears to also work as well (where
another
>>> machine
>>> connects to scsi-target-utils on the funk server above.
>>>
>>>
>>>
>>>
>>>
>>> Longterm Purpose of the above setup, was to get direct access to a
>>> filesystem image hosted on a gluster setup, using bs-type glfs on
>>> scsi-target-utils.
>>>
>>> -- 
>>> Nathan Coulson
>>> www.bravenet.com
>>> nathan at bravenet.com
>>> _______________________________________________
>>> CentOS-virt mailing list
>>> CentOS-virt at centos.org
>>> https://lists.centos.org/mailman/listinfo/centos-virt
>> _______________________________________________
>> CentOS-virt mailing list
>> CentOS-virt at centos.org
>> https://lists.centos.org/mailman/listinfo/centos-virt
>>
>
>

-- 
Nathan Coulson
www.bravenet.com
nathan at bravenet.com

On 04/12/2016 05:18 PM, Nathan Coulson wrote:
> (Apologies for the earlier top post)
> 
> Running the kernel natively, on 3.10 or 3.18 (kernel from virt sig)
> * CentOS Linux (3.18.25-19.el7.x86_64) 7 (Core))
> * CentOS Linux (3.10.0-327.13.1.el7.x86_64) 7 (Core)
> 
> It works as expected with no issues.
> 
> 
> 
> But when booting as dom0 using the xen hypervisor
> * CentOS Linux, with Xen hypervisor
> 
> tested with dom0_mem=2048M,max:2048M (as well as 1024M)
> 
> the problems occur as I have describe.
> 
This probably isn't it, but there was a problem I was having where the
workaround was to give the dom0 4096M+ so that 64 bit references were used
instead of 32 bit. You could try that and see what happens.

--Sarah

On Wed, Apr 13, 2016 at 1:18 AM, Nathan Coulson <nathan at bravenet.com>
wrote:
> On 2016-04-12 09:43 AM, Nathan Coulson wrote:
>>
>> By natively, I take it using
>> kernel /vmlinuz (vs kernel /xen)
>>
>> Not yet, but working on setting up such an environment.
>>
>> (At this time, I was using virt-install to reproduce the problem, and
the
>> original server we are testing on did not support kvm but the 2nd
server
>> does).
>>
>> On 2016-04-12 03:26 AM, George Dunlap wrote:
>>>
>>> On Mon, Apr 11, 2016 at 9:14 PM, Nathan Coulson <nathan at
bravenet.com>
>>> wrote:
>>>>
>>>> Hello
>>>>
>>>> We were attempting to use scsi-target-utils, hosted on a xen
dom0 vm
>>>> using
>>>> localhost, and running into some problems.  I was not able to
reproduce
>>>> this
>>>> on a centos 7.2 server using the default kernel.
>>>
>>> Have you tried booting the Virt SIG kernel natively and seeing if
you
>>> can reproduce the problem at all?
>>>
>>> Thanks,
>>>   -George
>
>
>
> (Apologies for the earlier top post)
>
> Running the kernel natively, on 3.10 or 3.18 (kernel from virt sig)
> * CentOS Linux (3.18.25-19.el7.x86_64) 7 (Core))
> * CentOS Linux (3.10.0-327.13.1.el7.x86_64) 7 (Core)
>
> It works as expected with no issues.
>
>
>
> But when booting as dom0 using the xen hypervisor
> * CentOS Linux, with Xen hypervisor
>
> tested with dom0_mem=2048M,max:2048M (as well as 1024M)
>
> the problems occur as I have describe.
Thanks Nathan.  It looks then like this may be an issue with upstream
Xen, then.  Would you be willing to re-post this bug report (with the
information that it works running the same kernel on native) to
xen-users?  I can make sure it's seen by the appropriate kernel-side
developers if necessary.

Thanks,
 -George