Xen users - Oct 2011 - domUs with iSCSI disks... connect in dom0 or domU?

Hi List,

I don''t know if this is something that is a simple matter of opinion or
if
there are strong reasons to take one route or the other.  I have dom0 nodes
with dedicated bond interfaces that connect to a storage traffic-only VLAN.
 Currently I have a few domUs running that have large-ish volumes on the
iSCSI SAN, and to present the volumes I''m connecting the dom0s to the
storage, then in the domU config I give it the iSCSI volumes as
''phy:/dev/disk/by-path/ip-10.10.10.5-iqn....:volume1'' and so
on.

I''m wondering if instead it''s considered better to simply have
the domU make
those iSCSI connections, by giving it a second interface that connects to a
bridge with access to the iSCSI VLAN?  I set one up that way instead, and it
works as expected though it seems to perform less well in some admittedly
simple tests using dd with ''oflag=direct'' set.  The dom0 can
write at about
110MB/s sustained, but if the domU makes the iSCSI connection instead
it''s
doing well to manage 60MB/s.

So I''m looking to the wisdom of the list, is it crazy to set things up
one
way or the other, or are they both an OK approach?  It seems like having the
dom0 handle the iSCSI connection is a big win performance-wise, but perhaps
that comes at a big negative cost?  Having the domU make the connection
simplifies the Pacemaker configuration, but with an apparent loss in disk
throughput.

This setup is initially just some fairly busy MySQL servers.  Running as
they are with dom0 iSCSI volumes, they''re very easily handling the load
but
I had to reboot one of them today because it started throwing errors
accessing a volume:

     Oct 16 12:00:36 slave3 kernel: end_request: I/O error, dev xvdc, sector
983424

The dom0 still showed the iSCSI connection as active and appeared fine, but
the domU couldn''t access the disk.  A reboot fixed it right up, further
indicating to me that the underlying iSCSI connection from dom0 was fine all
along since I did nothing with it at all.  Perhaps there''s some xm
command
that could have cleared the problem for domU as well, and I just don''t
know
it.

Thanks for any insight,
Mark


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

xenusers wrote:
>I don''t know if this is something that is a simple matter of
opinion
>or if there are strong reasons to take one route or the other.
There are good performance reasons ...
>Currently I have a few domUs running that have large-ish volumes on 
>the iSCSI SAN, and to present the volumes I''m connecting the dom0s 
>to the storage, then in the domU config I give it the iSCSI volumes 
>as ''phy:/dev/disk/by-path/ip-10.10.10.5-iqn....:volume1''
and so on.
>
>I''m wondering if instead it''s considered better to simply
have the
>domU make those iSCSI connections, by giving it a second interface 
>that connects to a bridge with access to the iSCSI VLAN?  I set one 
>up that way instead, and it works as expected though it seems to 
>perform less well in some admittedly simple tests using dd with 
>''oflag=direct'' set.
It''s been a topic of discussion here on and off in the past. 
Basically, as you found, attaching the iSCSI volume in the DomU 
results in a big performance hit, although IMO it is a much 
preferable setup as it isolates the guest requirements from the host 
and simplifies moving a guest.

AIUI, the networking in Xen is single threaded, and so your iSCSI 
traffic competes for this single thread with all the other traffic. I 
don''t know about disks, but I strongly suspect support for them is 
multi-threaded.

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 10/16/11 23:14, xenusers wrote:
> Hi List,
>
> I don''t know if this is something that is a simple matter of
opinion or
> if there are strong reasons to take one route or the other.  I have dom0
> nodes with dedicated bond interfaces that connect to a storage
> traffic-only VLAN.  Currently I have a few domUs running that have
> large-ish volumes on the iSCSI SAN, and to present the volumes I''m
> connecting the dom0s to the storage, then in the domU config I give it
> the iSCSI volumes as
> ''phy:/dev/disk/by-path/ip-10.10.10.5-iqn....:volume1'' and
so on.
>
> I''m wondering if instead it''s considered better to simply
have the domU
> make those iSCSI connections, by giving it a second interface that
> connects to a bridge with access to the iSCSI VLAN?  I set one up that
> way instead, and it works as expected though it seems to perform less
> well in some admittedly simple tests using dd with
''oflag=direct'' set.
>   The dom0 can write at about 110MB/s sustained, but if the domU makes
> the iSCSI connection instead it''s doing well to manage 60MB/s.
>
> So I''m looking to the wisdom of the list, is it crazy to set
things up
> one way or the other, or are they both an OK approach?  It seems like
> having the dom0 handle the iSCSI connection is a big win
> performance-wise, but perhaps that comes at a big negative cost?  Having
> the domU make the connection simplifies the Pacemaker configuration, but
> with an apparent loss in disk throughput.
Do you use HVM or PV DomUs? I wonder if this performance drop should 
hebben with PV.
Anyway, I definitely use the iSCSI connections in Dom0. As far as 
complexity is concerned: I combine iSCSI with multipath, allowing me to 
change the SCSI disks'' names to the names of the DomUs that actually
use
them. Complexity for the bigger part gone. It also provides you with the 
benefit of redundant paths towards your iSCSI.


HTH,

B.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello Bart,

On Mon, Oct 17, 2011 at 3:01 AM, Bart Coninckx
<bart.coninckx@telenet.be>wrote:
> On 10/16/11 23:14, xenusers wrote:
>
>> Hi List,
>>
>> I don''t know if this is something that is a simple matter of
opinion or
>> if there are strong reasons to take one route or the other.  I have
dom0
>> nodes with dedicated bond interfaces that connect to a storage
>> traffic-only VLAN.  Currently I have a few domUs running that have
>> large-ish volumes on the iSCSI SAN, and to present the volumes
I''m
>> connecting the dom0s to the storage, then in the domU config I give it
>> the iSCSI volumes as
>>
''phy:/dev/disk/by-path/ip-10.**10.10.5-iqn....:volume1'' and so
on.
>>
>> I''m wondering if instead it''s considered better to
simply have the domU
>> make those iSCSI connections, by giving it a second interface that
>> connects to a bridge with access to the iSCSI VLAN?  I set one up that
>> way instead, and it works as expected though it seems to perform less
>> well in some admittedly simple tests using dd with
''oflag=direct'' set.
>>  The dom0 can write at about 110MB/s sustained, but if the domU makes
>> the iSCSI connection instead it''s doing well to manage 60MB/s.
>>
>> So I''m looking to the wisdom of the list, is it crazy to set
things up
>> one way or the other, or are they both an OK approach?  It seems like
>> having the dom0 handle the iSCSI connection is a big win
>> performance-wise, but perhaps that comes at a big negative cost? 
Having
>> the domU make the connection simplifies the Pacemaker configuration,
but
>> with an apparent loss in disk throughput.
>>
>
> Do you use HVM or PV DomUs? I wonder if this performance drop should hebben
> with PV.
> Anyway, I definitely use the iSCSI connections in Dom0. As far as
> complexity is concerned: I combine iSCSI with multipath, allowing me to
> change the SCSI disks'' names to the names of the DomUs that
actually use
> them. Complexity for the bigger part gone. It also provides you with the
> benefit of redundant paths towards your iSCSI.
>
>
Yes, I should have mentioned that, these are PV domUs, so I think that
60MB/s is about as good as I''d be able to hope for if setting up iSCSI
within the domU.  My preference is to have all of the iSCSI configuration
done within dom0, anyhow.

I may fiddle around with multipathing again.  I tried both and bonding with
LACP seemed to be the fastest recovery for a lost link (1 second or so,
while muiltipath was taking two minutes).  I know there are some timers
you''re supposed to be able to adjust for multipath to get a much faster
failover, so if I get some time I''ll experiment.

Thanks,
Mark


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users