Xen users - Nov 2008 - Performance of network block devices (iSCSI)

I have a ''backup'' server to which I have a number of machines
dumping
their filesystems using rdiff-backup. The backup server is storing 
this data on a volume mounted off an iSCSI store (Dell/EMC AX100i). 
I''ve found the performance to be ''very poor'' and
asked on the
rdiff-backup list, a response I got was :
>I found that the network and I/O scheduler in Xen was a single 
>pipeline and contention was terrible. We got terrible performance 
>when we used network block devices with Xen, as the VMs would just 
>sit in waitI/O all the time when accessing the network block devices 
>(we tried AoE, NBD, iSCSI).
>...
>We ended up moving to OpenVZ and haven''t looked back.
I''ve done a test after copying the store to a local disk (xvda) which 
is another volume in the LVM setup of the Xen host - it''s notable 
that copying the backup off the iSCSI volume ran at only about 
1/2G/hr. The difference is quite dramatic, a backup from one client 
takes 36s to a local disk, but 9 1/2 minutes to the iSCSI box - 
that''s a 15 fold difference.

While copying to or from the iSCSI volume the backup server sits at 
100% (occasionally 99%) wait-io, while backing up to the virtual disk 
it shows the normal levels of processor activity I would expect (with 
minimal wait-io).

Systems are Debian Lenny, running on a Dell 2650 with hardware raid 
(PERC) and plenty of RAM.

Is there something I''ve missed ? Is there anything I can do ?

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I have a ''backup'' server to which I have a number of machines
dumping
their filesystems using rdiff-backup. The backup server is storing 
this data on a volume mounted off an iSCSI store (Dell/EMC AX100i). 
I''ve found the performance to be ''very poor'' and
asked on the
rdiff-backup list, a response I got was :
>I found that the network and I/O scheduler in Xen was a single 
>pipeline and contention was terrible. We got terrible performance 
>when we used network block devices with Xen, as the VMs would just 
>sit in waitI/O all the time when accessing the network block devices 
>(we tried AoE, NBD, iSCSI).
>...
>We ended up moving to OpenVZ and haven''t looked back.
I''ve done a test after copying the store to a local disk (xvda) which 
is another volume in the LVM setup of the Xen host - it''s notable 
that copying the backup off the iSCSI volume ran at only about 
1/2G/hr. The difference is quite dramatic, a backup from one client 
takes 36s to a local disk, but 9 1/2 minutes to the iSCSI box - 
that''s a 15 fold difference.

While copying to or from the iSCSI volume the backup server sits at 
100% (occasionally 99%) wait-io, while backing up to the virtual disk 
it shows the normal levels of processor activity I would expect (with 
minimal wait-io).

Systems are Debian Lenny, running on a Dell 2650 with hardware raid 
(PERC) and plenty of RAM.

Is there something I''ve missed ? Is there anything I can do ?

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thursday 13 November 2008 03:58:24 pm Simon Hobson
wrote:
> Is there something I''ve missed ? Is there anything I can do ?
Is the initiator in dom0 or domU?  I believe the recommendation is for dom0.

John






-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

John Madden wrote:
>Is the initiator in dom0 or domU?  I believe the recommendation is for dom0.
DomU

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thursday 13 November 2008 04:48:43 pm Simon Hobson
wrote:
> >Is the initiator in dom0 or domU?  I believe the recommendation is for
> > dom0.
>
> DomU
I''d try moving it to dom0 so only the block i/o component of the
picture goes
over the hypervisor.  You''re doing a lot of double-work by doing both
block
i/o and the networking needed for iscsi when you do the initiator in domU.  
Configuration will be easier, too.

John




-- 
John Madden
Sr. UNIX Systems Engineer
Ivy Tech Community College of Indiana
jmadden@ivytech.edu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

John Madden wrote:
>I''d try moving it to dom0 so only the block i/o component of the
picture goes
>over the hypervisor.  You''re doing a lot of double-work by doing
both block
>i/o and the networking needed for iscsi when you do the initiator in domU.
>Configuration will be easier, too.
Not sure what you are saying about the double work - with the DomU 
doing the iSCSI, all the Dom0 has to do is pass the network packets 
through. Moving the iSCSI initiator to the Dom0 means that the Dom0 
has to do the iSCSI stuff and then route the virtual block device to 
the DomU.

I''ve deliberately not put the iSCSI initiator in Dom0 as I want to 
run the absolute minimum in Dom0. Also, putting the iSCSI initiator 
in Dom0 makes it harder to move a VM to another host - having it in 
the DomU means that the VM can be moved without any config changes.

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Nov 13, 2008, at 5:58 PM, Simon Hobson <linux@thehobsons.co.uk>  
wrote:
> John Madden wrote:
>
>> I''d try moving it to dom0 so only the block i/o component of
the
>> picture goes
>> over the hypervisor.  You''re doing a lot of double-work by
doing
>> both block
>> i/o and the networking needed for iscsi when you do the initiator  
>> in domU.
>> Configuration will be easier, too.
>
> Not sure what you are saying about the double work - with the DomU  
> doing the iSCSI, all the Dom0 has to do is pass the network packets  
> through. Moving the iSCSI initiator to the Dom0 means that the Dom0  
> has to do the iSCSI stuff and then route the virtual block device to  
> the DomU.
>
> I''ve deliberately not put the iSCSI initiator in Dom0 as I want to
> run the absolute minimum in Dom0. Also, putting the iSCSI initiator  
> in Dom0 makes it harder to move a VM to another host - having it in  
> the DomU means that the VM can be moved without any config changes.
Try running netperf between dom0 and domU to see how well the  
networking is working, then between domU and the iSCSI target (or  
another external host).

There have been weird problems with vlans too, dropping ids on  
forwards depending on the source card and how it pads and such. Google  
for Xen network performance issue/problem.

-Ross


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> 
> I have a ''backup'' server to which I have a number of
machines dumping
> their filesystems using rdiff-backup. The backup server is storing
> this data on a volume mounted off an iSCSI store (Dell/EMC AX100i).
> I''ve found the performance to be ''very poor'' and
asked on the
> rdiff-backup list, a response I got was :
> 
> >I found that the network and I/O scheduler in Xen was a single
> >pipeline and contention was terrible. We got terrible performance
> >when we used network block devices with Xen, as the VMs would just
> >sit in waitI/O all the time when accessing the network block devices
> >(we tried AoE, NBD, iSCSI).
> >...
> >We ended up moving to OpenVZ and haven''t looked back.
> 
> I''ve done a test after copying the store to a local disk (xvda)
which
> is another volume in the LVM setup of the Xen host - it''s notable
> that copying the backup off the iSCSI volume ran at only about
> 1/2G/hr. The difference is quite dramatic, a backup from one client
> takes 36s to a local disk, but 9 1/2 minutes to the iSCSI box -
> that''s a 15 fold difference.
> 
> While copying to or from the iSCSI volume the backup server sits at
> 100% (occasionally 99%) wait-io, while backing up to the virtual disk
> it shows the normal levels of processor activity I would expect (with
> minimal wait-io).
> 
> Systems are Debian Lenny, running on a Dell 2650 with hardware raid
> (PERC) and plenty of RAM.
> 
> Is there something I''ve missed ? Is there anything I can do ?
> 
Try turning off all network offloading (ethtool -k ...) in DomU. If that
doesn''t improve things, try turning it off on the vif in Dom0, and then
if that doesn''t work either, the hardware adapter in Dom0.

James

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

James Harper wrote:
>Try turning off all network offloading (ethtool -k ...) in DomU. If that
>doesn''t improve things, try turning it off on the vif in Dom0, and
then
>if that doesn''t work either, the hardware adapter in Dom0.
No effect

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Nov 13, 2008 at 5:58 PM, Simon Hobson <linux@thehobsons.co.uk>
wrote:
> I''ve deliberately not put the iSCSI initiator in Dom0 as I want to
run the
> absolute minimum in Dom0. Also, putting the iSCSI initiator in Dom0 makes
it
> harder to move a VM to another host - having it in the DomU means that the
> VM can be moved without any config changes.
you should really try that.  the different nature of net- and
block-devices means that there are a lot less context switches per MB
transferred, and the point of acknowledging makes a huge difference in
latency sensibility.

about migrations, just make sure that the iSCSI volume is available
from all Dom0''s.  of course, it won''t be used on those where
that
specific DomU isn''t running.

-- 
Javier

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Simon Hobson <linux@thehobsons.co.uk> writes:
> I''ve deliberately not put the iSCSI initiator in Dom0 as I want to
run
> the absolute minimum in Dom0. Also, putting the iSCSI initiator in
> Dom0 makes it harder to move a VM to another host - having it in the
> DomU means that the VM can be moved without any config changes.
Yes, I run the same configuration for the same reasons.  Basically, I
couldn''t find a block-iscsi helper script to use with Debian Lenny,
and performance was acceptable with this setup: I''m able to copy data
between iSCSI devices at 10 MB/s.  So two of my domUs are iSCSI rooted.
On the other hand, iSCSI root support in Lenny isn''t that great either,
so I''ll surely test putting the initiator into the dom0s as well.
That will also simplify access control management, which is the weakest
point of the current solution.
-- 
Cheers,
Feri.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 14/11/08, Javier Guerra wrote:
>On Thu, Nov 13, 2008 at 5:58 PM, Simon Hobson <linux@thehobsons.co.uk>
wrote:
>>  I''ve deliberately not put the iSCSI initiator in Dom0 as I
want to run the
>>  absolute minimum in Dom0. Also, putting the iSCSI initiator in Dom0
makes it
>>  harder to move a VM to another host - having it in the DomU means that
the
>>  VM can be moved without any config changes.
>
>you should really try that.  the different nature of net- and
>block-devices means that there are a lot less context switches per MB
>transferred, and the point of acknowledging makes a huge difference in
>latency sensibility.
OK, I''ve *finally* managed to scrounge another box to do some testing 
on. I''ve setup open-iscsi in Dom0 and performance seems to be a lot 
better - I''m getting about 20MB/s and the backup server isn''t
sat on
99% wio :-)


I have another question though ...

on this page http://wiki.debian.org/iSCSI/open-iscsi it says
:
>You can then mount the new device, but under most situation, you 
>will want to use LABEL= syntax in /etc/fstab (see e2label).
I assume this is because the device name can change across restarts. 
Since I''m not mounting the volume in Dom0, just passing it to the 
guest in a "disk = [ ''phy:v ..." line, any suggestions on the
best
way to deal with this ?


-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Simon Hobson <linux@thehobsons.co.uk> writes:
> on this page http://wiki.debian.org/iSCSI/open-iscsi it says :
>
>> You can then mount the new device, but under most situation, you
>> will want to use LABEL= syntax in /etc/fstab (see e2label).
>
> I assume this is because the device name can change across
> restarts. Since I''m not mounting the volume in Dom0, just passing
it
> to the guest in a "disk = [ ''phy:v ..." line, any
suggestions on the
> best way to deal with this ?
Use the /dev/disk/by-path/ip-* symlinks.
-- 
Feri.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ferenc Wagner wrote:
> Simon Hobson <linux@thehobsons.co.uk> writes:
>
>   
>> on this page http://wiki.debian.org/iSCSI/open-iscsi it says :
>>
>>     
>>> You can then mount the new device, but under most situation, you
>>> will want to use LABEL= syntax in /etc/fstab (see e2label).
>>>       
>> I assume this is because the device name can change across
>> restarts. Since I''m not mounting the volume in Dom0, just
passing it
>> to the guest in a "disk = [ ''phy:v ..." line, any
suggestions on the
>> best way to deal with this ?
>>     
>
> Use the /dev/disk/by-path/ip-* symlinks.
>   
Or /dev/disk/by-id/*-part symlinks. Or you can try hacking your own udev 
rules. I kind of liked the methods described in 
http://www.performancemagic.com/iscsi-xen-howto/iSCSI.html . But any of 
the above should work and be unique. IIRC, the by-id links or what not 
should also be unique across machines.

Mike


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Mike Lovell wrote:
>Ferenc Wagner wrote:
>
>>Use the /dev/disk/by-path/ip-* symlinks.
>>
>>
>Or /dev/disk/by-id/*-part symlinks. Or you can try hacking your own 
>udev rules. I kind of liked the methods described in 
>http://www.performancemagic.com/iscsi-xen-howto/iSCSI.html . But any 
>of the above should work and be unique. IIRC, the by-id links or 
>what not should also be unique across machines.
Thanks for both of those suggestions. I''ll have a good read of that 
howto when I get a few minutes to think !

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 24 Feb 2009, Simon Hobson wrote:
> Mike Lovell wrote:
> >Ferenc Wagner wrote:
> >
> >>Use the /dev/disk/by-path/ip-* symlinks.
> >>
> >>
> >Or /dev/disk/by-id/*-part symlinks. Or you can try hacking your own
> >udev rules. I kind of liked the methods described in
> >http://www.performancemagic.com/iscsi-xen-howto/iSCSI.html . But any
> >of the above should work and be unique. IIRC, the by-id links or
> >what not should also be unique across machines.
>
> Thanks for both of those suggestions. I''ll have a good read of
that
> howto when I get a few minutes to think !
http://www.kinkrsoftware.nl/contrib/xen/block-iscsi


;) alternative ;)


Stefan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, 23 Feb 2009, Simon Hobson wrote:
> On 14/11/08, Javier Guerra wrote:
>> On Thu, Nov 13, 2008 at 5:58 PM, Simon Hobson
<linux@thehobsons.co.uk>
>> wrote:
>>>  I''ve deliberately not put the iSCSI initiator in Dom0 as
I want to run
>>> the
>>>  absolute minimum in Dom0. Also, putting the iSCSI initiator in
Dom0 makes
>>> it
>>>  harder to move a VM to another host - having it in the DomU means
that
>>> the
>>>  VM can be moved without any config changes.
>> 
>> you should really try that.  the different nature of net- and
>> block-devices means that there are a lot less context switches per MB
>> transferred, and the point of acknowledging makes a huge difference in
>> latency sensibility.
>
> OK, I''ve *finally* managed to scrounge another box to do some
testing on.
> I''ve setup open-iscsi in Dom0 and performance seems to be a lot
better - I''m
> getting about 20MB/s and the backup server isn''t sat on 99% wio
:-)
1. I wonder what test did you make to check performance?
I have the same concept with iscsi initiator directly on the domU via 
bonding interface-->dedicated vlan on top of bonding-->transparent 
bridge in Dom0-->DomU''s eth1.

I would expect more advantages (performance, administration) with 
iscsi-->DomU model.
> I assume this is because the device name can change across restarts. Since 
> I''m not mounting the volume in Dom0, just passing it to the guest
in a "disk
> = [ ''phy:v ..." line, any suggestions on the best way to deal
with this ?
2. how can you pass the storage without mounting it/having access to, in 
Dom0?

regards
Longina
-- 
--
Longina Przybyszewska, system programmer
IT@Naturvidenskab

IMADA, Department of Mathematics and Computer Science

University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +45 6550 2359 - http://www.imada.sdu.dk
email: longina@imada.sdu.dk
--

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Longina Przybyszewska wrote:
>>OK, I''ve *finally* managed to scrounge another box to do some 
>>testing on. I''ve setup open-iscsi in Dom0 and performance seems
to
>>be a lot better - I''m getting about 20MB/s and the backup
server
>>isn''t sat on 99% wio :-)
>
>1. I wonder what test did you make to check performance?
>I have the same concept with iscsi initiator directly on the domU 
>via bonding interface-->dedicated vlan on top of 
>bonding-->transparent bridge in Dom0-->DomU''s eth1.
>
>I would expect more advantages (performance, administration) with 
>iscsi-->DomU model.
I found that having the iSCSI initiator on DomU gave "poor" 
performance (check the start of this thread). It''s been suggested 
that the single threaded model of Dom0 networking may be to blame. 
Having the iSCSI on the DomU was my preferred setup as that would 
allow me to shut down a guest, move it''s config and storage to 
another host, start it up and it would be just the same.

Performance was tested rather crudely with :

time ( dd if=/dev/zero of=testfile bs=1024k count=2048 ; sync )

this writes a file with 2GB of zeros, chosen to be larger than memory 
available, and the sync makes sure we include the time taken to write 
what does fit in memory.
>>I assume this is because the device name can change across 
>>restarts. Since I''m not mounting the volume in Dom0, just
passing
>>it to the guest in a "disk = [ ''phy:v ..." line, any
suggestions on
>>the best way to deal with this ?
>
>2. how can you pass the storage without mounting it/having access to, in
Dom0?
Dom0 does have ''access'' since it appears as a scsi drive (eg
sdc,
with a partition sdc1). It is NOT mounted in Dom0, the device 
(/dev/sdc1, or /dev/disk/by-id/...) is given to the DomU in the 
"disk=" line. The DomU mounts it just the same as the rest of the 
volumes given to it.

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users