Maximilian Wilhelm
2008-Jan-24 00:44 UTC
[Xen-users] What is the "fastest" way to access disk storage from DomU?
Hi! I''ve some Xen servers with local disks as DomU system and data storage and for some days one with SAN storage. I''ve read some threads about storage speed here but didn''t really got a clue on what''s the "best" or fastest way to set it up. At the moment all the virtual disks are configured as disk = [ "phy:/dev/vg_xen1/<LV>,xvda1,w", ... ] The volumes residing on the SAN storage are configured via EVMS and I''ve 200MB/s writing speed from Dom0 (mesured with dd if=/dev/zero of=/mnt/file) and "only" around 150MB/s when doing the same from DomU. Is this expected speed loss or is there any other way to give the DomU access to the devices? Ciao Max -- Follow the white penguin. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Todd Deshane
2008-Jan-24 02:27 UTC
Re: [Xen-users] What is the "fastest" way to access disk storage from DomU?
you should look into the iozone bechmark, http://www.iozone.org/ Try running on different configuration and report back your results. Regards, Todd On Jan 23, 2008 7:44 PM, Maximilian Wilhelm <max@rfc2324.org> wrote:> Hi! > > I''ve some Xen servers with local disks as DomU system and data storage > and for some days one with SAN storage. > > I''ve read some threads about storage speed here but didn''t really got > a clue on what''s the "best" or fastest way to set it up. > > At the moment all the virtual disks are configured as > > disk = [ "phy:/dev/vg_xen1/<LV>,xvda1,w", ... ] > > The volumes residing on the SAN storage are configured via EVMS and > I''ve 200MB/s writing speed from Dom0 (mesured with dd if=/dev/zero > of=/mnt/file) and "only" around 150MB/s when doing the same from DomU. > > Is this expected speed loss or is there any other way to give the DomU > access to the devices? > > Ciao > Max > -- > Follow the white penguin. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Todd Deshane
2008-Jan-24 02:38 UTC
Re: [Xen-users] What is the "fastest" way to access disk storage from DomU?
Sorry for top-posting. Re-sending to fix it. On Jan 23, 2008 7:44 PM, Maximilian Wilhelm <max@rfc2324.org> wrote:> > > Hi! > > > > I''ve some Xen servers with local disks as DomU system and data storage > > and for some days one with SAN storage. > > > > I''ve read some threads about storage speed here but didn''t really got > > a clue on what''s the "best" or fastest way to set it up. > > > > At the moment all the virtual disks are configured as > > > > disk = [ "phy:/dev/vg_xen1/<LV>,xvda1,w", ... ] > > > > The volumes residing on the SAN storage are configured via EVMS and > > I''ve 200MB/s writing speed from Dom0 (mesured with dd if=/dev/zero > > of=/mnt/file) and "only" around 150MB/s when doing the same from DomU. > > > > Is this expected speed loss or is there any other way to give the DomU > > access to the devices? > > > > Ciao > > Max > > >On Jan 23, 2008 9:27 PM, Todd Deshane <deshantm@gmail.com> wrote: You should look into the iozone bechmark, http://www.iozone.org/ Try running on different configuration and report back your results. We have actually had a few different setups some with iSCSI, AoE, LVM and non-LVM. I will see if I can get some benchmarks run on some of our setups too. Some things to watch out for are network latency in general or the load on the NAS/SAN box. Regards, Todd> -- > > Follow the white penguin. > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson
2008-Jan-24 23:18 UTC
Re: [Xen-users] What is the "fastest" way to access disk storage from DomU?
> I''ve read some threads about storage speed here but didn''t really got > a clue on what''s the "best" or fastest way to set it up. > > At the moment all the virtual disks are configured as > > disk = [ "phy:/dev/vg_xen1/<LV>,xvda1,w", ... ] > > The volumes residing on the SAN storage are configured via EVMS and > I''ve 200MB/s writing speed from Dom0 (mesured with dd if=/dev/zero > of=/mnt/file) and "only" around 150MB/s when doing the same from DomU.When you do the tests of writing speed from dom0, are you writing to the domU''s filesystem LV? Otherwise you''re not testing like-for-like since you''re using a different part of the storage. I''m not sure if this makes a difference in your case, but different parts of a physical disk can have surprisingly big differences in bandwidth (outer edge of the disk moves faster, so better bandwidth). I''m not too familiar with EVMS, maybe there''s some bottleneck there I''m not familiar with and therefore missing. Does EVMS do cluster volume management? I guess it does, as you''re using it on a SAN ;-)> Is this expected speed loss or is there any other way to give the DomU > access to the devices?You can only give domUs direct access to whole PCI devices at the moment, so unless you gave each a separate SAN adaptor, you can''t really give them any more direct access. There''s some work on SCSI passthrough being done by various people, so maybe at some point that''ll let you pass individual LUNs through from the SAN. For really high performance SAN access from domUs, the solution will eventually (one fine day, in the future) to use SAN adaptors with virtualization support that can natively give shared direct access to multiple domUs. We''re not quite there yet though! Cheers, Mark -- Push Me Pull You - Distributed SCM tool (http://www.cl.cam.ac.uk/~maw48/pmpu/) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Maximilian Wilhelm
2008-Jan-25 08:40 UTC
Re: [Xen-users] What is the "fastest" way to access disk storage from DomU?
Am Thursday, den 24 January hub Mark Williamson folgendes in die Tasten:> > I''ve read some threads about storage speed here but didn''t really got > > a clue on what''s the "best" or fastest way to set it up.> > At the moment all the virtual disks are configured as> > disk = [ "phy:/dev/vg_xen1/<LV>,xvda1,w", ... ]> > The volumes residing on the SAN storage are configured via EVMS and > > I''ve 200MB/s writing speed from Dom0 (mesured with dd if=/dev/zero > > of=/mnt/file) and "only" around 150MB/s when doing the same from DomU.> When you do the tests of writing speed from dom0, are you writing to the > domU''s filesystem LV? Otherwise you''re not testing like-for-like since > you''re using a different part of the storage. I''m not sure if this makes a > difference in your case, but different parts of a physical disk can have > surprisingly big differences in bandwidth (outer edge of the disk moves > faster, so better bandwidth).Sure I used the same EVMS volume. Anything other would have been pointless :)> I''m not too familiar with EVMS, maybe there''s some bottleneck there I''m not > familiar with and therefore missing. Does EVMS do cluster volume management? > I guess it does, as you''re using it on a SAN ;-)Paired with heartbeat (neccessary for EVMS) there is a Cluster Volume Manager plugin/module (maybe the buzzword is called different), so it''s somehow possible to have the volumes shared among hosts.> > Is this expected speed loss or is there any other way to give the DomU > > access to the devices?> You can only give domUs direct access to whole PCI devices at the moment, so > unless you gave each a separate SAN adaptor, you can''t really give them any > more direct access.> There''s some work on SCSI passthrough being done by various people, so maybe > at some point that''ll let you pass individual LUNs through from the SAN.Hmm. That would most probably not really helpful in my case as I''m not using the /dev/sd* devices I get from the SAN about 4 ways (dual-head HBA connected to SAN with two SPs) but the /dev/mapper/<foo> device handled via multipathd. OK, I could push all the according SCSI devices to the DomU and multipath inside (if possible), but it''s not a simple task to figure out which sd* belong to which LUN as far I know of. (Ok, multipath can do so, so there has to be a way...)> For really high performance SAN access from domUs, the solution will > eventually (one fine day, in the future) to use SAN adaptors with > virtualization support that can natively give shared direct access to > multiple domUs. We''re not quite there yet though!So let''s hope :) Thanks Ciao Max -- Follow the white penguin. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Todd Deshane
2008-Jan-26 05:17 UTC
Re: [Xen-users] What is the "fastest" way to access disk storage from DomU?
On Jan 25, 2008 3:40 AM, Maximilian Wilhelm <max@rfc2324.org> wrote:> Am Thursday, den 24 January hub Mark Williamson folgendes in die Tasten: > > > > I''ve read some threads about storage speed here but didn''t really got > > > a clue on what''s the "best" or fastest way to set it up. > > > > At the moment all the virtual disks are configured as > > > > disk = [ "phy:/dev/vg_xen1/<LV>,xvda1,w", ... ] > > > > The volumes residing on the SAN storage are configured via EVMS and > > > I''ve 200MB/s writing speed from Dom0 (mesured with dd if=/dev/zero > > > of=/mnt/file) and "only" around 150MB/s when doing the same from DomU. > > > When you do the tests of writing speed from dom0, are you writing to the > > domU''s filesystem LV? Otherwise you''re not testing like-for-like since > > you''re using a different part of the storage. I''m not sure if this > makes a > > difference in your case, but different parts of a physical disk can have > > surprisingly big differences in bandwidth (outer edge of the disk moves > > faster, so better bandwidth). > > Sure I used the same EVMS volume. > Anything other would have been pointless :) > > > I''m not too familiar with EVMS, maybe there''s some bottleneck there I''m > not > > familiar with and therefore missing. Does EVMS do cluster volume > management? > > I guess it does, as you''re using it on a SAN ;-) > > Paired with heartbeat (neccessary for EVMS) there is a Cluster Volume > Manager plugin/module (maybe the buzzword is called different), so > it''s somehow possible to have the volumes shared among hosts. > > > > Is this expected speed loss or is there any other way to give the DomU > > > access to the devices? > > > You can only give domUs direct access to whole PCI devices at the > moment, so > > unless you gave each a separate SAN adaptor, you can''t really give them > any > > more direct access. > > > There''s some work on SCSI passthrough being done by various people, so > maybe > > at some point that''ll let you pass individual LUNs through from the SAN. > > Hmm. > That would most probably not really helpful in my case as I''m not > using the /dev/sd* devices I get from the SAN about 4 ways (dual-head > HBA connected to SAN with two SPs) but the /dev/mapper/<foo> device > handled via multipathd. > > OK, I could push all the according SCSI devices to the DomU and > multipath inside (if possible), but it''s not a simple task to figure out > which sd* belong to which LUN as far I know of. > (Ok, multipath can do so, so there has to be a way...)I haven''t been following the details of this thread very closely, but can''t you use a udev rule or trick to do this? For example if you look in /dev/disk/by-uuid, you get symlinks to the sda* devices. And the uuid should be unique based on the LUN right?> > > > For really high performance SAN access from domUs, the solution will > > eventually (one fine day, in the future) to use SAN adaptors with > > virtualization support that can natively give shared direct access to > > multiple domUs. We''re not quite there yet though! > > So let''s hope :) > > Thanks > Ciao > Max > -- > Follow the white penguin. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Madden
2008-Jan-28 15:01 UTC
Re: [Xen-users] What is the "fastest" way to access disk storage from DomU?
On Sat, 2008-01-26 at 00:17 -0500, Todd Deshane wrote:> > OK, I could push all the according SCSI devices to the DomU > and > multipath inside (if possible), but it''s not a simple task to > figure out > which sd* belong to which LUN as far I know of. > (Ok, multipath can do so, so there has to be a way...)I''d multipath in dom0 if I were you. Otherwise, all your multipath traffic will cross the hypervisor in duplicate (one per path), which would reduce performance. Do all of your SAN work in dom0 and export a simple scsi device to your domU''s... John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users