Christopher Chen
2009-Jan-13 22:48 UTC
[Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...
Hi there! I''ve been wrestling with an issue for a little bit now-- In my test environment, I have tgtd running on a Centos 5.2 box, with a raid 10 array backing it. The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos). Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec reads. I''ve attached the iSCSI LUN to the DomU as a virtual block device, and I''m seeing 47MB/sec writes, and 39MB/sec reads. I''ve tried a few things, like running against a local disk, and suprisingly, writes on the DomU are faster than the Dom0--can I assume the writes are buffered by the Dom0. I''m going to give a shot doing the initialization from the DomU (just for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads. This is all with bonnie++ -d /mnt -f -u root:root Anyone seen this, or have any ideas? Is any additional latency provided by the xen virtual block device causing a degradation in TCP performance (i.e. a window size or delayed ACK problem) or is the buffering also causing pain? I''m going to keep looking, but I thought I''d ask all of you. Cheers cc -- Chris Chen <muffaleta@gmail.com> "I want the kind of six pack you can''t drink." -- Micah _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Barry van Someren
2009-Jan-13 23:17 UTC
Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...
Hi there, I''ve used Citrix Xenserver 5.0 before this and never experienced this problem. I did however store the entire VM on the LUN instead of mounting it to the VM. Maybe you can trying doing that and seeing if it makes a big difference? Regards, Barry (Failed to send this to the entire list, so sending again, sorry) On Tue, Jan 13, 2009 at 11:48 PM, Christopher Chen <muffaleta@gmail.com> wrote:> Hi there! > > I''ve been wrestling with an issue for a little bit now-- > > In my test environment, I have tgtd running on a Centos 5.2 box, with > a raid 10 array backing it. > > The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland > with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos). > > Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec reads. > > I''ve attached the iSCSI LUN to the DomU as a virtual block device, and > I''m seeing 47MB/sec writes, and 39MB/sec reads. > > I''ve tried a few things, like running against a local disk, and > suprisingly, writes on the DomU are faster than the Dom0--can I assume > the writes are buffered by the Dom0. > > I''m going to give a shot doing the initialization from the DomU (just > for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads. > > This is all with bonnie++ -d /mnt -f -u root:root > > Anyone seen this, or have any ideas? > > Is any additional latency provided by the xen virtual block device > causing a degradation in TCP performance (i.e. a window size or > delayed ACK problem) or is the buffering also causing pain? I''m going > to keep looking, but I thought I''d ask all of you. > > Cheers > > cc > > -- > Chris Chen <muffaleta@gmail.com> > "I want the kind of six pack you can''t drink." > -- Micah > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Barry van Someren --------------------------------------- Email: barry@bvansomeren.com Email: goltharnl@gmail.com Linked in: http://www.linkedin.com/in/barryvansomeren _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher Chen
2009-Jan-13 23:35 UTC
Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...
Here''s my data so far:
I''m just wondering what the block device layer is doing to kill
performance. I''m looking for any buffering directives in the xend
config that I may be able to modify.
iSCSI Performance Profiling
iSCSI target on Heart, Initiator is Dom0 br04-61 as LV
Version 1.94 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
br04-61.adtech2. 8G 110289 29 33781 3 45053 1 1052 9
Latency 2610ms 312ms 191ms 291ms
Version 1.94 ------Sequential Create------ --------Random Create--------
br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 627us 157us 235us 506us 183us 282us
1.93c,1.94,br04-61,1,1231849837,8G,,,,110289,29,33781,3,,,450s
iSCSI target on Heart, Initiator on Dom0 br04-61 as LV, bonnie++ on
DomU aenu via xvd
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
aenu 4G 47100 3 15069 0 39072 0 3971 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
aenu,4G,,,47100,3,15069,0,,,39072,0,3971.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
Local Logical Volume on Dom0 br04-61, bonnie++ on Dom0
Version 1.94 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
br04-61. 8G 39000 7 21419 0 62958 0 158.0 1
Latency 16524ms 1111ms 54552us 565ms
Version 1.94 ------Sequential Create------ --------Random Create--------
br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 29249 40 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 22439us 281us 227us 589us 220us 253us
1.93c,1.94,br04-61,1,1231848192,8G,,,,39000,7,21419,0,,,62958,0,158.0,1,16,,,,,29249,40,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,16524ms,1111ms,,54552us,565ms,22439us,281us,227us,589us,220us,253us
Local Logical Volume on Dom0 br04-61, bonnie++ on DomU aenu via xvd
RAM was reduced to 512M on the DomU
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
aenu 1G 64860 5 23617 0 58329 0 212.3 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 6038 96 +++++ +++ +++++ +++ 6333 99 +++++ +++ 19371 100
aenu,1G,,,64860,5,23617,0,,,58329,0,212.3,0,16,6038,96,+++++,+++,+++++,+++,6333,99,+++++,+++,19371,10
iSCSI target on Heart, Initiator on Dom0 br04-61 as /dev/sd, bonnie++ on br04-61
Version 1.94 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
br04-61.adtech2. 8G 111523 30 34132 3 58284 3 1184 11
Latency 4579ms 250ms 72926us 44558us
Version 1.94 ------Sequential Create------ --------Random Create--------
br04-61 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 4643us 123us 191us 7013us 153us 195us
1.93c,1.94,br04-61,1,1231852980,8G,,,,111523,30,34132,3,,,58284,3,1184,11,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,4579ms,250ms,,72926us,44558us,4643us,123us,191us,7013us,153us,195us
iSCSI target on Heart, Initiator on Dom0 br04-61 as /dev/sd, bonnie++
on aenu domU as xvd
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
aenu 4G 52625 4 15065 0 46256 0 3859 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
aenu,4G,,,52625,4,15065,0,,,46256,0,3859.1,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
iSCSI target on Heart, Initiator on DomU aenu, bonnie++ on aenu DomU as /dev/sd
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
aenu 4G 129048 22 28832 2 49896 0 5174 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
aenu,4G,,,129048,22,28832,2,,,49896,0,5173.6,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
On Tue, Jan 13, 2009 at 3:17 PM, Barry van Someren
<barry@coffeesprout.com> wrote:> Hi there,
>
> I''ve used Citrix Xenserver 5.0 before this and never experienced
this problem.
> I did however store the entire VM on the LUN instead of mounting it to the
VM.
>
> Maybe you can trying doing that and seeing if it makes a big difference?
>
> Regards,
>
> Barry
>
> (Failed to send this to the entire list, so sending again, sorry)
>
> On Tue, Jan 13, 2009 at 11:48 PM, Christopher Chen
<muffaleta@gmail.com> wrote:
>> Hi there!
>>
>> I''ve been wrestling with an issue for a little bit now--
>>
>> In my test environment, I have tgtd running on a Centos 5.2 box, with
>> a raid 10 array backing it.
>>
>> The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland
>> with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos).
>>
>> Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec reads.
>>
>> I''ve attached the iSCSI LUN to the DomU as a virtual block
device, and
>> I''m seeing 47MB/sec writes, and 39MB/sec reads.
>>
>> I''ve tried a few things, like running against a local disk,
and
>> suprisingly, writes on the DomU are faster than the Dom0--can I assume
>> the writes are buffered by the Dom0.
>>
>> I''m going to give a shot doing the initialization from the
DomU (just
>> for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads.
>>
>> This is all with bonnie++ -d /mnt -f -u root:root
>>
>> Anyone seen this, or have any ideas?
>>
>> Is any additional latency provided by the xen virtual block device
>> causing a degradation in TCP performance (i.e. a window size or
>> delayed ACK problem) or is the buffering also causing pain?
I''m going
>> to keep looking, but I thought I''d ask all of you.
>>
>> Cheers
>>
>> cc
>>
>> --
>> Chris Chen <muffaleta@gmail.com>
>> "I want the kind of six pack you can''t drink."
>> -- Micah
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>>
>
>
>
> --
> Barry van Someren
> ---------------------------------------
> Email: barry@bvansomeren.com
> Email: goltharnl@gmail.com
> Linked in: http://www.linkedin.com/in/barryvansomeren
>
--
Chris Chen <muffaleta@gmail.com>
"I want the kind of six pack you can''t drink."
-- Micah
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Ross Walker
2009-Jan-13 23:37 UTC
Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...
On Jan 13, 2009, at 5:48 PM, "Christopher Chen" <muffaleta@gmail.com> wrote:> Hi there! > > I''ve been wrestling with an issue for a little bit now-- > > In my test environment, I have tgtd running on a Centos 5.2 box, with > a raid 10 array backing it. > > The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland > with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos). > > Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec reads.That''s kind of lopsided I''d expect it the other way around. Is this hardware RAID on the backend with write-back cache?> > I''ve attached the iSCSI LUN to the DomU as a virtual block device, and > I''m seeing 47MB/sec writes, and 39MB/sec reads.How did you attach it, what Xen driver did you use phy: or file:?> > I''ve tried a few things, like running against a local disk, and > suprisingly, writes on the DomU are faster than the Dom0--can I assume > the writes are buffered by the Dom0.I''m confused. I thought you said above you got 110MB/s on dom0 and 45MB/s on the domU?> > I''m going to give a shot doing the initialization from the DomU (just > for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads.You''ve completely lost me now, what do you mean initialization? Do you mean boot domU off of iSCSI directly?> > This is all with bonnie++ -d /mnt -f -u root:root > > Anyone seen this, or have any ideas? > > Is any additional latency provided by the xen virtual block device > causing a degradation in TCP performance (i.e. a window size or > delayed ACK problem) or is the buffering also causing pain? I''m going > to keep looking, but I thought I''d ask all of you.Any layer you add is going to create latency. If you can be a little more clearer I''m sure an accurate explanation can be made. -Ross _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher Chen
2009-Jan-13 23:51 UTC
Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...
On Tue, Jan 13, 2009 at 3:37 PM, Ross Walker <rswwalker@gmail.com> wrote:> On Jan 13, 2009, at 5:48 PM, "Christopher Chen" <muffaleta@gmail.com> wrote: > >> Hi there! >> >> I''ve been wrestling with an issue for a little bit now-- >> >> In my test environment, I have tgtd running on a Centos 5.2 box, with >> a raid 10 array backing it. >> >> The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland >> with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos). >> >> Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec reads. > > That''s kind of lopsided I''d expect it the other way around. > > Is this hardware RAID on the backend with write-back cache?Yes, there''s a raid 10 on the backend, with a battery backed cache.>> I''ve attached the iSCSI LUN to the DomU as a virtual block device, and >> I''m seeing 47MB/sec writes, and 39MB/sec reads. > > How did you attach it, what Xen driver did you use phy: or file:?I used phy--in fact, in all this testing I''ve been using phy. The Dom0 sees the iSCSI lun, and re-exports that block device to the DomU as a xvd.>> I''ve tried a few things, like running against a local disk, and >> suprisingly, writes on the DomU are faster than the Dom0--can I assume >> the writes are buffered by the Dom0. > > I''m confused. > > I thought you said above you got 110MB/s on dom0 and 45MB/s on the domU?Not exactly--I said I was seeing 110M/s writes, and 45M/s reads on the Dom0. I''m seeing 47M/s writes, and 39M/s reads on the DomU. One more thing I tried was cutting iSCSI out of the picture altogether, and exporting one of the Dom0''s local block devices to the DomU via phy:xvd. In that case, the write speeds were greater on the DomU side, which indicate some sort of buffering going on. This makes sense since the Dom0 has more ram than the DomU.> >> >> I''m going to give a shot doing the initialization from the DomU (just >> for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads. > > You''ve completely lost me now, what do you mean initialization? Do you mean > boot domU off of iSCSI directly?No. In this case, I''m just mounting a iSCSI lun directly in the DomU with it''s initiator. Sorry, I guess initiator != initializer. In that case, performance is pretty good, because the bridging is fast.> >> >> This is all with bonnie++ -d /mnt -f -u root:root >> >> Anyone seen this, or have any ideas? >> >> Is any additional latency provided by the xen virtual block device >> causing a degradation in TCP performance (i.e. a window size or >> delayed ACK problem) or is the buffering also causing pain? I''m going >> to keep looking, but I thought I''d ask all of you. > > Any layer you add is going to create latency. > > If you can be a little more clearer I''m sure an accurate explanation can be > made.Well, of course, I know that. What confuses me is what''s happening that that block device abstraction layer that''s killing performance. I think it might be in the buffering, which on a local disk, can probably provide some gain, but in this case, may be causing the iscsi target and initiator to throttle the TCP connection. Sorry if I''m confusing you. Cheers cc -- Chris Chen <muffaleta@gmail.com> "I want the kind of six pack you can''t drink." -- Micah _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ross Walker
2009-Jan-13 23:53 UTC
Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...
On Jan 13, 2009, at 6:37 PM, Ross Walker <rswwalker@gmail.com> wrote:> On Jan 13, 2009, at 5:48 PM, "Christopher Chen" > <muffaleta@gmail.com> wrote: > >> Hi there! >> >> I''ve been wrestling with an issue for a little bit now-- >> >> In my test environment, I have tgtd running on a Centos 5.2 box, with >> a raid 10 array backing it. >> >> The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland >> with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos). >> >> Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec >> reads. > > That''s kind of lopsided I''d expect it the other way around. > > Is this hardware RAID on the backend with write-back cache? > >> >> I''ve attached the iSCSI LUN to the DomU as a virtual block device, >> and >> I''m seeing 47MB/sec writes, and 39MB/sec reads. > > How did you attach it, what Xen driver did you use phy: or file:?Sorry, missed the virtual block device bit...>> I''ve tried a few things, like running against a local disk, and >> suprisingly, writes on the DomU are faster than the Dom0--can I >> assume >> the writes are buffered by the Dom0. > > I''m confused. > > I thought you said above you got 110MB/s on dom0 and 45MB/s on the > domU?Never mind my comment, writes are only buffered using file: io, but they are buffered in the domU''s page cache which is where you might be seeing the performance difference.>> I''m going to give a shot doing the initialization from the DomU (just >> for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads. > > You''ve completely lost me now, what do you mean initialization? Do > you mean boot domU off of iSCSI directly?After re-reading I guessed you meant you attached to the iSCSI lun after booting into the VM not as the OS disk. Again you are most likely seeing all cache affect and not the real io.>> This is all with bonnie++ -d /mnt -f -u root:root >> >> Anyone seen this, or have any ideas? >> >> Is any additional latency provided by the xen virtual block device >> causing a degradation in TCP performance (i.e. a window size or >> delayed ACK problem) or is the buffering also causing pain? I''m going >> to keep looking, but I thought I''d ask all of you. > > Any layer you add is going to create latency. > > If you can be a little more clearer I''m sure an accurate explanation > can be made.Try increasing the size of the bonnie test file to defeat the cache, say 2x the memory of the dom0 or domU or target which ever is largest. -Ross _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher Chen
2009-Jan-14 00:03 UTC
Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...
On Tue, Jan 13, 2009 at 3:53 PM, Ross Walker <rswwalker@gmail.com> wrote:> > On Jan 13, 2009, at 6:37 PM, Ross Walker <rswwalker@gmail.com> wrote: > >> On Jan 13, 2009, at 5:48 PM, "Christopher Chen" <muffaleta@gmail.com> >> wrote: >> >>> Hi there! >>> >>> I''ve been wrestling with an issue for a little bit now-- >>> >>> In my test environment, I have tgtd running on a Centos 5.2 box, with >>> a raid 10 array backing it. >>> >>> The initiators are also Centos 5.2 boxes running Xen 3.0.3 userland >>> with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos). >>> >>> Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec reads. >> >> That''s kind of lopsided I''d expect it the other way around. >> >> Is this hardware RAID on the backend with write-back cache? >> >>> >>> I''ve attached the iSCSI LUN to the DomU as a virtual block device, and >>> I''m seeing 47MB/sec writes, and 39MB/sec reads. >> >> How did you attach it, what Xen driver did you use phy: or file:? > > Sorry, missed the virtual block device bit... > >>> I''ve tried a few things, like running against a local disk, and >>> suprisingly, writes on the DomU are faster than the Dom0--can I assume >>> the writes are buffered by the Dom0. >> >> I''m confused. >> >> I thought you said above you got 110MB/s on dom0 and 45MB/s on the domU? > > Never mind my comment, writes are only buffered using file: io, but they are > buffered in the domU''s page cache which is where you might be seeing the > performance difference. > >>> I''m going to give a shot doing the initialization from the DomU (just >>> for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads. >> >> You''ve completely lost me now, what do you mean initialization? Do you >> mean boot domU off of iSCSI directly? > > After re-reading I guessed you meant you attached to the iSCSI lun after > booting into the VM not as the OS disk. > > Again you are most likely seeing all cache affect and not the real io. > >>> This is all with bonnie++ -d /mnt -f -u root:root >>> >>> Anyone seen this, or have any ideas? >>> >>> Is any additional latency provided by the xen virtual block device >>> causing a degradation in TCP performance (i.e. a window size or >>> delayed ACK problem) or is the buffering also causing pain? I''m going >>> to keep looking, but I thought I''d ask all of you. >> >> Any layer you add is going to create latency. >> >> If you can be a little more clearer I''m sure an accurate explanation can >> be made. > > Try increasing the size of the bonnie test file to defeat the cache, say 2x > the memory of the dom0 or domU or target which ever is largest.The nice thing about bonnie++ -f is it sizes the file for 2x memory. These are the numbers. In any case, the ~110MB/sec writes to the iSCSI target is our baseline number writing across the network. The Dom0 has 4G allocated to it--bonnie++''s test file is 8G. Any reading lower than that (in my mind) is degradation. I, of course, expect some effect from the layering, but 50%? cc -- Chris Chen <muffaleta@gmail.com> "I want the kind of six pack you can''t drink." -- Micah _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher Chen
2009-Jan-14 02:47 UTC
Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...
And here''s more!
I decided to try something more. I mounted the iSCSI LUN on the Dom0,
created a ext3 file system, and created a image file, 10G big. The
Dom0 has 4G of RAM. The DomU has 2G allocated to it.
So I thought, what the hell...
I block-attached the image file as file://mnt/image xvdc w to the
guest, and mkfs''d it, and mounted, it and ran bonnie to use a 8G test
size (twice the size of the Dom0''s RAM, and 4x the DomU''s
RAM). Here''s
the results:
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
aenu 8G 130195 22 27364 1 46949 0 313.6 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 6228 99 +++++ +++ +++++ +++ 6182 97 +++++ +++ 19075 100
aenu,8G,,,130195,22,27364,1,,,46949,0,313.6,0,16,6228,99,+++++,+++,+++++,+++,6182,97,+++++,+++,19075,100
Yeah, um, so, sustained IO is pretty good via file: driver, but still,
performance via the xvd driver perplexes me:
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
aenu 4G 47100 3 15069 0 39072 0 3971 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
aenu,4G,,,47100,3,15069,0,,,39072,0,3971.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++
So yeah, 36%. Pretty awesome.
I am stumped.
Cheers
cc
On Tue, Jan 13, 2009 at 4:03 PM, Christopher Chen <muffaleta@gmail.com>
wrote:> On Tue, Jan 13, 2009 at 3:53 PM, Ross Walker <rswwalker@gmail.com>
wrote:
>>
>> On Jan 13, 2009, at 6:37 PM, Ross Walker <rswwalker@gmail.com>
wrote:
>>
>>> On Jan 13, 2009, at 5:48 PM, "Christopher Chen"
<muffaleta@gmail.com>
>>> wrote:
>>>
>>>> Hi there!
>>>>
>>>> I''ve been wrestling with an issue for a little bit
now--
>>>>
>>>> In my test environment, I have tgtd running on a Centos 5.2
box, with
>>>> a raid 10 array backing it.
>>>>
>>>> The initiators are also Centos 5.2 boxes running Xen 3.0.3
userland
>>>> with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos).
>>>>
>>>> Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec
reads.
>>>
>>> That''s kind of lopsided I''d expect it the other
way around.
>>>
>>> Is this hardware RAID on the backend with write-back cache?
>>>
>>>>
>>>> I''ve attached the iSCSI LUN to the DomU as a virtual
block device, and
>>>> I''m seeing 47MB/sec writes, and 39MB/sec reads.
>>>
>>> How did you attach it, what Xen driver did you use phy: or file:?
>>
>> Sorry, missed the virtual block device bit...
>>
>>>> I''ve tried a few things, like running against a local
disk, and
>>>> suprisingly, writes on the DomU are faster than the Dom0--can I
assume
>>>> the writes are buffered by the Dom0.
>>>
>>> I''m confused.
>>>
>>> I thought you said above you got 110MB/s on dom0 and 45MB/s on the
domU?
>>
>> Never mind my comment, writes are only buffered using file: io, but
they are
>> buffered in the domU''s page cache which is where you might be
seeing the
>> performance difference.
>>
>>>> I''m going to give a shot doing the initialization from
the DomU (just
>>>> for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads.
>>>
>>> You''ve completely lost me now, what do you mean
initialization? Do you
>>> mean boot domU off of iSCSI directly?
>>
>> After re-reading I guessed you meant you attached to the iSCSI lun
after
>> booting into the VM not as the OS disk.
>>
>> Again you are most likely seeing all cache affect and not the real io.
>>
>>>> This is all with bonnie++ -d /mnt -f -u root:root
>>>>
>>>> Anyone seen this, or have any ideas?
>>>>
>>>> Is any additional latency provided by the xen virtual block
device
>>>> causing a degradation in TCP performance (i.e. a window size or
>>>> delayed ACK problem) or is the buffering also causing pain?
I''m going
>>>> to keep looking, but I thought I''d ask all of you.
>>>
>>> Any layer you add is going to create latency.
>>>
>>> If you can be a little more clearer I''m sure an accurate
explanation can
>>> be made.
>>
>> Try increasing the size of the bonnie test file to defeat the cache,
say 2x
>> the memory of the dom0 or domU or target which ever is largest.
>
> The nice thing about bonnie++ -f is it sizes the file for 2x memory.
> These are the numbers. In any case, the ~110MB/sec writes to the iSCSI
> target is our baseline number writing across the network. The Dom0 has
> 4G allocated to it--bonnie++''s test file is 8G. Any reading lower
than
> that (in my mind) is degradation. I, of course, expect some effect
> from the layering, but 50%?
>
> cc
>
> --
> Chris Chen <muffaleta@gmail.com>
> "I want the kind of six pack you can''t drink."
> -- Micah
>
--
Chris Chen <muffaleta@gmail.com>
"I want the kind of six pack you can''t drink."
-- Micah
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Ross Walker
2009-Jan-15 01:08 UTC
Re: [Xen-users] iSCSI initiator on Dom0, exported to DomU via xvd, Disk IO Drops in Half...
I''ll top post this since the thread is top heavy now... Yeah, file: io uses VFS I believe so it leverages page cache for heavy caching. The I am unsure if the vbd uses the Linux block io or a modified version of VFS that does sync writes, but if it uses VFS with sync io that''d explain the performance loss, not so much the synchronicity, but the chopping of io into 4k blocks (still page cache/memory addressed) and going sync. Run a bonnie with 4k io size and see if it is such a big difference. I''d either run iSCSI within the domU, which could be problematic down the line with Xen domain networking, or mount in dom0 and present to domU over file io and leave a chunk of memory in dom0 for page caching. -Ross On Jan 13, 2009, at 9:47 PM, "Christopher Chen" <muffaleta@gmail.com> wrote:> And here''s more! > > I decided to try something more. I mounted the iSCSI LUN on the Dom0, > created a ext3 file system, and created a image file, 10G big. The > Dom0 has 4G of RAM. The DomU has 2G allocated to it. > > So I thought, what the hell... > > I block-attached the image file as file://mnt/image xvdc w to the > guest, and mkfs''d it, and mounted, it and ran bonnie to use a 8G test > size (twice the size of the Dom0''s RAM, and 4x the DomU''s RAM). Here''s > the results: > > Version 1.03 ------Sequential Output------ --Sequential > Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec > %CP /sec %CP > aenu 8G 130195 22 27364 1 46949 0 313.6 0 > ------Sequential Create------ --------Random > Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP /sec %CP > 16 6228 99 +++++ +++ +++++ +++ 6182 97 +++++ +++ 19075 100 > aenu,8G,,,130195,22,27364,1,,,46949,0,313.6,0,16,6228,99,+++++,+++,++ > +++,+++,6182,97,+++++,+++,19075,100 > > Yeah, um, so, sustained IO is pretty good via file: driver, but still, > performance via the xvd driver perplexes me: > > Version 1.03 ------Sequential Output------ --Sequential > Input- --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec > %CP /sec %CP > aenu 4G 47100 3 15069 0 39072 0 3971 0 > ------Sequential Create------ --------Random > Create-------- > -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec > %CP /sec %CP > 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ > +++++ +++ > aenu,4G,,,47100,3,15069,0,,,39072,0,3971.4,0,16,+++++,+++,+++++,+++,+ > ++++,+++,+++++,+++,+++++,+++,+++++,+++ > > So yeah, 36%. Pretty awesome. > > I am stumped. > > Cheers > > cc > > On Tue, Jan 13, 2009 at 4:03 PM, Christopher Chen > <muffaleta@gmail.com> wrote: >> On Tue, Jan 13, 2009 at 3:53 PM, Ross Walker <rswwalker@gmail.com> >> wrote: >>> >>> On Jan 13, 2009, at 6:37 PM, Ross Walker <rswwalker@gmail.com> >>> wrote: >>> >>>> On Jan 13, 2009, at 5:48 PM, "Christopher Chen" <muffaleta@gmail.com >>>> > >>>> wrote: >>>> >>>>> Hi there! >>>>> >>>>> I''ve been wrestling with an issue for a little bit now-- >>>>> >>>>> In my test environment, I have tgtd running on a Centos 5.2 box, >>>>> with >>>>> a raid 10 array backing it. >>>>> >>>>> The initiators are also Centos 5.2 boxes running Xen 3.0.3 >>>>> userland >>>>> with a Xen 3.1.2/Linux 2.6.18 kernel (as from repos). >>>>> >>>>> Bonnie++ on the Dom0 shows about 110MB/sec writes, and 45MB/sec >>>>> reads. >>>> >>>> That''s kind of lopsided I''d expect it the other way around. >>>> >>>> Is this hardware RAID on the backend with write-back cache? >>>> >>>>> >>>>> I''ve attached the iSCSI LUN to the DomU as a virtual block >>>>> device, and >>>>> I''m seeing 47MB/sec writes, and 39MB/sec reads. >>>> >>>> How did you attach it, what Xen driver did you use phy: or file:? >>> >>> Sorry, missed the virtual block device bit... >>> >>>>> I''ve tried a few things, like running against a local disk, and >>>>> suprisingly, writes on the DomU are faster than the Dom0--can I >>>>> assume >>>>> the writes are buffered by the Dom0. >>>> >>>> I''m confused. >>>> >>>> I thought you said above you got 110MB/s on dom0 and 45MB/s on >>>> the domU? >>> >>> Never mind my comment, writes are only buffered using file: io, >>> but they are >>> buffered in the domU''s page cache which is where you might be >>> seeing the >>> performance difference. >>> >>>>> I''m going to give a shot doing the initialization from the DomU >>>>> (just >>>>> for kicks...)...and wow! 129MB/sec writes, 49MB/sec reads. >>>> >>>> You''ve completely lost me now, what do you mean initialization? >>>> Do you >>>> mean boot domU off of iSCSI directly? >>> >>> After re-reading I guessed you meant you attached to the iSCSI lun >>> after >>> booting into the VM not as the OS disk. >>> >>> Again you are most likely seeing all cache affect and not the real >>> io. >>> >>>>> This is all with bonnie++ -d /mnt -f -u root:root >>>>> >>>>> Anyone seen this, or have any ideas? >>>>> >>>>> Is any additional latency provided by the xen virtual block device >>>>> causing a degradation in TCP performance (i.e. a window size or >>>>> delayed ACK problem) or is the buffering also causing pain? I''m >>>>> going >>>>> to keep looking, but I thought I''d ask all of you. >>>> >>>> Any layer you add is going to create latency. >>>> >>>> If you can be a little more clearer I''m sure an accurate >>>> explanation can >>>> be made. >>> >>> Try increasing the size of the bonnie test file to defeat the >>> cache, say 2x >>> the memory of the dom0 or domU or target which ever is largest. >> >> The nice thing about bonnie++ -f is it sizes the file for 2x memory. >> These are the numbers. In any case, the ~110MB/sec writes to the >> iSCSI >> target is our baseline number writing across the network. The Dom0 >> has >> 4G allocated to it--bonnie++''s test file is 8G. Any reading lower >> than >> that (in my mind) is degradation. I, of course, expect some effect >> from the layering, but 50%? >> >> cc >> >> -- >> Chris Chen <muffaleta@gmail.com> >> "I want the kind of six pack you can''t drink." >> -- Micah >> > > -- > Chris Chen <muffaleta@gmail.com> > "I want the kind of six pack you can''t drink." > -- Micah_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users