<ryan@ryanholt.net>
2009-Dec-05 17:40 UTC
[Xen-devel] Poor NIC performance on domUs / Crashing
Hi All, Setup is as follows: Hardware: AMD Sempron 2800+ (No VT) 3GB RAM 8x250GB IDE Drives in RAID5 Software Xen-3.4.1 dom0: Gentoo built with Sources 2.6.29-r4 domU: Gentoo built with Sources 2.6.29-r4 dom0 is running as my firewall while domU is running mail/web services. Here''s my dd tests on both the dom0 and domU: dom0: fs ~ # time dd if=/dev/sda of=/dev/null bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB) copied, 4.71219 s, 111 MB/s real 0m4.718s user 0m0.000s sys 0m1.844s fs ~ # time hdparm -t /dev/sda /dev/sda: Timing buffered disk reads: 598 MB in 3.19 seconds = 187.22 MB/sec real 0m6.598s user 0m0.131s sys 0m2.059s domU: mail ~ # time dd if=/dev/xvda of=/dev/null bs=1M count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB) copied, 9.39719 s, 55.8 MB/s real 0m9.404s user 0m0.000s sys 0m1.317s mail ~ # time hdparm -t /dev/xvda /dev/xvda: Timing buffered disk reads: 568 MB in 3.00 seconds = 189.19 MB/sec real 0m6.161s user 0m0.121s sys 0m1.789s Here''s my config file for the domU: fs ~ # cat /etc/xen/mail # general name = "mail"; memory = 1024; # booting kernel = "/mnt/data/vms/xen/kernels/vmlinuz-2.6.29-r4-xenU"; # virtual harddisk disk = [ "file:/mnt/data/vms/mail/domU-mail,xvda,w" ]; root = "/dev/xvda ro"; # virtual network vif = [ "ip=10.9.18.13, vifname=eth3" ]; #vif = [ "" ]; #dhcp = "dhcp"; # If you can''t see anything from init in your "xm console" command try uncommenting this line. extra = ''xencons=tty'' When I run rsync from a network host into the domU my pings to the domU go to 1000ms and my write speed is around 1MB/s. I''ve already set the ethtool -K eth0 tx off on the domU: mail ~ # ethtool -k eth0 Offload parameters for eth0: Cannot get device rx csum settings: Operation not supported rx-checksumming: off tx-checksumming: off scatter-gather: off tcp segmentation offload: off udp fragmentation offload: off generic segmentation offload: off Also, I''ve had the whole kernel crash on the dom0 when doing moderate network activity to the domU. Got some message about insufficient memory on the eth0 interface (WAN facing on the dom0). Any ideas as to how to troubleshoot this? Thanks! Ryan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Dec-11 18:24 UTC
Re: [Xen-devel] Poor NIC performance on domUs / Crashing
On 12/05/09 09:40, ryan@ryanholt.net wrote:> Hi All, > > Setup is as follows: > > Hardware: > > AMD Sempron 2800+ (No VT) > 3GB RAM > 8x250GB IDE Drives in RAID5 > > Software > Xen-3.4.1 > > dom0: Gentoo built with Sources 2.6.29-r4 > domU: Gentoo built with Sources 2.6.29-r4 > > dom0 is running as my firewall while domU is running mail/web services. > Here''s my dd tests on both the dom0 and domU: >Do you know what source base these kernels are based on? Are they pvops? Novell forward-port? In particular, what''s the origin of the 2.6.29 dom0? J> > dom0: > > fs ~ # time dd if=/dev/sda of=/dev/null bs=1M count=500 > 500+0 records in > 500+0 records out > 524288000 bytes (524 MB) copied, 4.71219 s, 111 MB/s > > real 0m4.718s > user 0m0.000s > sys 0m1.844s > > fs ~ # time hdparm -t /dev/sda > > /dev/sda: > Timing buffered disk reads: 598 MB in 3.19 seconds = 187.22 MB/sec > > real 0m6.598s > user 0m0.131s > sys 0m2.059s > > > domU: > > mail ~ # time dd if=/dev/xvda of=/dev/null bs=1M count=500 > 500+0 records in > 500+0 records out > 524288000 bytes (524 MB) copied, 9.39719 s, 55.8 MB/s > > real 0m9.404s > user 0m0.000s > sys 0m1.317s > > mail ~ # time hdparm -t /dev/xvda > > /dev/xvda: > Timing buffered disk reads: 568 MB in 3.00 seconds = 189.19 MB/sec > > real 0m6.161s > user 0m0.121s > sys 0m1.789s > > Here''s my config file for the domU: > > fs ~ # cat /etc/xen/mail > # general > name = "mail"; > memory = 1024; > > # booting > kernel = "/mnt/data/vms/xen/kernels/vmlinuz-2.6.29-r4-xenU"; > > # virtual harddisk > disk = [ "file:/mnt/data/vms/mail/domU-mail,xvda,w" ]; > root = "/dev/xvda ro"; > > # virtual network > vif = [ "ip=10.9.18.13, vifname=eth3" ]; > #vif = [ "" ]; > #dhcp = "dhcp"; > > # If you can''t see anything from init in your "xm console" command try > uncommenting this line. > extra = ''xencons=tty'' > > > When I run rsync from a network host into the domU my pings to the domU go > to 1000ms and my write speed is around 1MB/s. I''ve already set the ethtool > -K eth0 tx off on the domU: > > mail ~ # ethtool -k eth0 > Offload parameters for eth0: > Cannot get device rx csum settings: Operation not supported > rx-checksumming: off > tx-checksumming: off > scatter-gather: off > tcp segmentation offload: off > udp fragmentation offload: off > generic segmentation offload: off > > > Also, I''ve had the whole kernel crash on the dom0 when doing moderate > network activity to the domU. Got some message about insufficient memory on > the eth0 interface (WAN facing on the dom0). > > Any ideas as to how to troubleshoot this? > > Thanks! > Ryan > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew Lyon
2009-Dec-11 19:17 UTC
Re: [Xen-devel] Poor NIC performance on domUs / Crashing
On Fri, Dec 11, 2009 at 6:24 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:> On 12/05/09 09:40, ryan@ryanholt.net wrote: >> >> Hi All, >> >> Setup is as follows: >> >> Hardware: >> >> AMD Sempron 2800+ (No VT) >> 3GB RAM >> 8x250GB IDE Drives in RAID5 >> >> Software >> Xen-3.4.1 >> >> dom0: Gentoo built with Sources 2.6.29-r4 >> domU: Gentoo built with Sources 2.6.29-r4 >> >> dom0 is running as my firewall while domU is running mail/web services. >> Here''s my dd tests on both the dom0 and domU: >> > > Do you know what source base these kernels are based on? Are they pvops? > Novell forward-port? In particular, what''s the origin of the 2.6.29 dom0?>From the description I believe they are the rebased novell forwardported patches which I rebased to apply to vanilla without other novell/suse patches and made available for download at http://code.google.com/p/gentoo-xen-kernel/downloads/list> > J > >> >> dom0: >> >> fs ~ # time dd if=/dev/sda of=/dev/null bs=1M count=500 >> 500+0 records in >> 500+0 records out >> 524288000 bytes (524 MB) copied, 4.71219 s, 111 MB/s >> >> real 0m4.718s >> user 0m0.000s >> sys 0m1.844s >> >> fs ~ # time hdparm -t /dev/sda >> >> /dev/sda: >> Timing buffered disk reads: 598 MB in 3.19 seconds = 187.22 MB/sec >> >> real 0m6.598s >> user 0m0.131s >> sys 0m2.059s >> >> >> domU: >> >> mail ~ # time dd if=/dev/xvda of=/dev/null bs=1M count=500 >> 500+0 records in >> 500+0 records out >> 524288000 bytes (524 MB) copied, 9.39719 s, 55.8 MB/s >> >> real 0m9.404s >> user 0m0.000s >> sys 0m1.317s >> >> mail ~ # time hdparm -t /dev/xvda >> >> /dev/xvda: >> Timing buffered disk reads: 568 MB in 3.00 seconds = 189.19 MB/sec >> >> real 0m6.161s >> user 0m0.121s >> sys 0m1.789s >> >> Here''s my config file for the domU: >> >> fs ~ # cat /etc/xen/mail >> # general >> name = "mail"; >> memory = 1024; >> >> # booting >> kernel = "/mnt/data/vms/xen/kernels/vmlinuz-2.6.29-r4-xenU"; >> >> # virtual harddisk >> disk = [ "file:/mnt/data/vms/mail/domU-mail,xvda,w" ]; >> root = "/dev/xvda ro"; >> >> # virtual network >> vif = [ "ip=10.9.18.13, vifname=eth3" ]; >> #vif = [ "" ]; >> #dhcp = "dhcp"; >> >> # If you can''t see anything from init in your "xm console" command try >> uncommenting this line. >> extra = ''xencons=tty'' >> >> >> When I run rsync from a network host into the domU my pings to the domU go >> to 1000ms and my write speed is around 1MB/s. I''ve already set the ethtool >> -K eth0 tx off on the domU: >> >> mail ~ # ethtool -k eth0 >> Offload parameters for eth0: >> Cannot get device rx csum settings: Operation not supported >> rx-checksumming: off >> tx-checksumming: off >> scatter-gather: off >> tcp segmentation offload: off >> udp fragmentation offload: off >> generic segmentation offload: off >> >> >> Also, I''ve had the whole kernel crash on the dom0 when doing moderate >> network activity to the domU. Got some message about insufficient memory >> on >> the eth0 interface (WAN facing on the dom0). >> >> Any ideas as to how to troubleshoot this? >> >> Thanks! >> Ryan >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> >> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Andrew, Jeremy, Andrew is right. I used the sources listed there. The domU seems to run fine so long as I don''t push any high amount of data to it. As soon as I start an FTP transfer the ping milliseconds jumps up to 1000ms. Thanks, Ryan -----Original Message----- From: Andrew Lyon [mailto:andrew.lyon@gmail.com] Sent: Friday, December 11, 2009 2:17 PM To: Jeremy Fitzhardinge Cc: ryan@ryanholt.net; xen-devel@lists.xensource.com Subject: Re: [Xen-devel] Poor NIC performance on domUs / Crashing On Fri, Dec 11, 2009 at 6:24 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:> On 12/05/09 09:40, ryan@ryanholt.net wrote: >> >> Hi All, >> >> Setup is as follows: >> >> Hardware: >> >> AMD Sempron 2800+ (No VT) >> 3GB RAM >> 8x250GB IDE Drives in RAID5 >> >> Software >> Xen-3.4.1 >> >> dom0: Gentoo built with Sources 2.6.29-r4 >> domU: Gentoo built with Sources 2.6.29-r4 >> >> dom0 is running as my firewall while domU is running mail/web services. >> Here''s my dd tests on both the dom0 and domU: >> > > Do you know what source base these kernels are based on? Are they pvops? > Novell forward-port? In particular, what''s the origin of the 2.6.29dom0?>From the description I believe they are the rebased novell forwardported patches which I rebased to apply to vanilla without other novell/suse patches and made available for download at http://code.google.com/p/gentoo-xen-kernel/downloads/list> > J > >> >> dom0: >> >> fs ~ # time dd if=/dev/sda of=/dev/null bs=1M count=500 >> 500+0 records in >> 500+0 records out >> 524288000 bytes (524 MB) copied, 4.71219 s, 111 MB/s >> >> real 0m4.718s >> user 0m0.000s >> sys 0m1.844s >> >> fs ~ # time hdparm -t /dev/sda >> >> /dev/sda: >> Timing buffered disk reads: 598 MB in 3.19 seconds = 187.22 MB/sec >> >> real 0m6.598s >> user 0m0.131s >> sys 0m2.059s >> >> >> domU: >> >> mail ~ # time dd if=/dev/xvda of=/dev/null bs=1M count=500 >> 500+0 records in >> 500+0 records out >> 524288000 bytes (524 MB) copied, 9.39719 s, 55.8 MB/s >> >> real 0m9.404s >> user 0m0.000s >> sys 0m1.317s >> >> mail ~ # time hdparm -t /dev/xvda >> >> /dev/xvda: >> Timing buffered disk reads: 568 MB in 3.00 seconds = 189.19 MB/sec >> >> real 0m6.161s >> user 0m0.121s >> sys 0m1.789s >> >> Here''s my config file for the domU: >> >> fs ~ # cat /etc/xen/mail >> # general >> name = "mail"; >> memory = 1024; >> >> # booting >> kernel = "/mnt/data/vms/xen/kernels/vmlinuz-2.6.29-r4-xenU"; >> >> # virtual harddisk >> disk = [ "file:/mnt/data/vms/mail/domU-mail,xvda,w" ]; >> root = "/dev/xvda ro"; >> >> # virtual network >> vif = [ "ip=10.9.18.13, vifname=eth3" ]; >> #vif = [ "" ]; >> #dhcp = "dhcp"; >> >> # If you can''t see anything from init in your "xm console" command try >> uncommenting this line. >> extra = ''xencons=tty'' >> >> >> When I run rsync from a network host into the domU my pings to the domUgo>> to 1000ms and my write speed is around 1MB/s. I''ve already set theethtool>> -K eth0 tx off on the domU: >> >> mail ~ # ethtool -k eth0 >> Offload parameters for eth0: >> Cannot get device rx csum settings: Operation not supported >> rx-checksumming: off >> tx-checksumming: off >> scatter-gather: off >> tcp segmentation offload: off >> udp fragmentation offload: off >> generic segmentation offload: off >> >> >> Also, I''ve had the whole kernel crash on the dom0 when doing moderate >> network activity to the domU. Got some message about insufficient memory >> on >> the eth0 interface (WAN facing on the dom0). >> >> Any ideas as to how to troubleshoot this? >> >> Thanks! >> Ryan >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> >> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Dec-11 21:51 UTC
Re: [Xen-devel] Poor NIC performance on domUs / Crashing
On 12/11/2009 11:20 AM, Ryan Holt wrote:> Andrew is right. I used the sources listed there. > > The domU seems to run fine so long as I don''t push any high amount of data > to it. As soon as I start an FTP transfer the ping milliseconds jumps up to > 1000ms. >It sounds rather like a bug that was fixed a long time ago in 2.6.18-xen and more recently in pvops dom0. But you''ll need to talk to Gentoo to get support for those kernels; they are many steps removed from the Xen project. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
There another kernel source you''d recommend using? Sent from my iPhone On Dec 11, 2009, at 4:51 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:> On 12/11/2009 11:20 AM, Ryan Holt wrote: >> Andrew is right. I used the sources listed there. >> >> The domU seems to run fine so long as I don''t push any high amount >> of data >> to it. As soon as I start an FTP transfer the ping milliseconds >> jumps up to >> 1000ms. >> > > It sounds rather like a bug that was fixed a long time ago in 2.6.18- > xen and more recently in pvops dom0. But you''ll need to talk to > Gentoo to get support for those kernels; they are many steps removed > from the Xen project. > > J_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Dec-11 23:22 UTC
Re: [Xen-devel] Poor NIC performance on domUs / Crashing
On 12/11/2009 03:21 PM, Ryan Holt wrote:> There another kernel source you''d recommend using?http://wiki.xensource.com/xenwiki/XenParavirtOps J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
So is this an issue with the dom0 or the domU? Jeremy Fitzhardinge wrote:> On 12/11/2009 03:21 PM, Ryan Holt wrote: >> There another kernel source you''d recommend using? > > http://wiki.xensource.com/xenwiki/XenParavirtOps > > J_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Dec-12 00:23 UTC
Re: [Xen-devel] Poor NIC performance on domUs / Crashing
On 12/11/2009 04:13 PM, Ryan Holt wrote:> So is this an issue with the dom0 or the domU? >The bug I''m referring to was in netback in dom0, where it did the wrong thing with gso packets. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks Jeremy. I installed xen-sources-2.6.18-r12 as my dom0 and the issue seems to have gone away. Will do some more thorough throughput testing on it later. I''m using gentoo-sources-2.6.31-r6 for the domU. Question; what''s the typical type of performance decrease I should expect to the domU over the network? Getting around 12MB/sec to the dom0, and anywhere from 5-10MB/sec to the domU; that typical/expected? Thanks, Ryan Jeremy Fitzhardinge wrote:> On 12/11/2009 04:13 PM, Ryan Holt wrote: >> So is this an issue with the dom0 or the domU? >> > The bug I''m referring to was in netback in dom0, where it did the > wrong thing with gso packets. > > J_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Dec-13 19:11 UTC
Re: [Xen-devel] Poor NIC performance on domUs / Crashing
On 12/12/09 09:10, Ryan Holt wrote:> I installed xen-sources-2.6.18-r12 as my dom0 and the issue seems to > have gone away. Will do some more thorough throughput testing on it > later. I''m using gentoo-sources-2.6.31-r6 for the domU. > > Question; what''s the typical type of performance decrease I should > expect to the domU over the network? Getting around 12MB/sec to the > dom0, and anywhere from 5-10MB/sec to the domU; that typical/expected? >That sounds reasonable to me, but I haven''t done much performance-oriented testing so I don''t know whether that''s really expected. There are a lot of variables in there.. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel