I''m trying to characterize the performance of 10G ethernet to a user
domain.
The machine has 2 configured ethernet cards, eth0 is a 1G card, and eth2 is a
10G card. (eth1 is another 1G card and eth3 is another 10G, but these are not
configured)
I have network bridge set up on eth2 with defaults, nothing special, just:
/etc/xen/scripts/network-bridge start vifnum=2
I have a user domain that attaches eth1 to this vif, and when I try netperf
tests, I get tons of packet loss in the vif interface.
vif3.1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:165602 errors:0 dropped:0 overruns:0 frame:0
TX packets:12982847 errors:0 dropped:48987465 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:8679444 (8.2 MiB) TX bytes:20888548648 (19.4 GiB)
As can be seen above, vif3.1 drops about 80% of the packets it should transmit.
Is this expected due to something in the dom0 -> domU interface just not
being able to keep up, or is there something I can do about it? It almost looks
like the dom0->domU link is throttled at about 1Gb/s. I say this because the
10G link is running about 50-60% capacity when the drop rate is 80%.
Everything has MTU 1500, btw - I noticed that Xen is pretty unhappy with an MTU
of 9000.
thanks,
-reese
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
sorry - missing some possibly useful info: 2.6.18.5 with the fedora patch, Xen 3.0.3, no other mods. dual 3GHz Xeon. 2 machines connected back-to-back, no ethernet switch. I''m trying to characterize the performance of 10G ethernet to a user domain. The machine has 2 configured ethernet cards, eth0 is a 1G card, and eth2 is a 10G card. (eth1 is another 1G card and eth3 is another 10G, but these are not configured) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I''m trying to characterize the performance of 10G ethernet to a user > domain. > The machine has 2 configured ethernet cards, eth0 is a 1G card, andeth2 is> a 10G card. (eth1 is another 1G card and eth3 is another 10G, butthese> are not configured) > I have network bridge set up on eth2 with defaults, nothing special,just:> /etc/xen/scripts/network-bridge start vifnum=2 > I have a user domain that attaches eth1 to this vif, and when I trynetperf> tests, I get tons of packet loss in the vif interface. > > As can be seen above, vif3.1 drops about 80% of the packets it should > transmit. Is this expected due to something in the dom0 -> domUinterface> just not being able to keep up, or is there something I can do aboutit?> It almost looks like the dom0->domU link is throttled at about 1Gb/s.I> say this because the 10G link is running about 50-60% capacity whenthe> drop rate is 80%.First off, try switching to 3.0.4 and use the included 2.6.16-xen kernel. How are you measuring the bandwidth? It''s much more realistic to use TCP as open-loop UDP blasts aren''t used by any real protocols.> Everything has MTU 1500, btw - I noticed that Xen is pretty unhappywith an> MTU of 9000.In what way? I''d expect larger MTUs to work OK, though I haven''t tested them myself. Does the larger MTU work in dom0, but not through to other domU''s? You may need to set the MTU on the bridge and all vif''s. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Santos, Jose Renato G
2007-Feb-06 18:12 UTC
RE: [Xen-devel] vif interface dropping 80% packets
Most likely your dom0 CPU is saturated.. It is unrealistic to expect full throughput of a 10 Gig NIC. I would be surprised if even linux could keep up with a receive rate of 10 Gb/s. Did you try the same experiment on native linux? In my experiments, on a 4-way 2.8 Ghz Xeon, dom0 consumes ~75% of one cpu for processing receive packets while the guest consume ~55% of another CPU. It is expected that your dual CPU system saturates at a rate slighlty above 1 Ghz. (You are also consuming extra cycles to drop packets) You could confirm that the CPU is saturating by running xentop while running your experiments. You could avoid dropping packets and get a more realistic experiment by running a TCP experiment as Ian suggested Regards Renato ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Reese Faucette Sent: Monday, February 05, 2007 4:31 PM To: xen-devel@lists.xensource.com Subject: [Xen-devel] vif interface dropping 80% packets I''m trying to characterize the performance of 10G ethernet to a user domain. The machine has 2 configured ethernet cards, eth0 is a 1G card, and eth2 is a 10G card. (eth1 is another 1G card and eth3 is another 10G, but these are not configured) I have network bridge set up on eth2 with defaults, nothing special, just: /etc/xen/scripts/network-bridge start vifnum=2 I have a user domain that attaches eth1 to this vif, and when I try netperf tests, I get tons of packet loss in the vif interface. vif3.1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:165602 errors:0 dropped:0 overruns:0 frame:0 TX packets:12982847 errors:0 dropped:48987465 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:8679444 (8.2 MiB) TX bytes:20888548648 (19.4 GiB) As can be seen above, vif3.1 drops about 80% of the packets it should transmit. Is this expected due to something in the dom0 -> domU interface just not being able to keep up, or is there something I can do about it? It almost looks like the dom0->domU link is throttled at about 1Gb/s. I say this because the 10G link is running about 50-60% capacity when the drop rate is 80%. Everything has MTU 1500, btw - I noticed that Xen is pretty unhappy with an MTU of 9000. thanks, -reese _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Most likely your dom0 CPU is saturated.. It is unrealistic to expect full throughput of a 10 Gig NIC. I would be surprised if even linux could keep up with a receive rate of 10 Gb/s. Did you try the same experiment on native linux? In my experiments, on a 4-way 2.8 Ghz Xeon, dom0 consumes ~75% of one cpu for processing receive packets while the guest consume ~55% of another CPU. With native->native or even native->dom0, I get 9.88 Gb/s with 21% utilization of receive side CPU. It''s a nice machine and we make nice NICs ;-) So, it''s not unrealistic. It is expected that your dual CPU system saturates at a rate slighlty above 1 Ghz. (You are also consuming extra cycles to drop packets) You could confirm that the CPU is saturating by running xentop while running your experiments. You could avoid dropping packets and get a more realistic experiment by running a TCP experiment as Ian suggested i have a pending note on that, TCP test is actually what I tried first, and the perf was terrible due to the same packet loss. more info to come. -reese _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Santos, Jose Renato G
2007-Feb-06 19:40 UTC
RE: [Xen-devel] vif interface dropping 80% packets
You are probably using jumbo packets for native, right? What is the MTU when you get 9.88 Gb/s? I was thinking MTU=1500 when I said unrealistic. Not sure why you cannot get higher MTU in Xen to work. I have not tried MTU > 1500 bytes in Xen myself but I believe Herbert Xu has done that successfully. If you get larger MTU to work, keep in mind that in Xen there is an extra data copy to transfer the packet from dom0 to guest which will have a higher impact on performance for larger MTUs when compared to linux. Renato ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Reese Faucette Sent: Tuesday, February 06, 2007 10:59 AM To: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] vif interface dropping 80% packets Most likely your dom0 CPU is saturated.. It is unrealistic to expect full throughput of a 10 Gig NIC. I would be surprised if even linux could keep up with a receive rate of 10 Gb/s. Did you try the same experiment on native linux? In my experiments, on a 4-way 2.8 Ghz Xeon, dom0 consumes ~75% of one cpu for processing receive packets while the guest consume ~55% of another CPU. With native->native or even native->dom0, I get 9.88 Gb/s with 21% utilization of receive side CPU. It''s a nice machine and we make nice NICs ;-) So, it''s not unrealistic. It is expected that your dual CPU system saturates at a rate slighlty above 1 Ghz. (You are also consuming extra cycles to drop packets) You could confirm that the CPU is saturating by running xentop while running your experiments. You could avoid dropping packets and get a more realistic experiment by running a TCP experiment as Ian suggested i have a pending note on that, TCP test is actually what I tried first, and the perf was terrible due to the same packet loss. more info to come. -reese _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> You are probably using jumbo packets for native, right? What is the MTU > when you get > 9.88 Gb/s? I was thinking MTU=1500 when I said unrealistic. Not sure why > you cannot > get higher MTU in Xen to work. I have not tried MTU > 1500 bytes in Xen > myself but > I believe Herbert Xu has done that successfully.ok, true - got me there. TCP test with native linux sending to dom0 and 1500 MTU is 9.1GB/s and 23% of 1 CPU used on receiver.> If you get larger MTU to work, keep in mind that in Xen there is an extra > data copy > to transfer the packet from dom0 to guest which will have a higher impact > on performance > for larger MTUs when compared to linux.of course - i don''t expect there to be no impact, but the numbers i was getting with mtu=1500 were horrid, like 70Mb/s using netperf with TCP_SENDFILE. a slower sender actually resulted in better throughput due to fewer dropped packets. (e.g. using TCP_STREAM got 220Mb/s, better but still stinky) update: MTU=9000 does work, I had failed to set one of the vif interfaces - plenty of interfaces to change ;-) The performance is better with 9k MTU, as expected: TCP_STREAM and TCP_SENDFILE both get ~1.8Gb/s, xentop reports 87% utilization for dom0, so I guess that''s about all I can expect without code changes. If I understand the Xen bridge network flow, though, it''s a real bummer that> First off, try switching to 3.0.4 and use the included 2.6.16-xen kernel.I can try, but the issue i had before was that 2.6.16 did not have the ethernet driver I need for the main ethernet connection: 0000:06:00.1 Ethernet controller: Intel Corporation Enterprise Southbridge DPT LAN Copper (rev 01) It comes with 2.6.18, but not 2.6.16, and the driver does not compile under 2.6.16, so i was stuck with 2.6.18. before I spend a lot of time converting to 3.0.4, a couple of questions: 1) any reason to expect better perf on 3.0.4 ? 2) if yes, Will the patch "fedora-36252.patch" work on the 2.6.16 kernel included with 3.0.4, or have conflicts crept in? thanks! -reese _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> update: > MTU=9000 does work, I had failed to set one of the vif interfaces -plenty> of interfaces to change ;-) > The performance is better with 9k MTU, as expected: > TCP_STREAM and TCP_SENDFILE both get ~1.8Gb/s, xentop reports 87% > utilization for dom0, so I guess that''s about all I can expect withoutcode> changes.I''ve seen rather better figures than this with another 10G card. Does your NIC/driver support TSO and LRO?> before I spend a lot of time converting to 3.0.4, a couple ofquestions:> 1) any reason to expect better perf on 3.0.4 ?Yes, there were changes in the RX path.> 2) if yes, Will the patch "fedora-36252.patch" work on the 2.6.16kernel> included with 3.0.4, or have conflicts crept in?No idea. What does the patch do? Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>> update: >> The performance is better with 9k MTU, as expected: >> TCP_STREAM and TCP_SENDFILE both get ~1.8Gb/s, xentop reports 87% >> utilization for dom0, so I guess that''s about all I can expect without >> code changes. > > I''ve seen rather better figures than this with another 10G card. > Does your NIC/driver support TSO and LRO?yes to both, and have tried with LRO both enabled and disabled. again, the dom0 performance is fine, it''s just to domU that is slower. I have network bridging set up on this interface, i.e. peth2, eth2, xenbr2, etc. - is that not the way I should be doing it?>> 2) if yes, Will the patch "fedora-36252.patch" work on the 2.6.16 >> kernel included with 3.0.4, or have conflicts crept in? > > No idea. What does the patch do?This was the mythical patch to make 2.6.18 into a Xen-aware kernel. Took me forever to track it down - is there a more obvious place to look? Sorry to just refer to the file by name, after googling forever i decided i was the only person in the Xen world not intimately familiar with this patch. -reese _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > I''ve seen rather better figures than this with another 10G card. > > Does your NIC/driver support TSO and LRO? > > yes to both, and have tried with LRO both enabled and disabled.again, the> dom0 performance is fine, it''s just to domU that is slower. I havenetwork> bridging set up on this interface, i.e. peth2, eth2, xenbr2, etc. - isthat> not the way I should be doing it?I presume this is a box with >1 core? Try running dom0 and the domU with one VCPU to keep things simple. The scheduler should put them on different physical CPUs. The packets are getting dropped being passed from the bridge into the domU, because the domU isn''t keeping up and creating slots in the ring buffer. For a TCP connection, this is surprising as you''d have to have a big socket buffer to saturate the ring buffers. Switching to 3.0.4 or unstable will certainly help. You could try putting netback.queue_length=256 on the module or dom0 kernel command line, but a better fix would be to increase the ring buffer size.> >> 2) if yes, Will the patch "fedora-36252.patch" work on the 2.6.16 > >> kernel included with 3.0.4, or have conflicts crept in? > > > > No idea. What does the patch do? > > This was the mythical patch to make 2.6.18 into a Xen-aware kernel.Took> me > forever to track it down - is there a more obvious place to look?Sorry to> just refer to the file by name, after googling forever i decided i wasthe> only person in the Xen world not intimately familiar with this patch.3.0.4 and xen-unstable currently come with a patched linux kernel src for convenience (though the intention is to move this out of tree right after 3.0.5 is released). Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Santos, Jose Renato G
2007-Feb-07 00:16 UTC
RE: [Xen-devel] vif interface dropping 80% packets
> This was the mythical patch to make 2.6.18 into a Xen-aware > kernel. Took me forever to track it down - is there a more > obvious place to look? Sorry to just refer to the file by > name, after googling forever i decided i was the only person > in the Xen world not intimately familiar with this patch. > -reese >You should consider trying xen-unstable which currently includes linux 2.6.18. Renato> > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Indeed, xen-unstable is MUCH better - TCP_SENDFILE looks more like 6.5Gb/s on my first try from native to domU with no fiddling. The fact that xen-unstable is based on 2.6.18 is a big help, also. Ian, is this more in line with what you have seen before? thanks for the pointers, -reese. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Indeed, xen-unstable is MUCH better - TCP_SENDFILE looks more like6.5Gb/s> on my first try from native to domU with no fiddling. > The fact that xen-unstable is based on 2.6.18 is a big help, also. > > Ian, is this more in line with what you have seen before?That''s more like it. If you work with us and do some profiling etc we can probably help you improve it some more. If you want really good network performance in domU''s you should write a driver that gives guests direct access to a virtual interface on the card in a safe and protected manner. The solarflare guys have got some excellent results by doing this, and have made their design and patches available. We''re hoping to get something based on their proposed architecture into mainline Xen, and are looking for interested parties to help... Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel