Sengul Thomas
2013-Apr-23 05:04 UTC
arm: network throughput decreases 5Mbps (Arndale Exynos5250)
Hello, While I was doing some tests in Arndale board, the network (iperf) throughput suddenly decreases to 5Mbps (from 100Mbps) out of nowhere. This typically happens when I generate traffic using iperf (100Mbps) while installing some packages using apt-get in dom0. Whenever this behavior happens, I can not see any distinguishable logs from any places. After some debugging, I found that usb_hcd_irq handler (linux/drivers/usb/core/hcd.c) is not called and the pending requests are only handled by periodic timer (100ms). That is the reason of the sharp throughput reduction. I also checked that the flag for usb_hcd_irq is not disabled in Linux, and saw no reason for not calling usb_hcd_irq. This is quite a odd behavior since I can not see this happening when I run native linux on top of Arndale, so I guess it is somehow related to xen (maybe vgic delivery). I would like to know if there is any tracing or debugging helpers (for vgic and evt_chn) in Xen or Linux to see further what triggers this behavior. I''m a bit tired of using printk all over the places :) Here goes the branch that I''m using: xen: git://xenbits.xen.org/people/julieng/xen-unstable.git Dom0: git://xenbits.xen.org/people/julieng/linux-arm.git Sincerely, Thomas
Ian Campbell
2013-Apr-23 09:03 UTC
Re: arm: network throughput decreases 5Mbps (Arndale Exynos5250)
On Tue, 2013-04-23 at 06:04 +0100, Sengul Thomas wrote:> Hello, > > While I was doing some tests in Arndale board, the network (iperf) > throughput suddenly decreases to 5Mbps (from 100Mbps) out of nowhere. > > This typically happens when I generate traffic using iperf (100Mbps) > while installing some packages using apt-get in dom0. > Whenever this behavior happens, I can not see any distinguishable logs > from any places.How strange. Do you get any packet loss or just slow delivery of the packets which are flowing? This is traffic from the dom0 to the outside world, rather than between virtual machines on the same host, right?> After some debugging, I found that usb_hcd_irq handler > (linux/drivers/usb/core/hcd.c) > is not called and the pending requests are only handled by periodic > timer (100ms). > That is the reason of the sharp throughput reduction. I also checked > that the flag for > usb_hcd_irq is not disabled in Linux, and saw no reason for not > calling usb_hcd_irq.So is the Arndale using a USB Ethernet device?> This is quite a odd behavior since I can not see this happening when I > run native linux > on top of Arndale, so I guess it is somehow related to xen (maybe vgic > delivery). > > I would like to know if there is any tracing or debugging helpers (for > vgic and evt_chn)If this is traffic with the outside world over a physical NIC (USB or otherwise) then vgic is the one to concentrate on, evtchn would be for virtual devices only.> in Xen or Linux to see further what triggers this behavior. > I''m a bit tired of using printk all over the places :)If you press the Ctrl-A key three times (assuming you didn''t change it with the consswitch hypervisor command line option) then you should get some info about the vgic''s state from the ''q'' debug key. It would also be useful to examine the physical interrupt to be sure it was arriving at the hypervisor. On x86 the ''i'' key dumps information about the physical interrupts (see xen/arch/x86/irq.c:dump_irqs_keyhandler) but AFAIK there is no equivalent on ARM, apart from a little bit of hardware state printed by the ''q'' key. If you wanted to code up a more comprehensive dump of the physical GIC state and wire it up to ''i; that would be very useful here as well as in the future I imagine. Thanks, Ian.
Sengul Thomas
2013-Apr-23 11:04 UTC
Re: arm: network throughput decreases 5Mbps (Arndale Exynos5250)
>> This typically happens when I generate traffic using iperf (100Mbps) >> while installing some packages using apt-get in dom0. >> Whenever this behavior happens, I can not see any distinguishable logs >> from any places. > > How strange. Do you get any packet loss or just slow delivery of the > packets which are flowing?Just slowing down packet flows, no packet losses. (If network load is moderate)> > This is traffic from the dom0 to the outside world, rather than between > virtual machines on the same host, right?Yes indeed. Actually, even this happens, network throughput between virtual machines are very fast (I vaguely remember it was larger than 1Gbps).> So is the Arndale using a USB Ethernet device? >Right.> > It would also be useful to examine the physical interrupt to be sure it > was arriving at the hypervisor. On x86 the ''i'' key dumps information > about the physical interrupts (see > xen/arch/x86/irq.c:dump_irqs_keyhandler) but AFAIK there is no > equivalent on ARM, apart from a little bit of hardware state printed by > the ''q'' key. If you wanted to code up a more comprehensive dump of the > physical GIC state and wire it up to ''i; that would be very useful here > as well as in the future I imagine.OK, I also think it can help debug this strange network problem. Let me try coding up. Thanks, Thomas
Reasonably Related Threads
- Help on getting serial output on the arndale board
- Data Abort while in booting when using Julien's new patches on Arndale Board
- Can not boot Dom0 when using Anthony's new XenARM source for Arndale.
- Arndale: domU not booting
- Re: Issue with ARM: Network doesn't work in the guest