Luis F Urrea
2008-Dec-11 21:10 UTC
[Xen-users] PCI passthrough, poor network performance in upload direction
Hi all, I have been having an issue for over a year with several Xen installations and have not been able to find out if this is just something I am missing. Basically I am trying to configure a firewall with two ethernet cards in a domU using pci passthrough. I am hiding the Ethernet cards at boot time via the pciback.hide kernel option. Right now I am running Xen 3.3.0 and the two Ethernet cards are passed to the FW domU. The FW domU is not configured with a vif interface to the bridge, so basically communication with DomO passes through the external hardware. Tests using iperf show me that download bandwidth from the FW domU to an external machine connected to the switch go as high as 94 Mbits/sec. However upload bandwidth from a laptop to the domU give me a maximum of 574 Kbits/sec which basically is not enough to cover the upload bw of some internet connections. I have read recommendations about using ethtool -K ethX tx off in dom U, but in this case the tx ring does not seem the be the issue since the download speeds are just fine. Using ''ethtool -K rx eth0 off'' did not show any improvements. Since the guest domain has exclusive access to both Ethernet cards I suppose that the overhead is introduced somewhere in the domU''s kernel but I wouldn''t exactly know where else to look for a possible fix. Your input is appreciated. TIA Luis _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Todd Deshane
2008-Dec-11 21:20 UTC
Re: [Xen-users] PCI passthrough, poor network performance in upload direction
On Thu, Dec 11, 2008 at 4:10 PM, Luis F Urrea <lfurrea@gmail.com> wrote:> Hi all, > > I have been having an issue for over a year with several Xen installations > and have not been able to find out if this is just something I am missing. > > Basically I am trying to configure a firewall with two ethernet cards in a > domU using pci passthrough. I am hiding the Ethernet cards at boot time via > the pciback.hide kernel option. > > Right now I am running Xen 3.3.0 and the two Ethernet cards are passed to > the FW domU. > > The FW domU is not configured with a vif interface to the bridge, so > basically communication with DomO passes through the external hardware. > > Tests using iperf show me that download bandwidth from the FW domU to an > external machine connected to the switch go as high as 94 Mbits/sec. However > upload bandwidth from a laptop to the domU give me a maximum of 574 > Kbits/sec which basically is not enough to cover the upload bw of some > internet connections. > > I have read recommendations about using ethtool -K ethX tx off in dom U, but > in this case the tx ring does not seem the be the issue since the download > speeds are just fine. Using ''ethtool -K rx eth0 off'' did not show any > improvements. > > Since the guest domain has exclusive access to both Ethernet cards I suppose > that the overhead is introduced somewhere in the domU''s kernel but I > wouldn''t exactly know where else to look for a possible fix. > > Your input is appreciated. >What is your guest kernel? Is this all PV? What is your hardware (both NIC and system)? What driver is the guest using for the device? Cheers, Todd -- Todd Deshane http://todddeshane.net http://runningxen.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Luis F Urrea
2008-Dec-11 21:26 UTC
Re: [Xen-users] PCI passthrough, poor network performance in upload direction
Guest kernel is: 2.6.18.8-xen This is a paravirtualized Debian installation on the guest booted with PyGrub NICs are both: Broadcom NetXtreme BCM5722 Gig Ethernet PCI Express System: AMD Opteron 2212 2GHz 4 GB RAM on Dell Poweredge T105 DomU''s 512 MB RAM Driver is tg3 compiled from source on guest domain. Thanks for your help! On Thu, Dec 11, 2008 at 2:20 PM, Todd Deshane <deshantm@gmail.com> wrote:> On Thu, Dec 11, 2008 at 4:10 PM, Luis F Urrea <lfurrea@gmail.com> wrote: > > Hi all, > > > > I have been having an issue for over a year with several Xen > installations > > and have not been able to find out if this is just something I am > missing. > > > > Basically I am trying to configure a firewall with two ethernet cards in > a > > domU using pci passthrough. I am hiding the Ethernet cards at boot time > via > > the pciback.hide kernel option. > > > > Right now I am running Xen 3.3.0 and the two Ethernet cards are passed to > > the FW domU. > > > > The FW domU is not configured with a vif interface to the bridge, so > > basically communication with DomO passes through the external hardware. > > > > Tests using iperf show me that download bandwidth from the FW domU to an > > external machine connected to the switch go as high as 94 Mbits/sec. > However > > upload bandwidth from a laptop to the domU give me a maximum of 574 > > Kbits/sec which basically is not enough to cover the upload bw of some > > internet connections. > > > > I have read recommendations about using ethtool -K ethX tx off in dom U, > but > > in this case the tx ring does not seem the be the issue since the > download > > speeds are just fine. Using ''ethtool -K rx eth0 off'' did not show any > > improvements. > > > > Since the guest domain has exclusive access to both Ethernet cards I > suppose > > that the overhead is introduced somewhere in the domU''s kernel but I > > wouldn''t exactly know where else to look for a possible fix. > > > > Your input is appreciated. > > > > What is your guest kernel? > > Is this all PV? > > What is your hardware (both NIC and system)? > > What driver is the guest using for the device? > > Cheers, > Todd > > -- > Todd Deshane > http://todddeshane.net > http://runningxen.com >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thomas Halinka
2008-Dec-11 21:47 UTC
Re: [Xen-users] PCI passthrough, poor network performance in upload direction
Am Donnerstag, den 11.12.2008, 14:26 -0700 schrieb Luis F Urrea:> Guest kernel is: > > 2.6.18.8-xen > > This is a paravirtualized Debian installation on the guest booted with > PyGrub > > NICs are both: > > Broadcom NetXtreme BCM5722 Gig Ethernet PCI Express > > System: > AMD Opteron 2212 2GHz > 4 GB RAM > on Dell Poweredge T105 > > DomU''s 512 MB RAM > > Driver is tg3 compiled from source on guest domain.Please post _full-output_ from one of these interfaces ifconfig ethX and also # cat /proc/interrupts Thomas _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Luis F Urrea
2008-Dec-11 21:51 UTC
Re: [Xen-users] PCI passthrough, poor network performance in upload direction
eth0 Link encap:Ethernet HWaddr 00:10:18:32:01:07
inet addr:172.16.1.130 Bcast:172.16.1.255 Mask:255.255.255.128
inet6 addr: fe80::210:18ff:fe32:107/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:355091 errors:0 dropped:0 overruns:0 frame:0
TX packets:750511 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:37691559 (35.9 MiB) TX bytes:1124647344 (1.0 GiB)
Interrupt:17
eth1 Link encap:Ethernet HWaddr 00:10:18:2f:df:91
inet addr:172.16.0.139 Bcast:172.16.0.255 Mask:255.255.255.128
inet6 addr: fe80::210:18ff:fe2f:df91/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5973 errors:0 dropped:0 overruns:0 frame:0
TX packets:3374 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4131763 (3.9 MiB) TX bytes:601655 (587.5 KiB)
Interrupt:1
firewall:~# cat /proc/interrupts
CPU0
254: 9363 Phys-irq eth1
255: 632464 Phys-irq eth0
256: 67710 Dynamic-irq timer0
257: 0 Dynamic-irq resched0
258: 0 Dynamic-irq callfunc0
259: 358 Dynamic-irq xenbus
260: 0 Dynamic-irq suspend
261: 0 Dynamic-irq pcifront
262: 36 Dynamic-irq xencons
263: 16841 Dynamic-irq blkif
264: 12 Dynamic-irq blkif
NMI: 0
LOC: 0
ERR: 0
MIS: 0
On Thu, Dec 11, 2008 at 2:47 PM, Thomas Halinka <lists@thohal.de> wrote:
> Am Donnerstag, den
11.12.2008<http://172.16.0.145/snom/clicktodial.php?number=11122008&ext=302>,
> 14:26 -0700 schrieb Luis F Urrea:
> > Guest kernel is:
> >
> > 2.6.18.8-xen
> >
> > This is a paravirtualized Debian installation on the guest booted with
> > PyGrub
> >
> > NICs are both:
> >
> > Broadcom NetXtreme BCM5722 Gig Ethernet PCI Express
> >
> > System:
> > AMD Opteron 2212 2GHz
> > 4 GB RAM
> > on Dell Poweredge T105
> >
> > DomU''s 512 MB RAM
> >
> > Driver is tg3 compiled from source on guest domain.
>
> Please post _full-output_ from one of these interfaces
>
> ifconfig ethX
>
> and also
>
> # cat /proc/interrupts
>
> Thomas
>
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Thomas Halinka
2008-Dec-11 22:11 UTC
Re: [Xen-users] PCI passthrough, poor network performance in upload direction
Hi, m Donnerstag, den 11.12.2008, 14:51 -0700 schrieb Luis F Urrea:> eth0 Link encap:Ethernet HWaddr 00:10xxxxxx:01:07 > inet addr:xxxxxxxxxxxxxsecretxxxxxxxxxxx > inet6 addr: fe80::210:18ff:fe32:107/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:355091 errors:0 dropped:0 overruns:0 frame:0 > TX packets:750511 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:37691559 (35.9 MiB) TX bytes:1124647344 (1.0 GiB) > Interrupt:17 > > eth1 Link encap:Ethernet HWaddr 00:10:1xxxxxdf:91 > inet addr:xxxxxxxxxxxxxsecretxxxxxxxxxxx > inet6 addr: fe80::210:18ff:fe2f:df91/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:5973 errors:0 dropped:0 overruns:0 frame:0 > TX packets:3374 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:4131763 (3.9 MiB) TX bytes:601655 (587.5 KiB) > Interrupt:1ok....> firewall:~# cat /proc/interrupts > CPU0 > 254: 9363 Phys-irq eth1 > 255: 632464 Phys-irq eth0 > 256: 67710 Dynamic-irq timer0 > 257: 0 Dynamic-irq resched0 > 258: 0 Dynamic-irq callfunc0 > 259: 358 Dynamic-irq xenbus > 260: 0 Dynamic-irq suspend > 261: 0 Dynamic-irq pcifront > 262: 36 Dynamic-irq xencons > 263: 16841 Dynamic-irq blkif > 264: 12 Dynamic-irq blkif > NMI: 0 > LOC: 0 > ERR: 0 > MIS: 0hmm looks ok. How is pci-hide "implemented" in Kernel ? # cat /boot/config-`uname -r` and the grub-entry for this kernel. have you ever tried running ethtool -t? maybe this points out this issue. Is it i386 or amd64? Thomas _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Luis F Urrea
2008-Dec-11 22:31 UTC
Re: [Xen-users] PCI passthrough, poor network performance in upload direction
This would be amd64 title Xen 3.3.0 / Debian GNU/Linux, kernel 2.6.18.8-xen root (hd0,1) kernel /xen-3.3.0.gz console=tty0 msi module /vmlinuz-2.6.18.8-xen root=UUID=ba92848e-0fd0-4f78-ae59-c77c1fb7d88a ro console=tty0 pciback.permissive pciback.hide=(0000:03:\ 00.0)(0000:04:00.0) module /initrd.img-2.6.18.8-xen firewall:~# ethtool -t eth0 The test result is PASS The test extra info: nvram test (online) 0 link test (online) 0 register test (offline) 0 memory test (offline) 0 loopback test (offline) 0 interrupt test (offline) 0 # XEN # CONFIG_XEN_PRIVILEGED_GUEST=y # CONFIG_XEN_UNPRIVILEGED_GUEST is not set CONFIG_XEN_PRIVCMD=y CONFIG_XEN_XENBUS_DEV=y CONFIG_XEN_BACKEND=y CONFIG_XEN_BLKDEV_BACKEND=y CONFIG_XEN_BLKDEV_TAP=y CONFIG_XEN_NETDEV_BACKEND=y # CONFIG_XEN_NETDEV_PIPELINED_TRANSMITTER is not set # CONFIG_XEN_NETDEV_LOOPBACK is not set CONFIG_XEN_PCIDEV_BACKEND=y # CONFIG_XEN_PCIDEV_BACKEND_VPCI is not set CONFIG_XEN_PCIDEV_BACKEND_PASS=y # CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set # CONFIG_XEN_PCIDEV_BACKEND_CONTROLLER is not set # CONFIG_XEN_PCIDEV_BE_DEBUG is not set CONFIG_XEN_TPMDEV_BACKEND=m CONFIG_XEN_SCSI_BACKEND=m CONFIG_XEN_BLKDEV_FRONTEND=y CONFIG_XEN_NETDEV_FRONTEND=y # CONFIG_XEN_NETDEV_ACCEL_SFC_FRONTEND is not set _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thomas Halinka
2008-Dec-11 23:15 UTC
Re: [Xen-users] PCI passthrough, poor network performance in upload direction
Am Donnerstag, den 11.12.2008, 15:31 -0700 schrieb Luis F Urrea:> This would be amd64 > > title Xen 3.3.0 / Debian GNU/Linux, kernel^^^^^^^^^^^^^^^ :) Have you ever tried this with the lenny 2.6.26-XEN-Kernel?> > 2.6.18.8-xen > root > (hd0,1) > kernel /xen-3.3.0.gz console=tty0 msi^^^^ why? my one with nics...... title CentOS (2.6.18.8 xen, the real kernel) root (hd0,1) kernel /boot/xen.gz-3.3.0 module /boot/vmlinuz-2.6.18.8 ro root=LABEL=/ rhgb quiet pciback.hide=(06:00.0)(01:06.0)(00:02.0) module /boot/initrd-2.6.18.8.img ..... evertried without (leaving blank) or nomsi ? Thomas _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Luis F Urrea
2008-Dec-11 23:36 UTC
Re: [Xen-users] PCI passthrough, poor network performance in upload direction
Thomas, I had to enable msi, since I was getting errors when the interfaces were brought up as suggested below: http://www.nabble.com/Xen-3.3.0-PCI-passthrough-with-pciback.hide-td20835971.html I could certainly try with the 2.6.26-XEN-Kernel. Could you please explain how would I go about using a different kernel than the one the Xen build downloads using hg? I am installing Xen 3.3.0 from sources. On Thu, Dec 11, 2008 at 5:15 PM, Thomas Halinka <lists@thohal.de> wrote:> Am Donnerstag, den 11.12.2008, 15:31 -0700 schrieb Luis F Urrea: > > This would be amd64 > > > > title Xen 3.3.0 / Debian GNU/Linux, kernel > ^^^^^^^^^^^^^^^ :) > > Have you ever tried this with the lenny 2.6.26-XEN-Kernel? > > > > > 2.6.18.8-xen > > root > > (hd0,1) > > kernel /xen-3.3.0.gz console=tty0 msi > ^^^^ > > why? > > my one with nics...... > title CentOS (2.6.18.8 xen, the real kernel) > root (hd0,1) > kernel /boot/xen.gz-3.3.0 > module /boot/vmlinuz-2.6.18.8 ro root=LABEL=/ rhgb quiet > pciback.hide=(06:00.0)(01:06.0)(00:02.0) > module /boot/initrd-2.6.18.8.img > ..... > > evertried without (leaving blank) or nomsi ? > > Thomas > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thomas Halinka
2008-Dec-12 00:13 UTC
Re: [Xen-users] PCI passthrough, poor network performance in upload direction
Am Donnerstag, den 11.12.2008, 17:36 -0600 schrieb Luis F Urrea:> Thomas, > > I had to enable msi, since I was getting errors when the interfaces > were brought up as suggested below: > > http://www.nabble.com/Xen-3.3.0-PCI-passthrough-with-pciback.hide-td20835971.html > > I could certainly try with the 2.6.26-XEN-Kernel. > > Could you please explain how would I go about using a different kernel > than the one the Xen build downloads using hg?in _domU_ : # cat /etc/apt/sources.list ..... deb http://ftp.uni-bayreuth.de/linux/Debian/debian/ lenny main non-free contrib # apt-get update && apt-get install linux-image-2.6.26-1-xen-amd64 && reboot -n should give you: # uname -r 2.6.26-1-xen-amd64> I am installing Xen 3.3.0 from sources.But your guest is still Linux with an "editable Bootloader from domu" through pvgrub. and yes - you can have multiple kernels and choose them through grub :) Ey - it still Linux. Linux itself is just a kernel and a bunch of files - so let''s treat them as files ;) Thomas _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users