Adam Wilbraham
2010-Apr-26 11:23 UTC
[Xen-users] Network traffic stalling between domUs on different servers
Up until now I''ve been using a single DL380 running Xen 3.3.1 on top of
Debian Lenny (using the Debian Etch Xen kernel) to host a number of
domUs, however I have had to install a second dom0 (this time a DL360,
again Lenny + Etch kernel, this time Xen 3.4.2) to take over some of the
domUs due to load issues.
Network connectivity between domUs on the same host has always been
absolutely fine, however I am having a problem with network traffic
between domUs on the new dom0 when connecting to domUs on the original
dom0. Basically, connections can be established but as soon as data
starts to flow then the connection stalls and eventually drops.
The network on each dom0 is configured as a series of VLAN bridges, eg:
auto eth1.38
iface eth1.38 inet manual
auto vlan38
iface vlan38 inet manual
pre-up ip link set eth1.38 down
pre-up brctl addbr vlan38
pre-up brctl addif vlan38 eth1.38
pre-up ip link set eth1.38 up
pre-up ip link set vlan38 up
post-down ip link set vlan38 down
post-down brctl delbr vlan38
My head keeps telling me that this issue is related to TCP checksum
offloading, but I''ve already turned that off and set the MTU to 1496 in
case the network card doesn''t cope with the VLAN header properly:
pre-up ethtool -K eth1 tx off
pre-up ifconfig eth1 mtu 1496
The NICs in both servers are Broadcom ones, bnx2 driver.
I managed to get an NFS export mounted and vaguely stable between domUs
when forcing the it to use UDP and an rsize and wsize of 1024, however
thats hardly workable and only for NFS. If I try and SCP a file between
boxes then only a few k gets transferred before it stalls, similar from
other services.
Does anyone have any experience of similar issues?
--
Adam Wilbraham - Senior Systems Administrator
TechnoPhobia Limited
The Workstation
15 Paternoster Row
SHEFFIELD
England
S1 2BX
t: +44 (0)114 2212123
f: +44 (0)114 2212124
e: adam.wilbraham@technophobia.com
w: http://www.technophobia.com/
Registered in England and Wales Company No. 3063669
VAT registration No. 598 7858 42
ISO 9001:2000 Accredited Company No. 21227
ISO 14001:2004 Accredited Company No. E997
ISO 27001:2005 (BS7799) Accredited Company No. IS 508906
Investor in People Certified No. 101507
The contents of this email are confidential to the addressee
and are intended solely for the recipients use. If you are not
the addressee, you have received this email in error.
Any disclosure, copying, distribution or action taken in
reliance on it is prohibited and may be unlawful.
Any opinions expressed in this email are those of the author
personally and not TechnoPhobia Limited who do not accept
responsibility for the contents of the message.
All email communications, in and out of TechnoPhobia,
are recorded for monitoring purposes.
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Adam Wilbraham
2010-Apr-26 15:25 UTC
Re: [Xen-users] Network traffic stalling between domUs on different servers
Sorted this - stupid misconfiguration on my part. I had still got my physical interfaces in the dom0 defined as "static" rather than "manual". Switched this to how it should be and all is now fine. On 26/04/10 12:23, Adam Wilbraham wrote:> Up until now I''ve been using a single DL380 running Xen 3.3.1 on top of > Debian Lenny (using the Debian Etch Xen kernel) to host a number of > domUs, however I have had to install a second dom0 (this time a DL360, > again Lenny + Etch kernel, this time Xen 3.4.2) to take over some of the > domUs due to load issues. > > Network connectivity between domUs on the same host has always been > absolutely fine, however I am having a problem with network traffic > between domUs on the new dom0 when connecting to domUs on the original > dom0. Basically, connections can be established but as soon as data > starts to flow then the connection stalls and eventually drops. > > The network on each dom0 is configured as a series of VLAN bridges, eg: > > auto eth1.38 > iface eth1.38 inet manual > > auto vlan38 > iface vlan38 inet manual > pre-up ip link set eth1.38 down > pre-up brctl addbr vlan38 > pre-up brctl addif vlan38 eth1.38 > pre-up ip link set eth1.38 up > pre-up ip link set vlan38 up > post-down ip link set vlan38 down > post-down brctl delbr vlan38 > > My head keeps telling me that this issue is related to TCP checksum > offloading, but I''ve already turned that off and set the MTU to 1496 in > case the network card doesn''t cope with the VLAN header properly: > > pre-up ethtool -K eth1 tx off > pre-up ifconfig eth1 mtu 1496 > > The NICs in both servers are Broadcom ones, bnx2 driver. > > I managed to get an NFS export mounted and vaguely stable between domUs > when forcing the it to use UDP and an rsize and wsize of 1024, however > thats hardly workable and only for NFS. If I try and SCP a file between > boxes then only a few k gets transferred before it stalls, similar from > other services. > > Does anyone have any experience of similar issues? >-- Adam Wilbraham - Senior Systems Administrator TechnoPhobia Limited The Workstation 15 Paternoster Row SHEFFIELD England S1 2BX t: +44 (0)114 2212123 f: +44 (0)114 2212124 e: adam.wilbraham@technophobia.com w: http://www.technophobia.com/ Registered in England and Wales Company No. 3063669 VAT registration No. 598 7858 42 ISO 9001:2000 Accredited Company No. 21227 ISO 14001:2004 Accredited Company No. E997 ISO 27001:2005 (BS7799) Accredited Company No. IS 508906 Investor in People Certified No. 101507 The contents of this email are confidential to the addressee and are intended solely for the recipients use. If you are not the addressee, you have received this email in error. Any disclosure, copying, distribution or action taken in reliance on it is prohibited and may be unlawful. Any opinions expressed in this email are those of the author personally and not TechnoPhobia Limited who do not accept responsibility for the contents of the message. All email communications, in and out of TechnoPhobia, are recorded for monitoring purposes. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users