Hi,
I have a group of physical xen servers (dom0) which have been working
well for some time. However, recently I converted an existing Linux
machine to a domU and have been having networking issues with this domU.
Specifically, if from a physical machine on the network, I ping the
domU, then I get normal ping response times, but for some periods
(assuming when under a heavier load) the ping time is slow, though no
packets are ever dropped. This impacts all domU's and the dom0 at the
same time, moving the domU (live migrate) to a different dom0 causes the
other dom0 to have the same problem (ie, the problem follows the domU).
ie,
20140529-14:53:18
x.x.10.9
50 pings in the range 0.18 to 0.27ms
0.27
157.61
117.40
472.00
623.85
956.83
1360.31
1472.61
2045.73
2528.93
2304.32
2651.66
2853.93
-
-
-
The last three pings were lost (ie, more than three seconds, and fping
didn't wait for them). When testing manually, if I wait long enough, no
packets are ever lost, but the delay can get very high).
This is the exact same period pinging to the dom0 on the xen bridge
interface
x.x.10.34
0.16
0.16
372.27
453.27
820.93
985.35
1358.14
1747.34
2000.70
2370.45
2660.26
2356.98
2859.98
-
-
-
Pinging to other physical network interfaces on the same dom0 is not
affected. eg in the same period of time:
2997 packets transmitted, 2997 received, 0% packet loss, time 2996002ms
rtt min/avg/max/mdev = 0.044/0.091/0.410/0.018 ms
No slow/delayed pings at all on the iSCSI network interface (not bridged
for xen, not exposed to the domU's
2998 packets transmitted, 2998 received, 0% packet loss, time 2997024ms
rtt min/avg/max/mdev = 0.084/6.858/1232.196/69.792 ms, pipe 2
Some delayed pings, up to 1.2 seconds as seen here:
64 bytes from x.x.10.34: icmp_seq=2211 ttl=64 time=0.294 ms
64 bytes from x.x.10.34: icmp_seq=2212 ttl=64 time=0.287 ms
64 bytes from x.x.10.34: icmp_seq=2213 ttl=64 time=0.288 ms
64 bytes from x.x.10.34: icmp_seq=2214 ttl=64 time=317 ms
64 bytes from x.x.10.34: icmp_seq=2215 ttl=64 time=724 ms
64 bytes from x.x.10.34: icmp_seq=2216 ttl=64 time=1155 ms
64 bytes from x.x.10.34: icmp_seq=2217 ttl=64 time=1232 ms
64 bytes from x.x.10.34: icmp_seq=2218 ttl=64 time=1202 ms
64 bytes from x.x.10.34: icmp_seq=2219 ttl=64 time=838 ms
64 bytes from x.x.10.34: icmp_seq=2220 ttl=64 time=556 ms
64 bytes from x.x.10.34: icmp_seq=2221 ttl=64 time=229 ms
64 bytes from x.x.10.34: icmp_seq=2222 ttl=64 time=0.266 ms
64 bytes from x.x.10.34: icmp_seq=2223 ttl=64 time=0.239 ms
64 bytes from x.x.10.34: icmp_seq=2224 ttl=64 time=0.238 ms
64 bytes from x.x.10.34: icmp_seq=2225 ttl=64 time=0.265 ms
System details of dom0 (Debian Stable):
root@pm04:~# uname -an
Linux pm04 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u1 x86_64 GNU/Linux
root@pm04:~# dpkg -l | grep xen
ii libxen-4.1 4.1.4-3+deb7u1 amd64 Public libs for Xen
ii libxenstore3.0 4.1.4-3+deb7u1 amd64 Xenstore
communications library for Xen
ii xen-hypervisor-4.1-amd64 4.1.4-3+deb7u1 amd64
Xen Hypervisor on AMD64
ii xen-linux-system-3.2.0-4-amd64 3.2.57-3+deb7u1
amd64 Xen system with Linux 3.2 on 64-bit PCs (meta-package)
ii xen-linux-system-amd64 3.2+46 amd64 Xen
system with Linux for 64-bit PCs (meta-package)
ii xen-system-amd64 4.1.4-3+deb7u1 amd64 Xen
System on AMD64 (meta-package)
ii xen-utils-4.1 4.1.4-3+deb7u1 amd64 XEN
administrative tools
ii xen-utils-common 4.1.4-3+deb7u1 all Xen
administrative tools - common files
ii xenstore-utils 4.1.4-3+deb7u1 amd64 Xenstore
utilities for Xen
domU config file:
name = "mail"
hostname = 'mail'
kernel =
"/mnt/images/server_images/mail/vmlinuz-3.2.0-4-686-pae"
ramdisk =
"/mnt/images/server_images/mail/initrd.img-3.2.0-4-686-pae"
root = "/dev/xvda1"
disk = ['phy:/dev/mapper/mpathmail,xvda,w' ]
memory = 2048
cpus = "2,3"
vcpus = 2
vif = ['bridge=xenbr5, mac=00:16:3e:43:a8:09',
'bridge=xenbr0,
mac=00:16:3e:43:d8:09']
The dom0 network config:
cat /etc/network/interfaces
# The primary network interface
allow-hotplug eth0
auto xenbr0
iface xenbr0 inet static
address x.x.10.34
netmask 255.255.240.0
gateway x.x.10.254
bridge_maxwait 5
bridge_ports regex eth0
# This is the vlan which has the DSL modem plus the problem domU/VM
auto xenbr5
iface xenbr5 inet manual
bridge_ports eth0.5
# Used by dom0 for iscsi SAN only
auto eth1
iface eth1 inet static
address x.x.16.34
netmask 255.255.255.0
mtu 9000
post-up /etc/network/ifup.eth1
# Used by dom0 for iscsi SAN only
auto eth2
iface eth2 inet static
address x.x.16.44
netmask 255.255.255.0
mtu 9000
post-up /etc/network/ifup.eth2
xm vcpu-list
Name ID VCPU CPU State Time(s) CPU
Affinity
Domain-0 0 0 0 -b- 965.7 0
Domain-0 0 1 1 r-- 662.4 1
mail 4 0 2 -b- 1861.0 2
mail 4 1 3 -b- 1134.2 3
oldcal 5 0 4 -b- 80.9 4-5
oldcal 5 1 5 -b- 32.7 4-5
xm network-list mail
Idx BE MAC Addr. handle state evt-ch tx-/rx-ring-ref BE-path
0 0 00:16:3e:43:a8:09 0 4 14 768 /769
/local/domain/0/backend/vif/4/0
1 0 00:16:3e:43:d8:09 1 4 15 770 /1280
/local/domain/0/backend/vif/4/1
The domU is running:
# uname -an
Linux mail 3.2.0-4-686-pae #1 SMP Debian 3.2.57-3+deb7u1 i686 GNU/Linux
The domU is the gateway/firewall for the network which routes between
the internal network and Internet, also runs email/proxy servers etc.
Can anyone suggest what I could do to resolve this issue? Additional
information I should provide?
Thanks,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au