Hi all, My setup is Debian testing dom0/domUs with a 2.6.32 pvops kernel from Debian unstable. Hypervisor is 3.4.2 from Debian testing. I use network- and vif- route with a default route in domU pointing to the nic because multiple IPs with bridge would trigger port shutdown on the switch (only one mac-address allowed per port). A subnet is routed to the dom0, which then knows which addresses the domUs have. The problem is that every first packet between a domU and a host not in the arp cache has a very high delay. This is protocol independent (happens for http and dns too) but I''ll provide ping as an example: Route in domU: -------------- $ ip route default dev eth0 scope link Ping from domU to remote host: ------------------------------ $ ping 1.2.3.4 PING 1.2.3.4 (1.2.3.4) 56(84) bytes of data. 64 bytes from 1.2.3.4: icmp_seq=1 ttl=54 time=304 ms 64 bytes from 1.2.3.4: icmp_seq=2 ttl=54 time=22.7 ms 64 bytes from 1.2.3.4: icmp_seq=3 ttl=54 time=22.7 ms ^C --- 1.2.3.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 22.702/116.691/304.574/132.853 ms Immediately replying remote host on second try. $ ping 1.2.3.4 PING 1.2.3.4 (1.2.3.4) 56(84) bytes of data. 64 bytes from 1.2.3.4: icmp_seq=1 ttl=54 time=23.0 ms 64 bytes from 1.2.3.4: icmp_seq=2 ttl=54 time=22.8 ms 64 bytes from 1.2.3.4: icmp_seq=3 ttl=54 time=23.1 ms ^C --- 1.2.3.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 22.895/23.010/23.103/0.151 ms Ping from remote host to domU: ------------------------------ Initial ping to domU from remote host. $ ping 5.6.7.8 PING 5.6.7.8 (5.6.7.8) 56(84) bytes of data. 64 bytes from 5.6.7.8: icmp_seq=1 ttl=51 time=458 ms 64 bytes from 5.6.7.8: icmp_seq=2 ttl=51 time=24.9 ms 64 bytes from 5.6.7.8: icmp_seq=3 ttl=51 time=25.3 ms ^C --- 5.6.7.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 24.936/169.512/458.272/204.184 ms Immediately replying remote host on second try. $ ping 5.6.7.8 PING 5.6.7.8 (5.6.7.8) 56(84) bytes of data. 64 bytes from 5.6.7.8: icmp_seq=1 ttl=51 time=24.6 ms 64 bytes from 5.6.7.8: icmp_seq=2 ttl=51 time=25.3 ms 64 bytes from 5.6.7.8: icmp_seq=3 ttl=51 time=25.2 ms ^C --- 5.6.7.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 24.612/25.068/25.300/0.371 ms Initial time goes up as high as 800ms. My guess is that this has something to do with arp. Since when a host is in the arp cache, the delay disappears, which is why the second ping gets an immediate reply. I tried specifing mac addresses in the domU cfg files and playing with domU routing flags without apparent benefit. Is this an issue with Xen? Is this related to my setup? thanks Luca _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Try simultaneous tcpdumps on dom0 and domU while trying this test. You can then see the ARP traffic and maybe be able to determine where the delay is introduced. I guess it''s possible that ARP resolution time could be incorporated into the first RTT result. I just tested ARP response times on my physical and xen servers and saw resolution usually occurring between 0.02 ms and 0.2 ms. If ARP isn''t your problem, tcpdumps should show this too. The command I used tcpdump -w file.pcap -nei eth0 Russ Purinton Director of Network Operations Office 207.591.6900 Direct: 207.591.6908 Fax: 207.591.6919 rpurinton@voipnettechnologies.com www.voipnettechnologies.com -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Luca Corti Sent: Tuesday, April 27, 2010 5:47 AM To: xen-users@lists.xensource.com Subject: [Xen-users] High network latency on first packet Hi all, My setup is Debian testing dom0/domUs with a 2.6.32 pvops kernel from Debian unstable. Hypervisor is 3.4.2 from Debian testing. I use network- and vif- route with a default route in domU pointing to the nic because multiple IPs with bridge would trigger port shutdown on the switch (only one mac-address allowed per port). A subnet is routed to the dom0, which then knows which addresses the domUs have. The problem is that every first packet between a domU and a host not in the arp cache has a very high delay. This is protocol independent (happens for http and dns too) but I''ll provide ping as an example: Route in domU: -------------- $ ip route default dev eth0 scope link Ping from domU to remote host: ------------------------------ $ ping 1.2.3.4 PING 1.2.3.4 (1.2.3.4) 56(84) bytes of data. 64 bytes from 1.2.3.4: icmp_seq=1 ttl=54 time=304 ms 64 bytes from 1.2.3.4: icmp_seq=2 ttl=54 time=22.7 ms 64 bytes from 1.2.3.4: icmp_seq=3 ttl=54 time=22.7 ms ^C --- 1.2.3.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 22.702/116.691/304.574/132.853 ms Immediately replying remote host on second try. $ ping 1.2.3.4 PING 1.2.3.4 (1.2.3.4) 56(84) bytes of data. 64 bytes from 1.2.3.4: icmp_seq=1 ttl=54 time=23.0 ms 64 bytes from 1.2.3.4: icmp_seq=2 ttl=54 time=22.8 ms 64 bytes from 1.2.3.4: icmp_seq=3 ttl=54 time=23.1 ms ^C --- 1.2.3.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 22.895/23.010/23.103/0.151 ms Ping from remote host to domU: ------------------------------ Initial ping to domU from remote host. $ ping 5.6.7.8 PING 5.6.7.8 (5.6.7.8) 56(84) bytes of data. 64 bytes from 5.6.7.8: icmp_seq=1 ttl=51 time=458 ms 64 bytes from 5.6.7.8: icmp_seq=2 ttl=51 time=24.9 ms 64 bytes from 5.6.7.8: icmp_seq=3 ttl=51 time=25.3 ms ^C --- 5.6.7.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 24.936/169.512/458.272/204.184 ms Immediately replying remote host on second try. $ ping 5.6.7.8 PING 5.6.7.8 (5.6.7.8) 56(84) bytes of data. 64 bytes from 5.6.7.8: icmp_seq=1 ttl=51 time=24.6 ms 64 bytes from 5.6.7.8: icmp_seq=2 ttl=51 time=25.3 ms 64 bytes from 5.6.7.8: icmp_seq=3 ttl=51 time=25.2 ms ^C --- 5.6.7.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 24.612/25.068/25.300/0.371 ms Initial time goes up as high as 800ms. My guess is that this has something to do with arp. Since when a host is in the arp cache, the delay disappears, which is why the second ping gets an immediate reply. I tried specifing mac addresses in the domU cfg files and playing with domU routing flags without apparent benefit. Is this an issue with Xen? Is this related to my setup? thanks Luca _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2010-04-27 at 08:34 -0400, Russ Purinton wrote:> Try simultaneous tcpdumps on dom0 and domU while trying this test. You > can then see the ARP traffic and maybe be able to determine where the > delay is introduced. I guess it''s possible that ARP resolution time > could be incorporated into the first RTT result. I just tested ARP > response times on my physical and xen servers and saw resolution usually > occurring between 0.02 ms and 0.2 ms. > > If ARP isn''t your problem, tcpdumps should show this too. > The command I used > > tcpdump -w file.pcap -nei eth0dump on domU: ping remotehost -> domU 14:50:38.926785 IP remotehost > domU: ICMP echo request, id 22598, seq 1, length 64 14:50:38.927786 ARP, Request who-has remotehost tell domU, length 28 14:50:39.437493 ARP, Reply remotehost is-at fe:ff:ff:ff:ff:ff (oui Unknown), length 28 14:50:39.437500 IP domU > remotehost: ICMP echo reply, id 22598, seq 1, length 64 The problem is with ARP taking > 500ms to reply. This seems very similar to http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=596 ciao Luca _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2010-04-27 at 15:06 +0200, Luca Corti wrote:> The problem is with ARP taking > 500ms to reply. > > This seems very similar to > > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=596A workaround is to add a gateway host in addition to the device route. This way arp lookups are skipped. $ netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 eth0 $ netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 1.2.3.4 0.0.0.0 UG 0 0 0 eth0 0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 eth0 Note that this works despite the gateway being on a subnet which is not configured on any interface, the IP address could also be fake (1.2.3.4 can be actually configured and solving the issue without harm, since it is a bogon). ARP lookups are skipped in the presence of any route with a gateway host because the gateway ip is itself resolved via ARP and then cached: $ arp -a ? (1.2.3.4) at fe:ff:ff:ff:ff:ff [ether] on eth0 This is ugly anyway, a cleaner solution would be great. ciao Luca _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users