Xen devel - Oct 2006 - PCI-E networking

Hi,

 

I see a weird problem with networking under Xen. I am using 2 dual ports
PCI-e nics in my platform and a single PCI-X nic. Each PCI-E nic shares
an irq. Each port is on a different subnet.

 

In native linux I can get line rate in each port.

In Xen (custome xen-unstable)  however, a ping on each subnet is very
slooow. A ping takes about 3000ms or even more. 

 

In fact I downloaded the Xen 3.0.2-3 testing build of xen and I see the
problem there too. Has anyone seen this kind of behavior? Is this
related to the interrupt sharing of the ports?

 

Suggestions

 

Thanks

-          Padma

 

 

 

 

 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> In Xen (custome xen-unstable)  however, a ping on each subnet is very
> slooow. A ping takes about 3000ms or even more.
Is that 3000ms for the first reply or every reply?

Make sure that you ping the IP address and use the -n option to make
sure that DNS is not causing any issues.

Then start a ping and do a tcpdump of icmp and arp packets and send the
results of the first 3 pings or so here. "tcpdump -I eth0 icmp or arp"
should do the trick.

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I see a weird problem with networking under Xen. I am using 2 dual
ports
> PCI-e nics in my platform and a single PCI-X nic. Each PCI-E nic
shares an
> irq. Each port is on a different subnet.
>  
> In native linux I can get line rate in each port.
> 
> In Xen (custome xen-unstable)  however, a ping on each subnet is very
> slooow. A ping takes about 3000ms or even more.
> 
> In fact I downloaded the Xen 3.0.2-3 testing build of xen and I see
the
> problem there too. Has anyone seen this kind of behavior? Is this
related
> to the interrupt sharing of the ports?
Odd. First thing to try is switching to xen-unstable or 3.0.3-testing
there have been some interrupt changes, but still no MSI support, I''m
afraid.

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks.


Here is the output. BTW these cards are PCI-MSI, if that makes a
difference.

Thanks- Padma

 

Ping -n Stats
>>>> ping -n 192.168.133.2
PING 192.168.133.2 (192.168.133.2) 56(84) bytes of data.

64 bytes from 192.168.133.2: icmp_seq=0 ttl=64 time=2400 ms

64 bytes from 192.168.133.2: icmp_seq=1 ttl=64 time=1400 ms

64 bytes from 192.168.133.2: icmp_seq=15 ttl=64 time=2410 ms

64 bytes from 192.168.133.2: icmp_seq=16 ttl=64 time=1410 ms

64 bytes from 192.168.133.2: icmp_seq=17 ttl=64 time=410 ms

64 bytes from 192.168.133.2: icmp_seq=18 ttl=64 time=7070 ms

64 bytes from 192.168.133.2: icmp_seq=19 ttl=64 time=6070 ms

 

Ping with tcpdump with icmp

 
>>>>ping 192.168.133.2
PING 192.168.133.2 (192.168.133.2) 56(84) bytes of data.
>From 192.168.133.1 icmp_seq=0 Destination Host Unreachable
>From 192.168.133.1 icmp_seq=1 Destination Host Unreachable
>From 192.168.133.1 icmp_seq=2 Destination Host Unreachable
64 bytes from 192.168.133.2: icmp_seq=3 ttl=64 time=1990 ms

64 bytes from 192.168.133.2: icmp_seq=4 ttl=64 time=990 ms

64 bytes from 192.168.133.2: icmp_seq=5 ttl=64 time=15.6 ms

64 bytes from 192.168.133.2: icmp_seq=6 ttl=64 time=4010 ms

64 bytes from 192.168.133.2: icmp_seq=7 ttl=64 time=3010 ms

64 bytes from 192.168.133.2: icmp_seq=8 ttl=64 time=2010 ms

64 bytes from 192.168.133.2: icmp_seq=9 ttl=64 time=1010 ms

64 bytes from 192.168.133.2: icmp_seq=10 ttl=64 time=10.7 ms

--- 192.168.133.2 ping statistics ---

22 packets transmitted, 8 received, +3 errors, 63% packet loss, time
21021ms

rtt min/avg/max/mdev = 10.743/1631.395/4010.772/1316.715 ms, pipe 6

 
>>> tcpdump -i eth7 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode

listening on eth7, link-type EN10MB (Ethernet), capture size 96 bytes

23:43:32.021469 IP 192.168.133.1 > 192.168.133.2: icmp 64: echo request
seq3

23:43:32.021473 IP 192.168.133.1 > 192.168.133.2: icmp 64: echo request
seq4

23:43:32.021751 IP 192.168.133.2 > 192.168.133.1: icmp 64: echo reply
seq 3

23:43:32.021754 IP 192.168.133.2 > 192.168.133.1: icmp 64: echo reply
seq 4

23:43:32.030887 IP 192.168.133.1 > 192.168.133.2: icmp 64: echo request
seq5

23:43:32.046498 IP 192.168.133.2 > 192.168.133.1: icmp 64: echo reply
seq 5

23:43:33.030840 IP 192.168.133.1 > 192.168.133.2: icmp 64: echo request
seq6

38 packets captured

38 packets received by filter

0 packets dropped by kernel

 

Ping with tcpdump arp
>>> ping 192.168.133.2
PING 192.168.133.2 (192.168.133.2) 56(84) bytes of data.

64 bytes from 192.168.133.2: icmp_seq=0 ttl=64 time=2896 ms

64 bytes from 192.168.133.2: icmp_seq=1 ttl=64 time=1890 ms

64 bytes from 192.168.133.2: icmp_seq=2 ttl=64 time=890 ms

64 bytes from 192.168.133.2: icmp_seq=3 ttl=64 time=28390 m34 packets
transmitted, 32 received, 5% packet loss, time 33015ms

rtt min/avg/max/mdev = 380.263/13214.810/28390.176/8765.006 ms, pipe 30s

 

 
>>> tcpdump -i eth7 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode

listening on eth7, link-type EN10MB (Ethernet), capture size 96 bytes

0 packets captured

8 packets received by filter

0 packets dropped by kernel

 

 

 

-----Original Message-----
From: James Harper [mailto:james.harper@bendigoit.com.au] 
Sent: Wednesday, October 11, 2006 12:00 AM
To: Apparao, Padmashree K; xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] PCI-E networking

 
> In Xen (custome xen-unstable)  however, a ping on each subnet is very
> slooow. A ping takes about 3000ms or even more.
 

Is that 3000ms for the first reply or every reply?

 

Make sure that you ping the IP address and use the -n option to make

sure that DNS is not causing any issues.

 

Then start a ping and do a tcpdump of icmp and arp packets and send the

results of the first 3 pings or so here. "tcpdump -I eth0 icmp or arp"

should do the trick.

 

James



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Apparao, Padmashree K wrote:
>  
> 
> Thanks.
> 
> 
> Here is the output. BTW these cards are PCI-MSI, if that makes a
difference.
> 
> Thanks- Padma
> 
> * *
> 
1. Try a crossover directly to the other host.
2. Pay attention to ``netstat -s''''
3. DNS wouldn''t affect a ping RTT calculation
4. Try different cables, network hardware, destination host (with a 
different kind of NIC)

-- 
Christopher G. Stach II

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel