thr3ads.net - Xen devel - [Xen-devel] network hang trigger [Sep 2004]

If this information is useful, please help other people find it:
Share via:

James Harper

2004-Sep-15 07:23 UTC

[Xen-devel] network hang trigger

I have found that simply doing a ping with packetsize > 1500 (maybe
mtu???) causes the network to hang for a short time. This is from xenU
and xen0 on the same machine.

Can someone else _please_ test this? I am able to make the network hang
by saying from domU:
ping -s 1473 <dom0 ip>
(1473 + 28 byte header = 1501 byte packet)

I''ll do more testing tomorrow.

thanks

James


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

James Harper

2004-Sep-15 08:38 UTC

head link

RE: [Xen-devel] network hang trigger

Lowering the MTU introduces the problem at a lower packetsize as
expected, so I''d suspect the problem is fragmentation.

I did a test running tcpdumps in both domains, and when I do a ping from
domU to dom0, I see the correct packets on domU, but dom0 only gets the
first fragment then nothing.

James
> -----Original Message-----
> From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> admin@lists.sourceforge.net] On Behalf Of James Harper
> Sent: Wednesday, 15 September 2004 17:24
> To: xen-devel@lists.sourceforge.net
> Subject: [Xen-devel] network hang trigger
> 
> I have found that simply doing a ping with packetsize > 1500 (maybe
> mtu???) causes the network to hang for a short time. This is from xenU
> and xen0 on the same machine.
> 
> Can someone else _please_ test this? I am able to make the network
hang> by saying from domU:
> ping -s 1473 <dom0 ip>
> (1473 + 28 byte header = 1501 byte packet)
> 
> I''ll do more testing tomorrow.
> 
> thanks
> 
> James
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Peri Hankey

2004-Sep-15 08:38 UTC

head link

Re: [Xen-devel] network hang trigger

James

I can confirm that your problem occurs in xen-2.0-20040914. In this 
example a4.local is the xen0 machine (2.6.8.1-xen0), a30 is one of 
several xenU guests on a4  (all 2.6.8.1-xenU). First a simple ping, 
which works OK. Then your special poison ping: nasty pause, and possibly 
useful messages.

    [administrator@a30 administrator]$ ping a4.local
    PING a4.local (192.168.0.4) 56(84) bytes of data.
    64 bytes from a4.local (192.168.0.4): icmp_seq=1 ttl=64 time=5.72 ms
    64 bytes from a4.local (192.168.0.4): icmp_seq=2 ttl=64 time=0.150 ms

    --- a4.local ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1010ms
    rtt min/avg/max/mdev = 0.150/2.937/5.724/2.787 ms
    [administrator@a30 administrator]$ ping -s 1473 a4.local
    PING a4.local (192.168.0.4) 1473(1501) bytes of data.

    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
    ping: sendmsg: No buffer space available
     From 192.168.0.4 icmp_seq=1 Frag reassembly time exceeded
    1481 bytes from 192.168.0.4: icmp_seq=2 ttl=64 time=28980 ms
    1481 bytes from 192.168.0.4: icmp_seq=3 ttl=64 time=27980 ms
    1481 bytes from a4.local (192.168.0.4): icmp_seq=4 ttl=64 time=26980 ms

    --- a4.local ping statistics ---
    11 packets transmitted, 3 received, +1 errors, 72% packet loss, time
    29287ms
    rtt min/avg/max/mdev = 26980.569/27980.589/28980.612/816.525 ms, pipe 11

I hope that helps.
 
Peri

James Harper wrote:
>I have found that simply doing a ping with packetsize > 1500 (maybe
>mtu???) causes the network to hang for a short time. This is from xenU
>and xen0 on the same machine.
>
>Can someone else _please_ test this? I am able to make the network hang
>by saying from domU:
>ping -s 1473 <dom0 ip>
>(1473 + 28 byte header = 1501 byte packet)
>
>I''ll do more testing tomorrow.
>
>thanks
>
>James
>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
>Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
>Camcorder. More prizes in the weekly Lunch Hour Challenge.
>Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/xen-devel
>
>
>  
>


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Keir Fraser

2004-Sep-15 09:54 UTC

head link

Re: [Xen-devel] network hang trigger

I can''t reproduce this. e.g.,
 [root@druid-0 root]# ping -s 2473 128.232.39.40
 PING 128.232.39.40 (128.232.39.40) 2473(2501) bytes of data.
 2481 bytes from 128.232.39.40: icmp_seq=0 ttl=64 time=0.074 ms
 2481 bytes from 128.232.39.40: icmp_seq=1 ttl=64 time=0.057 ms
 2481 bytes from 128.232.39.40: icmp_seq=2 ttl=64 time=0.055 ms
 2481 bytes from 128.232.39.40: icmp_seq=3 ttl=64 time=0.057 ms

Both domains have 256MB memory. MTU for both is 1500.

 -- Keir
> I have found that simply doing a ping with packetsize > 1500 (maybe
> mtu???) causes the network to hang for a short time. This is from xenU
> and xen0 on the same machine.
> 
> Can someone else _please_ test this? I am able to make the network hang
> by saying from domU:
> ping -s 1473 <dom0 ip>
> (1473 + 28 byte header = 1501 byte packet)
> 
> I''ll do more testing tomorrow.
> 
> thanks
> 
> James
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel -=- MIME -=- 
I have found that simply doing a ping with packetsize > 1500 (maybe
mtu???) causes the network to hang for a short time. This is from xenU
and xen0 on the same machine.

Can someone else _please_ test this? I am able to make the network hang
by saying from domU:
ping -s 1473 <dom0 ip>
(1473 + 28 byte header =3D 1501 byte packet)

I''ll do more testing tomorrow.

thanks

James


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Chris Andrews

2004-Sep-15 10:48 UTC

head link

Re: [Xen-devel] network hang trigger

I''m seeing this problem:

81.187.70.17 is a secondary on DOM0''s bridge. The host
''uml18'' is DOM1.

uml18:~# ping -s 2473 81.187.70.17
PING 81.187.70.17 (81.187.70.17) 2473(2501) bytes of data.
2481 bytes from 81.187.70.17: icmp_seq=1 ttl=64 time=0.559 ms

.. time passes, I hit ^C ..

ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
2481 bytes from 81.187.70.17: icmp_seq=6 ttl=64 time=30297 ms
--- 81.187.70.17 ping statistics ---
12 packets transmitted, 2 received, 83% packet loss, time 34627ms
rtt min/avg/max/mdev = 0.559/15149.215/30297.872/15148.657 ms, pipe 7

DOM0 has 384M, DOM1 has 64M, MTU 1500 in both.

Different packet sizes seem to trigger the bug more or less quickly -- I 
first tried with 2000 byte packets, and didn''t see the problem, then I
tried
2473, and saw the problem then. Trying 2000 bytes again did then show the 
problem.

This is running Xen-2.0 from 4th September -- I can give a more recent 
version a try, but should I be able to turn on writable pagetables with 
current code, given I had problems with that option with the code I''m 
running now?

Chris.


Keir Fraser wrote:> I can''t reproduce this. e.g.,
>  [root@druid-0 root]# ping -s 2473 128.232.39.40
>  PING 128.232.39.40 (128.232.39.40) 2473(2501) bytes of data.
>  2481 bytes from 128.232.39.40: icmp_seq=0 ttl=64 time=0.074 ms
>  2481 bytes from 128.232.39.40: icmp_seq=1 ttl=64 time=0.057 ms
>  2481 bytes from 128.232.39.40: icmp_seq=2 ttl=64 time=0.055 ms
>  2481 bytes from 128.232.39.40: icmp_seq=3 ttl=64 time=0.057 ms
> 
> Both domains have 256MB memory. MTU for both is 1500.
> 
>  -- Keir
> 
> 
>>I have found that simply doing a ping with packetsize > 1500 (maybe
>>mtu???) causes the network to hang for a short time. This is from xenU
>>and xen0 on the same machine.
>>
>>Can someone else _please_ test this? I am able to make the network hang
>>by saying from domU:
>>ping -s 1473 <dom0 ip>
>>(1473 + 28 byte header = 1501 byte packet)
>>
>>I''ll do more testing tomorrow.
>>
>>thanks
>>
>>James
>>
>>
>>-------------------------------------------------------
>>This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
>>Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
>>Camcorder. More prizes in the weekly Lunch Hour Challenge.
>>Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.sourceforge.net
>>https://lists.sourceforge.net/lists/listinfo/xen-devel
> 
>  -=- MIME -=- 
> I have found that simply doing a ping with packetsize > 1500 (maybe
> mtu???) causes the network to hang for a short time. This is from xenU
> and xen0 on the same machine.
> 
> Can someone else _please_ test this? I am able to make the network hang
> by saying from domU:
> ping -s 1473 <dom0 ip>
> (1473 + 28 byte header =3D 1501 byte packet)
> 
> I''ll do more testing tomorrow.
> 
> thanks
> 
> James
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Keir Fraser

2004-Sep-15 12:33 UTC

head link

Re: [Xen-devel] network hang trigger

Do you see this if you ping from DOM1 to an external machine? Or from
an external machine to DOM1?

My suspicion is that this is just some weird interaction between
bridging and local packet dleivery in DOM0. I also can cause lots of
packet loss when pinging DOM1 from DOM0, if I use DOM1''s hostname
rather than IP address(!!!). It''s very bizarre, and tcpdump tells me
that the ICMP packets come back to DOM0 via the VIF, but it looks like
they then get swallowed somehow within the bridge.

I think that the bridging code is really intended for dedicated bridge
boxes with no local IP interface. So I''m not sure whether the support
for that might not be flaky in some circumstances.

 -- Keir
> I''m seeing this problem:
> 
> 81.187.70.17 is a secondary on DOM0''s bridge. The host
''uml18'' is DOM1.
> 
> uml18:~# ping -s 2473 81.187.70.17
> PING 81.187.70.17 (81.187.70.17) 2473(2501) bytes of data.
> 2481 bytes from 81.187.70.17: icmp_seq=1 ttl=64 time=0.559 ms
> 
> .. time passes, I hit ^C ..
> 
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> 2481 bytes from 81.187.70.17: icmp_seq=6 ttl=64 time=30297 ms
> --- 81.187.70.17 ping statistics ---
> 12 packets transmitted, 2 received, 83% packet loss, time 34627ms
> rtt min/avg/max/mdev = 0.559/15149.215/30297.872/15148.657 ms, pipe 7
> 
> DOM0 has 384M, DOM1 has 64M, MTU 1500 in both.
> 
> Different packet sizes seem to trigger the bug more or less quickly -- I 
> first tried with 2000 byte packets, and didn''t see the problem,
then I tried
> 2473, and saw the problem then. Trying 2000 bytes again did then show the 
> problem.
> 
> This is running Xen-2.0 from 4th September -- I can give a more recent 
> version a try, but should I be able to turn on writable pagetables with 
> current code, given I had problems with that option with the code
I''m
> running now?

-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Chris Andrews

2004-Sep-15 12:56 UTC

head link

Re: [Xen-devel] network hang trigger

Keir Fraser wrote:> Do you see this if you ping from DOM1 to an external machine? Or from
> an external machine to DOM1?
Both these cases seem fine.

I also see the ''packet loss using hostname'' weirdness pinging
from 0 to 1,
yet tcpdump in dom1 tells me that icmp packets are going both ways...

I *also* see more icmp redirects than I might have expected, so maybe
I''ll
try changing things around a bit -- the xenU domains are using a secondary 
on dom0''s bridge device as their default as they''re not on the
same ip
address range as dom0, so I could try routing instead of bridging.

(wish I could get bidirectional serial console to dom0 working ... some odd 
interaction with the Dell console-redirect-foo, I suspect.)


Chris.






-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Keir Fraser

2004-Sep-15 13:13 UTC

head link

Re: [Xen-devel] network hang trigger

> Keir Fraser wrote:
> > Do you see this if you ping from DOM1 to an external machine? Or from
> > an external machine to DOM1?
> 
> Both these cases seem fine.
> 
> I also see the ''packet loss using hostname'' weirdness
pinging from 0 to 1,
> yet tcpdump in dom1 tells me that icmp packets are going both ways...
Seems this only makes it easier to cause packet loss. Sometimes I can
get the same behaviour using DOM1''s IP address directly.

There is defintiely some weird bridging thing going on here...

 -- Keir
> I *also* see more icmp redirects than I might have expected, so maybe
I''ll
> try changing things around a bit -- the xenU domains are using a secondary 
> on dom0''s bridge device as their default as they''re not
on the same ip
> address range as dom0, so I could try routing instead of bridging.
> 
> (wish I could get bidirectional serial console to dom0 working ... some odd
> interaction with the Dell console-redirect-foo, I suspect.)
> 
> 
> Chris.
> 
> 
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

James Harper

2004-Sep-15 14:43 UTC

head link

RE: [Xen-devel] network hang trigger

Hmmmm... are you running 2.6 and pinging from xenU to xen0?

I''ve read your other emails, and I can still get it to fail if I remove
vif1.0 from the bridge and give it its own ip address, but to properly
exclude bridging I guess I''d have to boot up the computer with bridging
never having been started...

I''m running code checked out and compiled yesterday (15th AEST), and
have writable pagetables on but only since that last compile. It fails
either way. I have no iptables/conntrack related modules loaded in
either domain.

I think we''ll have to start comparing .config''s, but I need to
get some
sleep.

James
> -----Original Message-----
> From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
> Sent: Wednesday, 15 September 2004 19:55
> To: James Harper
> Cc: xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] network hang trigger
> 
> 
> I can''t reproduce this. e.g.,
>  [root@druid-0 root]# ping -s 2473 128.232.39.40
>  PING 128.232.39.40 (128.232.39.40) 2473(2501) bytes of data.
>  2481 bytes from 128.232.39.40: icmp_seq=0 ttl=64 time=0.074 ms
>  2481 bytes from 128.232.39.40: icmp_seq=1 ttl=64 time=0.057 ms
>  2481 bytes from 128.232.39.40: icmp_seq=2 ttl=64 time=0.055 ms
>  2481 bytes from 128.232.39.40: icmp_seq=3 ttl=64 time=0.057 ms
> 
> Both domains have 256MB memory. MTU for both is 1500.
> 
>  -- Keir
> 
> > I have found that simply doing a ping with packetsize > 1500 (maybe
> > mtu???) causes the network to hang for a short time. This is from
xenU> > and xen0 on the same machine.
> >
> > Can someone else _please_ test this? I am able to make the network
hang> > by saying from domU:
> > ping -s 1473 <dom0 ip>
> > (1473 + 28 byte header = 1501 byte packet)
> >
> > I''ll do more testing tomorrow.
> >
> > thanks
> >
> > James
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: thawte''s Crypto Challenge
Vl
> > Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> > Camcorder. More prizes in the weekly Lunch Hour Challenge.
> > Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
>  -=- MIME -=- 
> 
> 
> I have found that simply doing a ping with packetsize > 1500 (maybe
> mtu???) causes the network to hang for a short time. This is from xenU
> and xen0 on the same machine.
> 
> Can someone else _please_ test this? I am able to make the network
hang> by saying from domU:
> ping -s 1473 <dom0 ip>
> (1473 + 28 byte header =3D 1501 byte packet)
> 
> I''ll do more testing tomorrow.
> 
> thanks
> 
> James
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Keir Fraser

2004-Sep-15 15:19 UTC

head link

Re: [Xen-devel] network hang trigger

Okay, I can reproduce this too.

DOM0:
 brctl delif xen-br0 vif1.0
 ifconfig vif1.0 10.0.0.1 up

DOM1:
 ifconfig eth0 10.0.0.2 up

Then if I ping 10.0.0.2 from DOM0 it appears to stop getting responses
after a few round trips, although tcpdump says that responses are
still getting received.

If I ping 10.0.0.1 from DOM1 then it just seems to seize up
occasionally. I can get the same buffer error messages as you if I set
a packet size > MTU.

Currently I have no idea what this could be due to. :-)

 -- Keir
> Hmmmm... are you running 2.6 and pinging from xenU to xen0?
> 
> I''ve read your other emails, and I can still get it to fail if I
remove
> vif1.0 from the bridge and give it its own ip address, but to properly
> exclude bridging I guess I''d have to boot up the computer with
bridging
> never having been started...
> 
> I''m running code checked out and compiled yesterday (15th AEST),
and
> have writable pagetables on but only since that last compile. It fails
> either way. I have no iptables/conntrack related modules loaded in
> either domain.
> 
> I think we''ll have to start comparing .config''s, but I
need to get some
> sleep.
> 
> James

-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Charles Coffing

2004-Sep-15 15:29 UTC

head link

RE: [Xen-devel] network hang trigger

I was able to reproduce the hang easily.  "ping -s 1473" worked, but
"ping -s 1499" hung.  While it was hung, I tried pinging the other
direction and that hung too.

My setup:
Pinging from DOMU to DOM0
Changeset 1.1307
DOM0: 2.6.8.1, Stock configuration
DOMU: 2.6.8.1, Stock, except writable pagetables are disabled

ccoffing2:~ # ping -s 1499 137.65.171.60
PING 137.65.171.60 (137.65.171.60) 1499(1527) bytes of data.
1507 bytes from 137.65.171.60: icmp_seq=1 ttl=64 time=0.455 ms
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available>From 137.65.171.60: icmp_seq=2 Frag reassembly time exceeded
>From 137.65.171.60 icmp_seq=2 Frag reassembly time exceeded1507 bytes from 137.65.171.60: icmp_seq=3 ttl=64 time=28980 ms
1507 bytes from 137.65.171.60: icmp_seq=4 ttl=64 time=27980 ms
1507 bytes from 137.65.171.60: icmp_seq=5 ttl=64 time=26980 ms

--- 137.65.171.60 ping statistics ---
15 packets transmitted, 4 received, +2 errors, 73% packet loss, time
33213ms
rtt min/avg/max/mdev = 0.455/20985.849/28980.992/12136.540 ms, pipe 11




>>> "James Harper" <JamesH@bendigoit.com.au> 09/15/04
8:43 AM >>>Hmmmm... are you running 2.6 and pinging from xenU to xen0?

I''ve read your other emails, and I can still get it to fail if I remove
vif1.0 from the bridge and give it its own ip address, but to properly
exclude bridging I guess I''d have to boot up the computer with bridging
never having been started...

I''m running code checked out and compiled yesterday (15th AEST), and
have writable pagetables on but only since that last compile. It fails
either way. I have no iptables/conntrack related modules loaded in
either domain.

I think we''ll have to start comparing .config''s, but I need to
get some
sleep.

James
> -----Original Message-----
> From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
> Sent: Wednesday, 15 September 2004 19:55
> To: James Harper
> Cc: xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] network hang trigger
> 
> 
> I can''t reproduce this. e.g.,
>  [root@druid-0 root]# ping -s 2473 128.232.39.40
>  PING 128.232.39.40 (128.232.39.40) 2473(2501) bytes of data.
>  2481 bytes from 128.232.39.40: icmp_seq=0 ttl=64 time=0.074 ms
>  2481 bytes from 128.232.39.40: icmp_seq=1 ttl=64 time=0.057 ms
>  2481 bytes from 128.232.39.40: icmp_seq=2 ttl=64 time=0.055 ms
>  2481 bytes from 128.232.39.40: icmp_seq=3 ttl=64 time=0.057 ms
> 
> Both domains have 256MB memory. MTU for both is 1500.
> 
>  -- Keir
> 
> > I have found that simply doing a ping with packetsize > 1500 (maybe
> > mtu???) causes the network to hang for a short time. This is from
xenU> > and xen0 on the same machine.
> >
> > Can someone else _please_ test this? I am able to make the network
hang> > by saying from domU:
> > ping -s 1473 <dom0 ip>
> > (1473 + 28 byte header = 1501 byte packet)
> >
> > I''ll do more testing tomorrow.
> >
> > thanks
> >
> > James
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: thawte''s Crypto Challenge
Vl
> > Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> > Camcorder. More prizes in the weekly Lunch Hour Challenge.
> > Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
>  -=- MIME -=- 
> 
> 
> I have found that simply doing a ping with packetsize > 1500 (maybe
> mtu???) causes the network to hang for a short time. This is from xenU
> and xen0 on the same machine.
> 
> Can someone else _please_ test this? I am able to make the network
hang> by saying from domU:
> ping -s 1473 <dom0 ip>
> (1473 + 28 byte header =3D 1501 byte packet)
> 
> I''ll do more testing tomorrow.
> 
> thanks
> 
> James
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Bin Ren

2004-Sep-15 21:19 UTC

head link

Re: [Xen-devel] network hang trigger

After some investigation, I seem to be pin down the cause as
skbuff_head_cache overflow. do ''cat /proc/slabinfo | grep
skbuff'', the
first column is the number of active skbuff_head. On my machine, it''s
like this:

(1) after normal ''ping dom0'', skbuff_head_cache active is 210
(2) after ''ping -s 3000 dom0'' and network hang, it shoots up
to 254
(3) further ''ping -s 3000 dom0'' and it shoots up to 340. Get a
few
replies. Entire network hang.
(4) wait for several minutes, skbuff_head_cache gets
garbage-collected, drops down to 120. Network recovers.

This problem also occurs to linux, freebsd, netbsd kernels with faulty
device drivers. It''s likely that network frontend driver is not
freeing skbuff properly. Skbuff cache gets overflowed until some
threshold that it''s gc''ed by force.

I''m taking a look at the netback source codes.

-- Bin Ren

On Wed, 15 Sep 2004 09:29:46 -0600, Charles Coffing <ccoffing@novell.com>
wrote:> I was able to reproduce the hang easily.  "ping -s 1473" worked,
but
> "ping -s 1499" hung.  While it was hung, I tried pinging the
other
> direction and that hung too.
> 
> My setup:
> Pinging from DOMU to DOM0
> Changeset 1.1307
> DOM0: 2.6.8.1, Stock configuration
> DOMU: 2.6.8.1, Stock, except writable pagetables are disabled
> 
> ccoffing2:~ # ping -s 1499 137.65.171.60
> PING 137.65.171.60 (137.65.171.60) 1499(1527) bytes of data.
> 1507 bytes from 137.65.171.60: icmp_seq=1 ttl=64 time=0.455 ms
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> From 137.65.171.60: icmp_seq=2 Frag reassembly time exceeded
> From 137.65.171.60 icmp_seq=2 Frag reassembly time exceeded
> 1507 bytes from 137.65.171.60: icmp_seq=3 ttl=64 time=28980 ms
> 1507 bytes from 137.65.171.60: icmp_seq=4 ttl=64 time=27980 ms
> 1507 bytes from 137.65.171.60: icmp_seq=5 ttl=64 time=26980 ms
> 
> --- 137.65.171.60 ping statistics ---
> 15 packets transmitted, 4 received, +2 errors, 73% packet loss, time
> 33213ms
> rtt min/avg/max/mdev = 0.455/20985.849/28980.992/12136.540 ms, pipe 11

-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Bin Ren

2004-Sep-15 21:23 UTC

head link

Re: [Xen-devel] network hang trigger

After some investigation, I seem to be pin down the cause as
skbuff_head_cache overflow. do ''cat /proc/slabinfo | grep
skbuff'', the
first column is the number of active skbuff_head. On my machine, it''s
like this:

(1) after normal ''ping dom0'', skbuff_head_cache active is 210
(2) after ''ping -s 3000 dom0'' and network hang, it shoots up
to 254
(3) further ''ping -s 3000 dom0'' and it shoots up to 340. Get a
few
replies. Entire network hang.
(4) wait for several minutes, skbuff_head_cache gets
garbage-collected, drops down to 120. Network recovers.

This problem also occurs to linux, freebsd, netbsd kernels with faulty
device drivers. It''s likely that network frontend driver is not
freeing skbuff properly. Skbuff cache gets overflowed until some
threshold that it''s gc''ed by force.

I''m taking a look at the netfront/back source codes.

-- Bin Ren
- Hide quoted text -

On Wed, 15 Sep 2004 09:29:46 -0600, Charles Coffing <ccoffing@novell.com> 
wrote:> I was able to reproduce the hang easily.  "ping -s 1473" worked,
but
> "ping -s 1499" hung.  While it was hung, I tried pinging the
other
> direction and that hung too.
>
> My setup:
> Pinging from DOMU to DOM0
> Changeset 1.1307
> DOM0: 2.6.8.1, Stock configuration
> DOMU: 2.6.8.1, Stock, except writable pagetables are disabled
>
> ccoffing2:~ # ping -s 1499 137.65.171.60
> PING 137.65.171.60 (137.65.171.60) 1499(1527) bytes of data.
> 1507 bytes from 137.65.171.60: icmp_seq=1 ttl=64 time=0.455 ms
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> ping: sendmsg: No buffer space available
> From 137.65.171.60: icmp_seq=2 Frag reassembly time exceeded
> From 137.65.171.60 icmp_seq=2 Frag reassembly time exceeded
> 1507 bytes from 137.65.171.60: icmp_seq=3 ttl=64 time=28980 ms
> 1507 bytes from 137.65.171.60: icmp_seq=4 ttl=64 time=27980 ms
> 1507 bytes from 137.65.171.60: icmp_seq=5 ttl=64 time=26980 ms
>
> --- 137.65.171.60 ping statistics ---
> 15 packets transmitted, 4 received, +2 errors, 73% packet loss, time
> 33213ms
> rtt min/avg/max/mdev = 0.455/20985.849/28980.992/12136.540 ms, pipe 11

-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Bin Ren

2004-Sep-15 22:11 UTC

head link

Re: [Xen-devel] network hang trigger

Here is what I find out:

In netfront.c, the transmit function is:

network_start_xmit(struct sk_buff *skb, struct net_device *dev)

Currently, whenever there is an error, it returns 1 or -ENOBUFS 
***WITHOUT*** freeing the ***skb***. This is based on the assumption that 
the caller, seeing a non-zero return value, will free the ***skb***.
Let''s
take a look at the caller:

int dev_queue_xmit(struct sk_buff *skb)

in file net/core.c
=====================================if (!dev->hard_start_xmit(skb, dev)) {
        HARD_TX_UNLOCK_BH(dev);
        goto out;
}
...

out_kfree_skb:
        kfree_skb(skb);
out:
        return rc;
=====================================
Bingo, it doesn''t. And take a look at other network driver source
codes,
e.g. 8139too.c, 3c501.c etc, they all ***free skb*** upon error and 
***always return 0***. This is *hidden contract* between the caller and the 
callee.

So, skbuffs don''t get freed until gc''ed.

I''m going to modify the files and see the result.
Keep tuned.

-- Bin Ren


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

James Harper

2004-Sep-15 23:53 UTC

head link

RE: [Xen-devel] network hang trigger

I see this problem after 1 ping of size > MTU (which probably explains
why NFS hangs too!). If I understand correctly, the situation you
describe should only come about if an error occurs for other reasons.

So do we still have another outstanding bug?

Thanks

James
> -----Original Message-----
> From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> admin@lists.sourceforge.net] On Behalf Of Bin Ren
> Sent: Thursday, 16 September 2004 08:11
> To: xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] network hang trigger
> 
> Here is what I find out:
> 
> In netfront.c, the transmit function is:
> 
> network_start_xmit(struct sk_buff *skb, struct net_device *dev)
> 
> Currently, whenever there is an error, it returns 1 or -ENOBUFS
> ***WITHOUT*** freeing the ***skb***. This is based on the assumption
that> the caller, seeing a non-zero return value, will free the ***skb***.
Let''s> take a look at the caller:
> 
> int dev_queue_xmit(struct sk_buff *skb)
> 
> in file net/core.c
> =====================================> if (!dev->hard_start_xmit(skb,
dev)) {
>         HARD_TX_UNLOCK_BH(dev);
>         goto out;
> }
> ...
> 
> out_kfree_skb:
>         kfree_skb(skb);
> out:
>         return rc;
> =====================================> 
> Bingo, it doesn''t. And take a look at other network driver source
codes,> e.g. 8139too.c, 3c501.c etc, they all ***free skb*** upon error and
> ***always return 0***. This is *hidden contract* between the caller
and> the
> callee.
> 
> So, skbuffs don''t get freed until gc''ed.
> 
> I''m going to modify the files and see the result.
> Keep tuned.
> 
> -- Bin Ren
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
> Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
> Camcorder. More prizes in the weekly Lunch Hour Challenge.
> Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Bin Ren

2004-Sep-15 23:54 UTC

head link

Re: [Xen-devel] network hang trigger

I''ve just modified the netfront.c (haven''t touched netback.c
yet). I''ve
done two modifications: (1) free sk_buff properly on the transmit path (2) 
In netif_poll(...) function, packets **should not** be passed to 
netif_rx(); instead, use: int netif_receive_skb(struct sk_buff *skb).

With these two modifications, under ''ping -s 6000'', network
only
occasionally loses a few packets but *very soon* recovers. It''s much
more
stable than before. I''ll take a closer look at netfront.c and netback.c
tomorrow.

Here is the patch. Please try it out. With your results, the changes may 
get pushed into the repository.

-- Bin Ren

===== linux-2.6.8.1-xen-sparse/drivers/xen/netfront/netfront.c 1.49 vs 
edited ===== --- 
1.49/linux-2.6.8.1-xen-sparse/drivers/xen/netfront/netfront.c 2004-08-27 
13:28:33 +01:00 +++ 
edited/linux-2.6.8.1-xen-sparse/drivers/xen/netfront/netfront.c 2004-09-16 
00:34:33 +01:00 @@ -329,7 +329,7 @@
     {
         printk(KERN_ALERT "%s: full queue wasn''t
stopped!\n", dev->name);
         netif_stop_queue(dev);
-        return -ENOBUFS;
+       goto drop;
     }

     if ( unlikely((((unsigned long)skb->data & ~PAGE_MASK) +
skb->len) >@@ -337,7 +337,7 @@
     {
         struct sk_buff *new_skb;
         if ( unlikely((new_skb = alloc_skb_page()) == NULL) )
-            return 1;
+            goto drop;
         skb_put(new_skb, skb->len);
         memcpy(new_skb->data, skb->data, skb->len);
         dev_kfree_skb(skb);
@@ -349,7 +349,7 @@
     if ( np->backend_state != BEST_CONNECTED )
     {
         spin_unlock_irq(&np->tx_lock);
-        return 1;
+        goto drop;
     }

     i = np->tx->req_prod;
@@ -385,6 +385,10 @@
         notify_via_evtchn(np->evtchn);

     return 0;
+
+ drop:
+    dev_kfree_skb(skb);
+    return 0;
 }


@@ -501,7 +505,7 @@
         skb->protocol = eth_type_trans(skb, dev);

         /* Pass it up. */
-        netif_rx(skb);
+        netif_receive_skb(skb);
         dev->last_rx = jiffies;
     }


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Bin Ren

2004-Sep-16 00:04 UTC

head link

Re: [Xen-devel] network hang trigger

Just another note:

Before the patch, ping -s 6000 from dom0 -> domU *and* domU -> dom0 *both*
make the network hang. Now with the patch, ping -s 6000 from dom0 -> domU 
*never* hangs (or, to be more accurate, successful for the first 127 
packets before Ctrl+C !). but from domU -> dom0 network occassionally loses 
packets.

Sth *asymmetric* is playing the trick. This may shed some light on the 
problem. I have to go to sleep before figuring it out.

-- Bin


-------------------------------------------------------
This SF.Net email is sponsored by: thawte''s Crypto Challenge Vl
Crack the code and win a Sony DCRHC40 MiniDV Digital Handycam
Camcorder. More prizes in the weekly Lunch Hour Challenge.
Sign up NOW http://ad.doubleclick.net/clk;10740251;10262165;m
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

James Harper

2004-Sep-16 00:48 UTC

head link

RE: [Xen-devel] network hang trigger

This patch makes no difference on my system. Looking at the line numbers
in your patch, your netfront.c appears to be a different vintage to
mine. I applied the patch manually and then rebuilt the xenU kernel and
then booted a domain with it. I haven''t touched xen0. Is this the
correct thing to be doing?

Ping >mtu from xenU to xen0 causes the network to hang immediately.
Ping >mtu from xen0 to xenU sometimes causes the network to hang, but
not always. ''ping -i 0.1 -s 6000 <xenU ip>'' will mostly
cause the hang
in under 30 iterations.

The recovery time appears to be in the order of 60 seconds or so, with a
partial recovery and then relapse at about 30 seconds.
 
When I was thinking about this problem, I was imagining a deadlock
condition where rapid back to back packets (eg a fragmented icmp packet
from ping or a fragmented udp packet from nfs) was causing a hang until
part of the deadlock timed itself out and the packets started flowing
again. I couldn''t see 1 packet causing a buffer exhaustion unless it
got
itself into a loop where it kept retrying to send the second fragment
and didn''t free the buffer each time, but even then the buffer bug
would
be a side effect.

The deadlock would have to be caused in the transmit from xenU to xen0,
and something about the difference between sending a ping and responding
to a ping is the difference between always causing a lockup and only
sometimes causing a lockup.

Maybe we''re seeing different manifestations of the same problem?

James
> -----Original Message-----
> From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> admin@lists.sourceforge.net] On Behalf Of Bin Ren
> Sent: Thursday, 16 September 2004 09:54
> To: xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] network hang trigger
> 
> I''ve just modified the netfront.c (haven''t touched
netback.c yet).
I''ve> done two modifications: (1) free sk_buff properly on the transmit path
(2)> In netif_poll(...) function, packets **should not** be passed to
> netif_rx(); instead, use: int netif_receive_skb(struct sk_buff *skb).
> 
> With these two modifications, under ''ping -s 6000'',
network only
> occasionally loses a few packets but *very soon* recovers. It''s
much
more> stable than before. I''ll take a closer look at netfront.c and
netback.c> tomorrow.
> 
> Here is the patch. Please try it out. With your results, the changes
may> get pushed into the repository.
> 

-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

James Harper

2004-Sep-16 02:32 UTC

head link

RE: [Xen-devel] network hang trigger

This lends weight to my idea that it might be a timing related deadlock,
I put some debug statements in to see just how often the drop: path was
called in network_start_xmit (never), and now I almost can''t get it to
fail. I put printk''s in at the start of the function, and before both
the returns.

This particular machine is not SMP btw. It is a single CPU PII running
at 350Mhz (697 bogomips)

James
> -----Original Message-----
> From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> admin@lists.sourceforge.net] On Behalf Of James Harper
> Sent: Thursday, 16 September 2004 10:48
> To: Bin Ren; xen-devel@lists.sourceforge.net
> Subject: RE: [Xen-devel] network hang trigger
> 
> This patch makes no difference on my system. Looking at the line
numbers> in your patch, your netfront.c appears to be a different vintage to
> mine. I applied the patch manually and then rebuilt the xenU kernel
and> then booted a domain with it. I haven''t touched xen0. Is this the
> correct thing to be doing?
> 
> Ping >mtu from xenU to xen0 causes the network to hang immediately.
> Ping >mtu from xen0 to xenU sometimes causes the network to hang, but
> not always. ''ping -i 0.1 -s 6000 <xenU ip>'' will
mostly cause the hang
> in under 30 iterations.
> 
> The recovery time appears to be in the order of 60 seconds or so, with
a> partial recovery and then relapse at about 30 seconds.
> 
> When I was thinking about this problem, I was imagining a deadlock
> condition where rapid back to back packets (eg a fragmented icmp
packet> from ping or a fragmented udp packet from nfs) was causing a hang
until> part of the deadlock timed itself out and the packets started flowing
> again. I couldn''t see 1 packet causing a buffer exhaustion unless
it
got> itself into a loop where it kept retrying to send the second fragment
> and didn''t free the buffer each time, but even then the buffer bug
would> be a side effect.
> 
> The deadlock would have to be caused in the transmit from xenU to
xen0,> and something about the difference between sending a ping and
responding> to a ping is the difference between always causing a lockup and only
> sometimes causing a lockup.
> 
> Maybe we''re seeing different manifestations of the same problem?
> 
> James
> 
> > -----Original Message-----
> > From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> > admin@lists.sourceforge.net] On Behalf Of Bin Ren
> > Sent: Thursday, 16 September 2004 09:54
> > To: xen-devel@lists.sourceforge.net
> > Subject: Re: [Xen-devel] network hang trigger
> >
> > I''ve just modified the netfront.c (haven''t touched
netback.c yet).
> I''ve
> > done two modifications: (1) free sk_buff properly on the transmit
path> (2)
> > In netif_poll(...) function, packets **should not** be passed to
> > netif_rx(); instead, use: int netif_receive_skb(struct sk_buff
*skb).> >
> > With these two modifications, under ''ping -s 6000'',
network only
> > occasionally loses a few packets but *very soon* recovers.
It''s much
> more
> > stable than before. I''ll take a closer look at netfront.c and
> netback.c
> > tomorrow.
> >
> > Here is the patch. Please try it out. With your results, the changes
> may
> > get pushed into the repository.
> >
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement
on> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Keir Fraser

2004-Sep-16 07:24 UTC

head link

Re: [Xen-devel] network hang trigger

> When I was thinking about this problem, I was imagining a deadlock
> condition where rapid back to back packets (eg a fragmented icmp packet
> from ping or a fragmented udp packet from nfs) was causing a hang until
> part of the deadlock timed itself out and the packets started flowing
> again. I couldn''t see 1 packet causing a buffer exhaustion unless
it got
> itself into a loop where it kept retrying to send the second fragment
> and didn''t free the buffer each time, but even then the buffer bug
would
> be a side effect.
> 
> The deadlock would have to be caused in the transmit from xenU to xen0,
> and something about the difference between sending a ping and responding
> to a ping is the difference between always causing a lockup and only
> sometimes causing a lockup.
Try tcpdumping each end of teh connecttion.

I find that for ping 0->U, the ''seizure'' is entirely within
DOM0 --
ping responses are still received, but for some reason they don''t make
it up to the ping application.

For ping U->0, it does look as though the network seizes up -- I see
no packets in either direction.

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Keir Fraser

2004-Sep-16 07:33 UTC

head link

Re: [Xen-devel] network hang trigger

> in file net/core.c
> =====================================> if (!dev->hard_start_xmit(skb,
dev)) {
>         HARD_TX_UNLOCK_BH(dev);
>         goto out;
> }
> ...
> 
> out_kfree_skb:
>         kfree_skb(skb);
> out:
>         return rc;
> =====================================> 
> Bingo, it doesn''t. And take a look at other network driver source
codes,
> e.g. 8139too.c, 3c501.c etc, they all ***free skb*** upon error and 
> ***always return 0***. This is *hidden contract* between the caller and the
> callee.
(1) See drivers like 3c59x.c. They do not free the skbuff when they
    return non-zero.

(2) Look again at the code fragment you''ve posted -- the body of the
    ''if'' statement is executed when hard_start_xmit returns
zero. So
    the body corresponds to the NON-ERROR case! You''ll see that the
    error case does in fact free the skbuff.
> So, skbuffs don''t get freed until gc''ed.
No, there is no bug here. And if we were leaking skbuffs they would
never be freed. There is no automatic garbage collection within the
kernel.

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

James Harper

2004-Sep-16 10:02 UTC

head link

RE: [Xen-devel] network hang trigger

I''ve tried this, and I see the first fragment of the ping get sent and
then a complete hang, which is what originally made me suspicious that
there was some sort of race with sending packets with a very small time
between one and the next.
 
It could be that Bin''s patch changed the timing of things on his
machine
such that the bug goes away for him. I can make the bug come and go by
placing printk''s in network_start_xmit as per my previous email.
 
This is a dump from a normal size ping.
Xen0
listening on vif13.0, link-type EN10MB (Ethernet), capture size 96 bytes
19:02:48.276397 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 64: echo
request seq 1
19:02:48.306646 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 64: echo
reply seq 1
19:02:49.275931 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 64: echo
request seq 2
19:02:49.276033 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 64: echo
reply seq 2
XenU
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
19:02:48.270125 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 64: echo
request seq 1
19:02:48.277577 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 64: echo
reply seq 1
19:02:49.275460 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 64: echo
request seq 2
19:02:49.276848 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 64: echo
reply seq 2
 
This is from a large ping (with printk''s in network_start_xmit so it
works)
Xen0
19:10:33.502706 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 1480:
echo request seq 1
19:10:33.502711 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp
19:10:33.502966 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 1480:
echo reply seq 1
19:10:33.502992 IP xen2.int.sbss.com.au > 192.168.200.200: icmp
19:10:34.496713 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 1480:
echo request seq 2
19:10:34.496717 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp
19:10:34.496872 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 1480:
echo reply seq 2
19:10:34.496895 IP xen2.int.sbss.com.au > 192.168.200.200: icmp
XenU
19:10:33.496431 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 1480:
echo request seq 1
19:10:33.498042 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp
19:10:33.507890 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 1480:
echo reply seq 1
19:10:33.507953 IP xen2.int.sbss.com.au > 192.168.200.200: icmp
19:10:34.492920 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 1480:
echo request seq 2
19:10:34.494703 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp
19:10:34.501604 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 1480:
echo reply seq 2
19:10:34.501639 IP xen2.int.sbss.com.au > 192.168.200.200: icmp
 
This is from the same large ping (with the printk''s removed so it
hangs)
Xen0
listening on vif14.0, link-type EN10MB (Ethernet), capture size 96 bytes
19:23:25.125927 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 1480:
echo request seq 1
19:23:55.122574 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 556: ip
reassembly time exceeded
19:23:55.122726 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp
19:23:55.122732 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 1480:
echo request seq 2
19:23:55.122734 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp
19:23:55.122735 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 1480:
echo request seq 3
19:23:55.122737 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp
19:23:55.122739 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp 1480:
echo request seq 4
19:23:55.122741 IP 192.168.200.200 > xen2.int.sbss.com.au: icmp
19:23:55.123850 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 1480:
echo reply seq 2
19:23:55.123873 IP xen2.int.sbss.com.au > 192.168.200.200: icmp
19:23:55.123955 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 1480:
echo reply seq 3
19:23:55.123977 IP xen2.int.sbss.com.au > 192.168.200.200: icmp
19:23:55.124050 IP xen2.int.sbss.com.au > 192.168.200.200: icmp 1480:
echo reply seq 4
19:23:55.124070 IP xen2.int.sbss.com.au > 192.168.200.200: icmp
XenU
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
19:23:25.126797 IP 192.168.200.200 > 192.168.200.204: icmp 1480: echo
request seq 1
19:23:25.129472 IP 192.168.200.200 > 192.168.200.204: icmp
19:23:26.143609 IP 192.168.200.200 > 192.168.200.204: icmp 1480: echo
request seq 2
19:23:26.143622 IP 192.168.200.200 > 192.168.200.204: icmp
19:23:27.143643 IP 192.168.200.200 > 192.168.200.204: icmp 1480: echo
request seq 3
19:23:27.143660 IP 192.168.200.200 > 192.168.200.204: icmp
19:23:28.143643 IP 192.168.200.200 > 192.168.200.204: icmp 1480: echo
request seq 4
19:23:28.143658 IP 192.168.200.200 > 192.168.200.204: icmp
19:23:55.124352 IP 192.168.200.204 > 192.168.200.200: icmp 556: ip
reassembly time exceeded
19:23:55.126145 IP 192.168.200.204 > 192.168.200.200: icmp 1480: echo
reply seq 2
19:23:55.126170 IP 192.168.200.204 > 192.168.200.200: icmp
19:23:55.126201 IP 192.168.200.204 > 192.168.200.200: icmp 1480: echo
reply seq 3
19:23:55.126208 IP 192.168.200.204 > 192.168.200.200: icmp
19:23:55.126224 IP 192.168.200.204 > 192.168.200.200: icmp 1480: echo
reply seq 4
19:23:55.126230 IP 192.168.200.204 > 192.168.200.200: icmp
 
The times are in sync between the two domains, so you can see that dom0
only sees the first fragment of the first ping and then a big delay,
then the rest come through.
 
Is it possible that there is a synchronisation problem in interdomain
communications?
 
James
 > -----Original Message-----
> From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
> Sent: Thursday, 16 September 2004 17:24
> To: James Harper
> Cc: Bin Ren; xen-devel@lists.sourceforge.net
> Subject: Re: [Xen-devel] network hang trigger
> 
> > When I was thinking about this problem, I was imagining a deadlock
> > condition where rapid back to back packets (eg a fragmented icmp
packet> > from ping or a fragmented udp packet from nfs) was causing a hang
until> > part of the deadlock timed itself out and the packets started
flowing> > again. I couldn''t see 1 packet causing a buffer exhaustion
unless it
got> > itself into a loop where it kept retrying to send the second
fragment> > and didn''t free the buffer each time, but even then the
buffer bug
would> > be a side effect.
> >
> > The deadlock would have to be caused in the transmit from xenU to
xen0,> > and something about the difference between sending a ping and
responding> > to a ping is the difference between always causing a lockup and only
> > sometimes causing a lockup.
> 
> Try tcpdumping each end of teh connecttion.
> 
> I find that for ping 0->U, the ''seizure'' is entirely
within DOM0 --
> ping responses are still received, but for some reason they don''t
make
> it up to the ping application.
> 
> For ping U->0, it does look as though the network seizes up -- I see
> no packets in either direction.
> 
>  -- Keir

Bin Ren

2004-Sep-16 11:06 UTC

head link

RE: [Xen-devel] network hang trigger

Please use ''tcpdump -nt icmp -vv -i interface'' to capture more
details.

Now, with the correct NAPI rx call, i cannot produce the hang, which is 
really confusing. I ping -s 6000 from domU to dom0 and get this:

NOTE: The large request is *NOT* fragmented!! The reply is!

IP (tos 0x0, ttl 64, id 35115, offset 0, flags [none], length: 6028) 
192.168.0.101 > 192.168.0.1: icmp 6008: echo request seq 5632 IP (tos 0x0, 
ttl 64, id 24314, offset 0, flags [+], length: 1500) 192.168.0.1 > 
192.168.0.101: icmp 1480: echo reply seq 5632 IP (tos 0x0, ttl 64, id 
24314, offset 1480, flags [+], length: 1500) 192.168.0.1 > 192.168.0.101: 
icmp IP (tos 0x0, ttl 64, id 24314, offset 2960, flags [+], length: 1500) 
192.168.0.1 > 192.168.0.101: icmp IP (tos 0x0, ttl 64, id 24314, offset 
4440, flags [+], length: 1500) 192.168.0.1 > 192.168.0.101: icmp IP (tos 
0x0, ttl 64, id 24314, offset 5920, flags [none], length: 108) 192.168.0.1
> 192.168.0.101: icmp


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Bin Ren

2004-Sep-16 11:22 UTC

head link

RE: [Xen-devel] network hang trigger

Sorry, in the previous post, ping output is messed up. Here it is:

IP (tos 0x0, ttl 64, id 39991, offset 0, flags [none], length: 1528) 
192.168.0.101 > 192.168.0.1: icmp 1508: echo request seq 1024

IP (tos 0x0, ttl 64, id 29480, offset 0, flags [+], length: 1500) 
192.168.0.1 > 192.168.0.101: icmp 1480: echo reply seq 1024

IP (tos 0x0, ttl 64, id 29480, offset 1480, flags [none], length: 48) 
192.168.0.1 > 192.168.0.101: icmp

-- Bin


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Bin Ren

2004-Sep-16 11:33 UTC

head link

RE: [Xen-devel] network hang trigger

I can no longer trigger a hang with the latest bkbits. I even use ''ping
-f
-s 6000 dom0'' to send hundreds of icmp requests per second, no hang.

The confusing part is that on James'' machine, both the big icmp request
and
reply are fragmented. On my machine, no matter how large is the packet, 
icmp request is *NOT* fragmented *without* IP Don''t Fragment (DF) flag
set!
And the MTU is 1500.

To make it even more confusing, with normal size (64 bytes in ping dom0), 
DF flag is set!

Could this be a kernel problem? Both of my dom0 and domU are running 2.6.8.1

-- Bin 


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Xen devel - Sep 2004 - network hang trigger

[Xen-devel] network hang trigger

RE: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

RE: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

RE: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

RE: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

RE: [Xen-devel] network hang trigger

RE: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

Re: [Xen-devel] network hang trigger

RE: [Xen-devel] network hang trigger

RE: [Xen-devel] network hang trigger

RE: [Xen-devel] network hang trigger

RE: [Xen-devel] network hang trigger