Xen users - Jan 2010 - Networking stops working for no apparent reason

Hi,

I have a Xen system with two NICs. One is delegated to domU which serves 
as router  (for an adsl modem) and one is bridged between the router 
domU, and the physical local net (including dom0).

Or: adsl NIC --- domU(router) --- xenbr (NIC2) --- internal net & dom0

This was working quite well for two weeks, however, for no apparent 
reason the networking has become very unstable over the last few days. 
There are no obvious error messages in either dom0 or domU but when I 
try pinging from the router domU to either dom0 or other domUs, I only 
get "Network unreachable.". Connections on the adsl NIC are still ok.

This is weird, as the system might run for a a day or two - or only 
minutes. Shutting down domUs, networking, xend, libvirtd, etc and 
restarting has no effect. Only restarting the entire system helps.

I am using Centos 5.4 with gitco xen hypervisor 3.4 and xenlinux 
2.6.18.8 on the router domU.

I would appreciate any hints on how to go about debugging the xen 
network. For all I can see it "should" to work just fine...

Many thanks,
Arik


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 20/01/2010 20:47, Arik Raffael Funke wrote:
> There are no obvious error messages in either dom0 or domU but when I
> try pinging from the router domU to either dom0 or other domUs, I only
> get "Network unreachable.". Connections on the adsl NIC are still
ok.
I should probably mention that the connection from the local physical 
net to dom0 still works. Just not the followin:
dom0<->domU
domU<->domU
domU<->physical local net (via brctl)

It thus seems to be a xen networking problem.

- Arik


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> 
> This was working quite well for two weeks, however, for no apparent
> reason the networking has become very unstable over the last few days.
> There are no obvious error messages in either dom0 or domU but when I
> try pinging from the router domU to either dom0 or other domUs, I only
> get "Network unreachable.". Connections on the adsl NIC are still
ok.
> 
> This is weird, as the system might run for a a day or two - or only
> minutes. Shutting down domUs, networking, xend, libvirtd, etc and
> restarting has no effect. Only restarting the entire system helps.
> 
I had similar problems where Dom0 would stop correcting TCP checksums
when routing packets out over a GRE tunnel. It caused huge amounts of
frustration. I resolved the problem by disabling checksum offloading
everywhere. Another server does this occasionally too.

It''s a bit different to your situation, but had the same pattern of
working for a bit then stopping.

Try disabling checksum offloading everywhere and see if it becomes more
stable.

James



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Jan 21, 2010 at 6:05 AM, James Harper
<james.harper@bendigoit.com.au> wrote:
>> This is weird, as the system might run for a a day or two - or only
>> minutes. Shutting down domUs, networking, xend, libvirtd, etc and
>> restarting has no effect. Only restarting the entire system helps.
>>
>
> Try disabling checksum offloading everywhere and see if it becomes more
> stable.
If it''s TCP offloading issue, shouldn''t ICMP still work?

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> 
> On Thu, Jan 21, 2010 at 6:05 AM, James Harper
> <james.harper@bendigoit.com.au> wrote:
> >> This is weird, as the system might run for a a day or two - or
only
> >> minutes. Shutting down domUs, networking, xend, libvirtd, etc and
> >> restarting has no effect. Only restarting the entire system helps.
> >>
> >
> 
> > Try disabling checksum offloading everywhere and see if it becomes
more
> > stable.
> 
> If it''s TCP offloading issue, shouldn''t ICMP still work?
> 
On the surface yes, but the problems I was seeing involved Linux getting
quite upset and refusing to do any packet routing when TCP offloading
was in use. Minutes later it would start again. Nothing was ever logged
anywhere to indicate problems it just stopped routing packets.

It''s one of those things that isn''t necessarily likely to
resolve the
problem but is really easy to check.

James


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 21/01/2010 03:39, James Harper wrote:
> On the surface yes, but the problems I was seeing involved Linux getting
> quite upset and refusing to do any packet routing when TCP offloading
> was in use. Minutes later it would start again. Nothing was ever logged
> anywhere to indicate problems it just stopped routing packets.
The problem desciption "minutes later it would start again" is just
what
is happening to me. I can provoke the xen network to "crash" by doing 
"yum update". After downloading just over 1 MB, it breaks the
networking
for a few minutes before getting back to normal.

I disabled checksum offloading on all network devices (in dom0 and domU) 
but with no effect.

Do you have any other ideas? This is really puzzling to me...

Another thing: while the network is broken, when I try pinging from domU 
to dom0, the result is either "Network unreachable" or sometimes 
"sendmsg: no buffer space available". However I checked with free and 
buffer space is well over 6 MB on domU and dom0.

- Arik


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users