Xen users - Mar 2009 - Urgent Network problem! Virtual network stops working after a few minutes/hours

Hi,

I''m using xen 3.3.0 on one server with 3 Guests (2 windows guest with
gplpv
drivers and one linux). After booting the machine all is fine. The domains 
came up and network is working. But after several minutes or hours the network 
stops to work. I can''t reach the guests any more! only dom0 is
reachable
though the network. I asume the virtual network crashes.
How can I find out where the problem is? Log files? command line tools?

This is an urgent problem as this machine is a live server! Any help would be 
very appreciated!

Best Regards,
Martin Fernau

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Hi,
> 
> I''m using xen 3.3.0 on one server with 3 Guests (2 windows guest
with
> gplpv
> drivers and one linux). After booting the machine all is fine. The
domains
> came up and network is working. But after several minutes or hours the
> network
> stops to work. I can''t reach the guests any more! only dom0 is
reachable
> though the network. I asume the virtual network crashes.
> How can I find out where the problem is? Log files? command line
tools?
> 
> This is an urgent problem as this machine is a live server! Any help
would
> be very appreciated!
Try turning off all the checksum and large send offload on all the
interfaces in dom0 and all the domU''s. I have had this happen before on
one server

ethtool -K <iface> rx off
ethtool -K <iface> tx off
ethtool -K <iface> tso off
ethtool -K <iface> gso off

<iface> is eth0, eth1, etc in the DomU''s, and the vifX.Y
interfaces in
the Dom0

See if that gets things going again.

James


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Try turning off all the checksum and large send offload on all the
> interfaces in dom0 and all the domU''s. I have had this happen
before on
> one server
>
> ethtool -K <iface> rx off
> ethtool -K <iface> tx off
> ethtool -K <iface> tso off
> ethtool -K <iface> gso off
>
> <iface> is eth0, eth1, etc in the DomU''s, and the vifX.Y
interfaces in
> the Dom0
>
> See if that gets things going again.
Thanks for the reply. Unfortunately I can''t set "rx off"
parameter as I get
the following error:

--- cut
$> ethtool -K vif3.0 rx off
Cannot set device rx csum settings: Operation not supported
--- cut

the three other commands executed in success. I crossing my fingers that this 
will help in any way as this is VERY unstable at the moment.

What does this mean to me if the network is more stable now? Turning off 
checksums don''t sound like a good idea to me.

Thanks,
Martin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Mar 5, 2009 at 7:25 PM, Martin Fernau <m.fernau@cps-net.de>
wrote:
> What does this mean to me if the network is more stable now? Turning off
> checksums don''t sound like a good idea to me.
It means that your network card is not well-supported by your O.S.
(i.e: driver problem).
Using RHEL/Centos 5.2 (and above) with bundled kernel-xen should work
fine with most NIC.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Try turning off all the checksum and large send offload on all the
> interfaces in dom0 and all the domU''s. I have had this happen
before on
> one server
>
> ethtool -K <iface> rx off
> ethtool -K <iface> tx off
> ethtool -K <iface> tso off
> ethtool -K <iface> gso off
>
> <iface> is eth0, eth1, etc in the DomU''s, and the vifX.Y
interfaces in
> the Dom0
>
> See if that gets things going again.
After nearly 2 hours of work - the network stopped again.
So, this didn''t helped here :(


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,
> I''m using xen 3.3.0 on one server with 3 Guests (2 windows guest
with gplpv
> drivers and one linux). After booting the machine all is fine. The domains
> came up and network is working. But after several minutes or hours the
network
> stops to work. I can''t reach the guests any more! only dom0 is
reachable
Has this always been like this or just recently?
> though the network. I asume the virtual network crashes.
> How can I find out where the problem is? Log files? command line tools?
>
> This is an urgent problem as this machine is a live server! Any help would
be
> very appreciated!
Do you have control over the network switches? Maybe port security is
enabled on the port in the switch that you are connected to? I had
this problem and the dom0 and domU would take turns going on and off
the network every few minutes. I had to tell the network guys to turn
port security off so that the switch allows multiple MAC addresses on
the same port.

HTH,
Sabuj Pattanayek

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Hi,
>
> > I''m using xen 3.3.0 on one server with 3 Guests (2 windows
guest with
> > gplpv drivers and one linux). After booting the machine all is fine.
The
> > domains came up and network is working. But after several minutes or
> > hours the network stops to work. I can''t reach the guests any
more! only
> > dom0 is reachable
>
> Has this always been like this or just recently?
The problems just started today. This machine runs "fine" since a few
month.
But after a reboot this morning the troubles begun...

> > though the network. I asume the virtual network crashes.
> > How can I find out where the problem is? Log files? command line
tools?
> >
> > This is an urgent problem as this machine is a live server! Any help
> > would be very appreciated!
>
> Do you have control over the network switches? Maybe port security is
> enabled on the port in the switch that you are connected to? I had
> this problem and the dom0 and domU would take turns going on and off
> the network every few minutes. I had to tell the network guys to turn
> port security off so that the switch allows multiple MAC addresses on
> the same port.
To be more clear:
I can''t reach the guests neither from the network nor from the dom0
itself -
but I can reach then dom0 from the network. This make me think that there must 
be something wrong with the internas of xen or the virtual networking stack.

I''m using the bridging mode by the way.

Regards,
Martin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> To be more clear:
> I can''t reach the guests neither from the network nor from the
dom0 itself -
> but I can reach then dom0 from the network. This make me think that there
must
> be something wrong with the internas of xen or the virtual networking
stack.
did you make sure port security was turned off on the switch?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > To be more clear:
> > I can''t reach the guests neither from the network nor from
the dom0
> > itself - but I can reach then dom0 from the network. This make me
think
> > that there must be something wrong with the internas of xen or the
> > virtual networking stack.
>
> did you make sure port security was turned off on the switch?
well, I can''t see why a switch should impcat a local connection between
the
dom0 and domU. 
However, the switch never changed and the related switch isn''t a
managed one.
So there is no security related thing on this switch which can be turned on or 
off. The server just worked fine since months. All the trouble started 
today...

-
Martin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>> did you make sure port security was turned off on the switch?
> well, I can''t see why a switch should impcat a local connection
between the
> dom0 and domU.
Yes it can. But since you''re saying that it''s an unmanaged
switch then
I don''t know what the problem could be. If the "network
crashes" then
there is something seriously wrong with the xen kernels. What distro
are you running?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> >> did you make sure port security was turned off on the switch?
> >
> > well, I can''t see why a switch should impcat a local
connection between
> > the dom0 and domU.
>
> Yes it can. But since you''re saying that it''s an
unmanaged switch then
> I don''t know what the problem could be. If the "network
crashes" then
> there is something seriously wrong with the xen kernels. What distro
> are you running?
This is a gentoo system. I run linux-2.6.27 but I try to downgraded to the 
latest supported kernel 2.6.18 at the moment. I hope this downgrade will have 
no impact on my windows guest running gplpv driver.

Another idea is to change the network card to see if it change anything. Does 
a damaged network card can impact the virtual network interfaces in suach a 
way?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Another idea is to change the network card to see if it change anything.
Does
> a damaged network card can impact the virtual network interfaces in suach a
> way?
No then you wouldn''t be able to connect to your dom0.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > Another idea is to change the network card to see if it change
anything.
> > Does a damaged network card can impact the virtual network interfaces
in
> > suach a way?
>
> No then you wouldn''t be able to connect to your dom0.
I really ran out of ideas.. My idea behind this was that if the NIC produce an 
error for a short time that this maybe brake the bridge or the virtual network 
because of unhandled errors or exceptions...
However, I''m not a kernel nor xen developer. So maybe this could be
nonsens :)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

ifconfig or netstat -i isn''t showing any dropped packets or RX-ERR or
TX-ERR on the physical NIC I really doubt it''s the hardware
that''s the
problem.
>> No then you wouldn''t be able to connect to your dom0.
> I really ran out of ideas.. My idea behind this was that if the NIC produce
an
> error for a short time that this maybe brake the bridge or the virtual
network
> because of unhandled errors or exceptions...
> However, I''m not a kernel nor xen developer. So maybe this could
be nonsens :)
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Mar 6, 2009 at 12:18 AM, Martin Fernau <m.fernau@cps-net.de>
wrote:
> The problems just started today. This machine runs "fine" since a
few month.
> But after a reboot this morning the troubles begun...
Did you by any chance upgrade the kernel some time ago, but forgot to
reboot, so that now the server is running a new version of kernel (or
Xen) that is different from the one before?

If yes, most likely kernel problem. Downgrade to the last version that
works for you.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> ifconfig or netstat -i isn''t showing any dropped packets or RX-ERR
or
> TX-ERR on the physical NIC I really doubt it''s the hardware
that''s the
> problem.
I only had some "dropped"-Packets.

Well - now my kernel-2.6.18 is ready to use and this time I''m using the
original NIC driver from the broadcom side itself -> 1.8.2b_1.46.12
referring to the changelog there were some errors in old drivers

---cut
bnx2 v1.8.1f, cnic v1.6.1 (Nov 5, 2008)
=======================================   Fixes
   -----
   1. Problem: All TSO packets corrupted on 5706/5708.

      Cause: Firmware bug zeroing the wrong field in IP header.

      Change: Updated to latest 4.6.11 5706/5708 firmware and 4.6.12 5709
              Firmware.

      Impact: 5706/5708/5709.
[...]
---cut

*crossing fingers*

-
Martin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Hi,
>
> I''m using xen 3.3.0 on one server with 3 Guests (2 windows guest
with gplpv
> drivers and one linux). After booting the machine all is fine. The domains
> came up and network is working. But after several minutes or hours the
> network stops to work. I can''t reach the guests any more! only
dom0 is
> reachable though the network. I asume the virtual network crashes.
> How can I find out where the problem is? Log files? command line tools?
>
> This is an urgent problem as this machine is a live server! Any help would
> be very appreciated!
I discovered to problem!

I don''t know why - but the trouble started from the time after I set
the ip-
adress for the dom0 to one ip from the same network as the guests. Until 
yesterday I had a private IP 172.16.0.1 for my dom0 where all other Computers 
in the LAN had public IPs. But to have better access to the dom0 I decided to 
change this to one IP out of the same public network - and this was the 
problem.
After I switched back to the private IP all is working fine again. network is 
stable since hours now

I don''t know what the problem is in this case. Maybe something with
arp?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

..what about accessibility via vnc/framebuffer?
...what about CPU-usage of domU?

Last time i had this was while migrating drbd-layerd images from one xen to the
other in bridged mode. It disturbt the drbd connection and drbd shut down. Then
filesystem was gone for the domUs and in advance the domU failed some seconds
later.

In that case you could reach it some more time via vnc until the system hangs
because of missing blockdevices.

I solved that via routed networking, the automatic bridge-script destroyed my
net-setup after migration.

Kind regards, Florian
> -----Ursprüngliche Nachricht-----
> Von: xen-users-bounces@lists.xensource.com 
> [mailto:xen-users-bounces@lists.xensource.com] Im Auftrag von 
> Martin Fernau
> Gesendet: Donnerstag, 5. März 2009 18:19
> An: xen-users@lists.xensource.com
> Betreff: Re: [Xen-users] Urgent Network problem! Virtual 
> network stops workingafter a few minutes/hours
> 
> > Hi,
> >
> > > I''m using xen 3.3.0 on one server with 3 Guests (2
windows guest
> > > with gplpv drivers and one linux). After booting the 
> machine all is 
> > > fine. The domains came up and network is working. But 
> after several 
> > > minutes or hours the network stops to work. I can''t
reach
> the guests 
> > > any more! only dom0 is reachable
> >
> > Has this always been like this or just recently?
> The problems just started today. This machine runs "fine" 
> since a few month. 
> But after a reboot this morning the troubles begun...
> 
> 
> > > though the network. I asume the virtual network crashes.
> > > How can I find out where the problem is? Log files? 
> command line tools?
> > >
> > > This is an urgent problem as this machine is a live 
> server! Any help 
> > > would be very appreciated!
> >
> > Do you have control over the network switches? Maybe port 
> security is 
> > enabled on the port in the switch that you are connected to? I had 
> > this problem and the dom0 and domU would take turns going 
> on and off 
> > the network every few minutes. I had to tell the network 
> guys to turn 
> > port security off so that the switch allows multiple MAC 
> addresses on 
> > the same port.
> 
> To be more clear:
> I can''t reach the guests neither from the network nor from 
> the dom0 itself - but I can reach then dom0 from the network. 
> This make me think that there must be something wrong with 
> the internas of xen or the virtual networking stack.
> 
> I''m using the bridging mode by the way.
> 
> Regards,
> Martin
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 
**********************************************************************************************
IMPORTANT: The contents of this email and any attachments are confidential. They
are intended for the
named recipient(s) only.
If you have received this email in error, please notify the system manager or
the sender immediately and do
not disclose the contents to anyone or make copies thereof.
*** eSafe scanned this email for viruses, vandals, and malicious content. ***
**********************************************************************************************


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> ..what about accessibility via vnc/framebuffer?
> ...what about CPU-usage of domU?
>
> Last time i had this was while migrating drbd-layerd images from one xen to
> the other in bridged mode. It disturbt the drbd connection and drbd shut
> down. Then filesystem was gone for the domUs and in advance the domU failed
> some seconds later.
>
> In that case you could reach it some more time via vnc until the system
> hangs because of missing blockdevices.
>
> I solved that via routed networking, the automatic bridge-script destroyed
> my net-setup after migration.
Thanks for your reply. I discoverd my problem as I posted to this mailinglist 
earlier.

--- cut
I discovered to problem!

I don''t know why - but the trouble started from the time after I set
the ip-
adress for the dom0 to one ip from the same network as the guests. Until 
yesterday I had a private IP 172.16.0.1 for my dom0 where all other Computers 
in the LAN had public IPs. But to have better access to the dom0 I decided to 
change this to one IP from the same public network - and this was the 
problem.
After I switched back to the private IP all is working fine again. network is 
stable since hours now

I don''t know what the problem is in this case. Maybe something with
arp?
--- cut

Regards,
Martin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 2009-03-05 Martin Fernau wrote:
>>>> did you make sure port security was turned off on the switch?
>>>  well, I can''t see why a switch should impcat a local
connection
>>> between the dom0 and domU.
>>  Yes it can. But since you''re saying that it''s an
unmanaged switch
then
>> I don''t know what the problem could be. If the "network
crashes" then
>> there is something seriously wrong with the xen kernels. What distro
>> are you running?
> This is a gentoo system. I run linux-2.6.27 but I try to downgraded to
> the latest supported kernel 2.6.18 at the moment. I hope this
downgrade
> will have no impact on my windows guest running gplpv driver.
I have experienced similar problems on my Gentoo / Xen system.  I
haven''t had time to properly investigate but it seems that the Xen
kernels and >=xen(-tools)-3.2 cause the problem, I''m currently on
3.1.3-r1.  Perhaps you had emerged new versions a while ago but only
just restarted and they''ve just taken effect?  I''m really
looking
forward to the merge of the pv_ops dom0 code to mainline to see if this
fixes it for me.

James

This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at http://www.amdocs.com/email_disclaimer.asp

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users