Xen users - Aug 2008 - Live migration - Networking take 2 minutes or so to return

I have live migration set on a Xen-DRBD setup. Migration of a 2Gb RAM
Win 2003 machine is very fast and I can VNC into it fine but networking
takes 2 minutes or so to come back on line. Disabling the nic and
re-enabling it within the Windows VM fixes the problem straight away. Is
there something wrong with my Xen Config? Any know fixes for this?

 

Thanks,

 

Rob


The SAQ Group
Registered Office: 18 Chapel Street, Petersfield, Hampshire. GU32 3DZ
SEMTEC Limited trading as SAQ is Registered in England & Wales
Company Number: 06481952

http://www.saqnet.co.uk AS29219
SAQ Group Delivers high quality, honestly priced communication and I.T. services
to UK Business.
DSL : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups :
Managed Networks : Remote Support.

Find us in http://www.thebestof.co.uk/petersfield



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,
> 2003 machine is very fast and I can VNC into it fine but networking takes 2
> minutes or so to come back on line. Disabling the nic and re-enabling it
> within the Windows VM fixes the problem straight away. Is there something
> wrong with my Xen Config? Any know fixes for this?
perhaps your switch didnt get it? You move the MAC address from one
port to another, so cam table has to be rebuild. Easiest way, start a
RDP session during migration and generate some traffic or get a vnc
console and ping your gateway, for example. does it fix the problem?

regards Martin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Martin,

Thanks for the help. 

I don''t know if this is related but I see very odd outgoing ping
results
from within the Windows VMs (Pinging the Windows VM from outside is
fine), here is an example, the times are wrong, it is definitely not
taking 8 seconds for some pings, the weird high number pings seem to
skip through very fast as if the system does not bother even sending the
ping).
Reply from 10.0.0.2: bytes=32 time<1ms TTL=64
Reply from 10.0.0.2: bytes=32 time<1ms TTL=64
Reply from 10.0.0.2: bytes=32 time=82987ms TTL=64
Reply from 10.0.0.2: bytes=32 time=82987ms TTL=64
Reply from 10.0.0.2: bytes=32 time<1ms TTL=64
Reply from 10.0.0.2: bytes=32 time=-83558ms TTL=64
Reply from 10.0.0.2: bytes=32 time<1ms TTL=64
Reply from 10.0.0.2: bytes=32 time<1ms TTL=64
Reply from 10.0.0.2: bytes=32 time<1ms TTL=64

An RDP connection seems to cut network migration time down to about
20seconds so at least it helps. Pinging the router from within the VM
during migration means I only lose 4 pings and about 5 seconds downtime.
I''m guessing this issue might be down to the cheap router and switches
we use in the office, hopefully it will be OK once it is in a production
environment with much better Cisco gear.

The only other possible cause I can think of is that I get an increase
MAX_SKBUFF_ORDER error on boot:
Aug 12 09:51:52 mrhappy kernel: Attempt to allocate order 5 skbuff.
Increase MAX_SKBUFF_ORDER.
Aug 12 09:51:53 mrhappy kernel: Attempt to allocate order 5 skbuff.
Increase MAX_SKBUFF_ORDER.
Aug 12 09:51:55 mrhappy kernel: Attempt to allocate order 5 skbuff.
Increase MAX_SKBUFF_ORDER.
Aug 12 09:51:55 mrhappy kernel: Attempt to allocate order 5 skbuff.
Increase MAX_SKBUFF_ORDER.
Aug 12 09:51:59 mrhappy kernel: Attempt to allocate order 5 skbuff.
Increase MAX_SKBUFF_ORDER.
Aug 12 09:52:03 mrhappy kernel: Attempt to allocate order 5 skbuff.
Increase MAX_SKBUFF_ORDER.
Aug 12 09:52:03 mrhappy kernel: Attempt to allocate order 5 skbuff.
Increase MAX_SKBUFF_ORDER.
Aug 12 09:52:03 mrhappy kernel: Attempt to allocate order 5 skbuff.
Increase MAX_SKBUFF_ORDER.
Aug 12 09:52:08 mrhappy kernel: Attempt to allocate order 5 skbuff.
Increase MAX_SKBUFF_ORDER. 

It does not seem to recur after boot though. Is this SKBUFF error
anything to worry about?

Thanks,

Rob


-----Original Message-----
From: xen-users-bounces@lists.xensource.com
[mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Martin
Hierling
Sent: 11 August 2008 18:54
To: xen-users
Subject: Re: [Xen-users] Live migration - Networking take 2 minutes or
so toreturn

Hi,
> 2003 machine is very fast and I can VNC into it fine but networking
takes 2
> minutes or so to come back on line. Disabling the nic and re-enabling
it
> within the Windows VM fixes the problem straight away. Is there
something
> wrong with my Xen Config? Any know fixes for this?
perhaps your switch didnt get it? You move the MAC address from one
port to another, so cam table has to be rebuild. Easiest way, start a
RDP session during migration and generate some traffic or get a vnc
console and ping your gateway, for example. does it fix the problem?

regards Martin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

The SAQ Group
Registered Office: 18 Chapel Street, Petersfield, Hampshire. GU32 3DZ
SEMTEC Limited trading as SAQ is Registered in England & Wales
Company Number: 06481952

http://www.saqnet.co.uk AS29219
SAQ Group Delivers high quality, honestly priced communication and I.T. services
to UK Business.
DSL : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups :
Managed Networks : Remote Support.

Find us in http://www.thebestof.co.uk/petersfield


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,
> An RDP connection seems to cut network migration time down to about
> 20seconds so at least it helps. Pinging the router from within the VM
> during migration means I only lose 4 pings and about 5 seconds downtime.
> I''m guessing this issue might be down to the cheap router and
switches
> we use in the office, hopefully it will be OK once it is in a production
> environment with much better Cisco gear.
i have my xen stuff connected to a cisco 3750 and sometimes i lose 1
ping, sometimes i only have some high ms ping time.
But that is switch related, not router. the router has the same IP/MAC
pair, even if you migrate, the switch needs the time to see the new
mac on the new switchport.

try it with cisco.

regards Martin

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Robert Dunkley wrote:
>
> I have live migration set on a Xen-DRBD setup. Migration of a 2Gb RAM 
> Win 2003 machine is very fast and I can VNC into it fine but 
> networking takes 2 minutes or so to come back on line. Disabling the 
> nic and re-enabling it within the Windows VM fixes the problem 
> straight away. Is there something wrong with my Xen Config? Any know 
> fixes for this?
>
What is the version of xen and on what distribution you are running it?
If it''s Centos or RHEL, http://people.redhat.com/dzickus/el5/ would
mitigate this delay a lot.

--Sadique
>  
>
> Thanks,
>
>  
>
> Rob
>
> **
>
> ** 
>
> *The SAQ Group*
>
> *Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ*
> SEMTEC Limited Trading as SAQ is Registered in England & Wales
> Company Number: 06481952
>
>  
>
> http://www.saqnet.co.uk <http://www.saqnet.co.uk/> AS29219
>
> SAQ Group Delivers high quality, honestly priced communication and 
> I.T. services to UK Business.
>
> DSL : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : 
> Backups : Managed Networks : Remote Support.
>
>  
>
> Find us in http://www.thebestof.co.uk/petersfield
>
>  
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, 2008-08-15 at 17:29 +0530, Sadique Puthen wrote:
> Robert Dunkley wrote:
> What is the version of xen and on what distribution you are running it?
> If it''s Centos or RHEL, http://people.redhat.com/dzickus/el5/
would
> mitigate this delay a lot.
Hi Sadique,

Interesting. Any further explanation accompanying those packages?  What
has been fixed?  Will it make it into the official RHEL updates?  Does
it need to be domU or dom0?

Thanks,

John.
-- 
http://www.brightbox.co.uk - UK Ruby on Rails hosting


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

John Leach wrote:
> On Fri, 2008-08-15 at 17:29 +0530, Sadique Puthen wrote:
>   
>> Robert Dunkley wrote:
>>     
>
>   
>> What is the version of xen and on what distribution you are running it?
>> If it''s Centos or RHEL, http://people.redhat.com/dzickus/el5/
would
>> mitigate this delay a lot.
>>     
>
> Hi Sadique,
>
> Interesting. Any further explanation accompanying those packages?  What
> has been fixed? 
A backport of 
http://xenbits.xensource.com/xen-unstable.hg?rev/42b29f084c31  to 
prevent delay in bringing up networking after live migration.
>  Will it make it into the official RHEL updates?
It should.
>   Does
> it need to be domU or dom0?
>
> Thanks,
>
> John.
>   

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sat, 16 Aug 2008, Sadique Puthen wrote:
> A backport of http://xenbits.xensource.com/xen-unstable.hg?rev/42b29f084c31
> to prevent delay in bringing up networking after live migration.
Is this required for every DomU, or just the Dom0s?

-Nathan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

nathan@robotics.net wrote:
> On Sat, 16 Aug 2008, Sadique Puthen wrote:
>
>> A backport of 
>> http://xenbits.xensource.com/xen-unstable.hg?rev/42b29f084c31 to 
>> prevent delay in bringing up networking after live migration.
>
> Is this required for every DomU, or just the Dom0s?
Read the patch. It brings changes to both netback and netfront drivers. 
So required on both.

--Sadique
>
> -Nathan
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users