Xen users - Sep 2006 - live migration time increases when using "mem-set"

Hi,
I''m noticing very strange behavior when I create a VM, reduce its
memory allocation using "xm mem-set" and then live migrate it.

I would expect reducing the memory allocation would make the migration
occur faster, but instead I am seeing the opposite.

For example, if I create a VM with 900MB of ram initially, these are
the times I measure for different reduced memory allocations:

RAM (MB) - Migration length (sec)
900 - 11
768 - 20
512 - 33
450 - 38
256 - 51
128 - 55

Just the opposite of what I would expect!

As reference, if a VM is created with 256 MB of ram and migrated, that
takes about 4 seconds.

The VM is idle and so is the gigabit network.

Looking at the xend.log files it looks like I am getting hundreds of
lines like this:
ERROR (XendCheckpoint:227) netbuf race: iter 4, pfn 28f78. mfn ffffffff

I am using xen-testing... I''m very curious if this still occurs in
xen-unstable but can''t test that myself right now.

-Tim

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tim -

What services are running on the dom-u''s , and did you take any
measures
from within the dom-u''s to clear idle child daemons out of cache?

In other words lets use lighttpd with php as fastcgi for example.
Lighttpd starts xx # of php children upon startup, and scales to the max
# allowed during use.

So if you had .. 30 children @ 15 MB each, and each one malloc()''ing an
additional 10 MB of elbow room, then reduce ram suddenly and migrate, I
would think those children would start paging dirty .. which would
increase i/o wait and could explain it.

If you did restart daemons that start idle children so the cache was
freed prior to reducing memory allocated .. then I''m baffled too.

HTH
- Tim


On Mon, 2006-09-04 at 16:48 -0400, Tim Wood wrote:
> Hi,
> I''m noticing very strange behavior when I create a VM, reduce its
> memory allocation using "xm mem-set" and then live migrate it.
> 
> I would expect reducing the memory allocation would make the migration
> occur faster, but instead I am seeing the opposite.
> 
> For example, if I create a VM with 900MB of ram initially, these are
> the times I measure for different reduced memory allocations:
> 
> RAM (MB) - Migration length (sec)
> 900 - 11
> 768 - 20
> 512 - 33
> 450 - 38
> 256 - 51
> 128 - 55
> 
> Just the opposite of what I would expect!
> 
> As reference, if a VM is created with 256 MB of ram and migrated, that
> takes about 4 seconds.
> 
> The VM is idle and so is the gigabit network.
> 
> Looking at the xend.log files it looks like I am getting hundreds of
> lines like this:
> ERROR (XendCheckpoint:227) netbuf race: iter 4, pfn 28f78. mfn ffffffff
> 
> I am using xen-testing... I''m very curious if this still occurs in
> xen-unstable but can''t test that myself right now.
> 
> -Tim
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 
> 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 9/4/06, Tim Post <tim.post@netkinetics.net>
wrote:
> Tim -
>
> What services are running on the dom-u''s , and did you take any
measures
> from within the dom-u''s to clear idle child daemons out of cache?
>
Good idea, but it doesn''t seem to be the cause of the problem.  I was
running apache and mysql, but even after stopping them I am getting
the same behavior.  The CPU is idle on the VMs.

Strange...

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tim -

What about inodes? Are you using any kind of network file systems from
within the dom-u''s? Exporting any NBD''s from dom-0 to the
dom-u''s?
Something is getting ''squished'' when you reduce memory, I
think. I just
don''t know what :)

I think the answer is going to lie in slabinfo in the dom-u, or dom-0,
or both. 

I recommend writing a script to continuously save /proc/slabinfo as a
journal on dom-0 and the dom-u while you duplicate the problem again,
and perhaps you''ll find the answer to it there. 

Maybe not quite the right guess, but I think its a guess in the right
direction.

Odd, indeed. :)

Thanks 
- Tim

On Wed, 2006-09-06 at 10:15 -0400, Tim Wood wrote:
> On 9/4/06, Tim Post <tim.post@netkinetics.net> wrote:
> > Tim -
> >
> > What services are running on the dom-u''s , and did you take
any measures
> > from within the dom-u''s to clear idle child daemons out of
cache?
> >
> 
> Good idea, but it doesn''t seem to be the cause of the problem.  I
was
> running apache and mysql, but even after stopping them I am getting
> the same behavior.  The CPU is idle on the VMs.
> 
> Strange...
> 
> 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users