Xen devel - Nov 2007 - Xend transaction reduction breaks migration

Hey Daniel,

It turns out that your changeset 16260 in xen-unstable breaks our localhost
non-live migration tests. What happens is that randomly (but fairly often)
xend will throw an assertion at line ''assert console_port'' in
XendCheckpoint.py. This is in the domain_restore_fd context, so result is
that the restorer dies and then of course the saver fails soon after since
the migration socket goes away.

Any idea how your patch could cause this rather bizarre behaviour? I
haven''t
worked it out myself, though I''m 99% sure it *is* that changeset that
has
caused this regression (based on tests with the changeset reverted).

It''s currently the only blocker in our automated tests on the staging
tree,
and blocks pushing to the main xen-unstable tree. :-)

 Thanks,
 Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Nov 14, 2007 at 11:44:42PM +0000, Keir Fraser
wrote:
> Hey Daniel,
> 
> It turns out that your changeset 16260 in xen-unstable breaks our localhost
> non-live migration tests. What happens is that randomly (but fairly often)
> xend will throw an assertion at line ''assert
console_port'' in
> XendCheckpoint.py. This is in the domain_restore_fd context, so result is
> that the restorer dies and then of course the saver fails soon after since
> the migration socket goes away.
> 
> Any idea how your patch could cause this rather bizarre behaviour? I
haven''t
> worked it out myself, though I''m 99% sure it *is* that changeset
that has
> caused this regression (based on tests with the changeset reverted).
Basically, instead of doing many small transactions, my change made some stuff
all happen in one large transaction. Oddly though, I tried to be careful so
that I only changed the codepath for the ''xm list'' operation
(well the SEXPR
call to the list operation). Other operations like create, save, restore
ought to be left using fine grained transactions as before.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 15/11/07 18:39, "Daniel P. Berrange" <berrange@redhat.com>
wrote:
>> Any idea how your patch could cause this rather bizarre behaviour? I
haven''t
>> worked it out myself, though I''m 99% sure it *is* that
changeset that has
>> caused this regression (based on tests with the changeset reverted).
> 
> Basically, instead of doing many small transactions, my change made some
stuff
> all happen in one large transaction. Oddly though, I tried to be careful so
> that I only changed the codepath for the ''xm list''
operation (well the SEXPR
> call to the list operation). Other operations like create, save, restore
> ought to be left using fine grained transactions as before.
I''ll have another look at nearby changesets in xen-unstable then. Yours
doesn''t really look all that likely to cause this problem, except that
the
empirical evidence does point at it.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 15/11/07 18:46, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk>
wrote:
> On 15/11/07 18:39, "Daniel P. Berrange"
<berrange@redhat.com> wrote:
> 
>> all happen in one large transaction. Oddly though, I tried to be
careful so
>> that I only changed the codepath for the ''xm list''
operation (well the SEXPR
>> call to the list operation). Other operations like create, save,
restore
>> ought to be left using fine grained transactions as before.
> 
> I''ll have another look at nearby changesets in xen-unstable then.
Yours
> doesn''t really look all that likely to cause this problem, except
that the
> empirical evidence does point at it.
It is your patch. The readDomTxn() functions and friends should prepend the
dompath, not the vmpath.

We wouldn''t notice this normally since self.console_port is not
actually
really used after domain building. So we have to race the migration thread
to get the assertion to fire, and that happens fairly rarely.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Nov 15, 2007 at 10:11:29PM +0000, Keir Fraser
wrote:
> On 15/11/07 18:46, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk>
wrote:
> 
> > On 15/11/07 18:39, "Daniel P. Berrange"
<berrange@redhat.com> wrote:
> > 
> >> all happen in one large transaction. Oddly though, I tried to be
careful so
> >> that I only changed the codepath for the ''xm
list'' operation (well the SEXPR
> >> call to the list operation). Other operations like create, save,
restore
> >> ought to be left using fine grained transactions as before.
> > 
> > I''ll have another look at nearby changesets in xen-unstable
then. Yours
> > doesn''t really look all that likely to cause this problem,
except that the
> > empirical evidence does point at it.
> 
> It is your patch. The readDomTxn() functions and friends should prepend the
> dompath, not the vmpath.
Aha. Thanks for finding & fixing that.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel